Computational properties of the hippocampus increase the efficiency of goal-directed foraging through hierarchical reinforcement learning
Gruber, Aaron J.
The mammalian brain is thought to use a version of Model-based Reinforcement Learning (MBRL) to guide “goal-directed” behavior, wherein animals consider goals and make plans to acquire desired outcomes. However, conventional MBRL algorithms do not fully explain animals’ ability to rapidly adapt to environmental changes, or learn multiple complex tasks. They also require extensive computation, suggesting that goal-directed behavior is cognitively expensive. We propose here that key features of processing in the hippocampus support a ﬂexible MBRL mechanism for spatial navigation that is computationally efﬁcient and can adapt quickly to change. We investigate this idea by implementing a computational MBRL framework that incorporates features inspired by computational properties of the hippocampus: a hierarchical representation of space, “forward sweeps” through future spatial trajectories, and context-driven remapping of place cells. We ﬁnd that a hierarchical abstraction of space greatly reduces the computational load (mental effort) required for adaptation to changing environmental conditions, and allows efﬁcient scaling to large problems. It also allows abstract knowledge gained at high levels to guide adaptation to new obstacles. Moreover, a context-driven remapping mechanism allows learning and memory of multiple tasks. Simulating dorsal or ventral hippocampal lesions in our computational framework qualitatively reproduces behavioral deﬁcits observed in rodents with analogous lesions. The framework may thus embody key features of how the brain organizes model-based RL to efﬁciently solve navigation and other difﬁcult tasks.
Sherpa Romeo green journal; open access
Reinforcement learning , Hierarchical learning , Hippocampus , Planning , Context , Adaptation
Chalmers, E., Luczak, A., & Gruber, A. J. (2016). Computational properties of the hippocampus increase the efficiency of goal-directed foraging through hierarchical reinforcement learning. Frontiers in Computational Neuroscience, 10(128). doi: 10.3389/fncom.2016.00128