How did people navigate the world before Google Maps? While I wasn’t personally around for most of this time, I am told that people used to keep track of their locations using mental maps. In fact, some people still do. Studying these mental maps is useful for people who design physical spaces, such as architects, urban planners, and VR world builders. But spatial representation might be inextricably connected to memory in general. For example, consider the memory palace, a method in which a person associates each item to be remembered with a distinct location along a well remembered route, such as the walk from the front door to the bedroom of their house. The memory palace is often used by competitors at the World Memory Championships to remember large numbers of numbers, words, and faces. But if your goal is to remember numbers, why would adding additional remembered information, like a series of locations, make the numbers easier to remember? Is there something special about spatial memory in the brain?

Evolutionarily, the ability to remember spatial locations seems quite useful. A mouse that can remember where to find food and mates will have a clear advantage over one that cannot. Given that location is a natural way to index remembered information, could the same mechanism – location-based memory – explain both app-free navigation and the success of the memory palace?

Research into the neural basis of spatial memory took a big step forward in the 1970s with the discovery of place cells – that is, cells in the hippocampus that respond to specific locations in space [1]. These cells reliably increased their firing rate when a mouse traveled through particular locations in a maze – more convincingly, the neurons corresponding to the “correct” maze route still fired when the rodents made a mistake, and the place cells corresponding to the learned route replayed their firing patterns during sleep [2]. Around the same time, neurosurgeons examining epileptic patients had realized that the hippocampus was also essential for episodic memory. For example, patient H.M., who had his hippocampus mostly removed during treatment, experienced a chronic inability to form new memories. Later studies confirmed the intuition that the hippocampus was critical for forming long-term episodic memories. But how could this result be reconciled with the view of the hippocampus as the brain’s mental map?

One reason why it might be hard to unify understandings of episodic memory and spatial memory in the hippocampus is the overly specific definition of episodic memory, which must be autobiographical and convey a sense of being relived – this is likely a complex experience involving many brain areas. An alternative account of hippocampal memories is the simpler “event memory”: “the mental construction of a scene recalled as a single occurrence” [7]. In contrast to episodic memories, event memories do not need to provide a sense of reliving the experience and do not need to be made by the participant. They may even combine multiple events – say, multiple trips to the same grocery store or TV-show grocery stores – into a single constructed scene. Perhaps the hippocampus indexes event memories, of which episodic memories and spatial memories are both subtypes.

The construct of event memory is inspired by the role of the hippocampus in scene construction, which Hassabis and Maguire define as “mentally generating and maintaining a complex and coherent scene or event” – that is, rather than storing anything exactly, the hippocampus is thought to represent memories in an abstract and general way, and each instance of recall is actually an instance of reconstruction colored by past experiences and expectations [3]. As evidence for this view, they cite that the imagination of fictitious scenes involves almost the same fMRI activation patterns as recalling past experiences, especially in the hippocampus. In addition, patients with hippocampal damage struggle not only to form new memories but also to imagine fictitious experiences, implying that the defect might lie with scene construction.

But the construction of a scene still requires the existence of a spatial context – the knowledge of one’s own place in the scene, which could be called the internal GPS. Knowledge of self-position might be supported by neurons from the medial entorhinal cortex (MEC). A few decades after place cells were identified, Moser and colleagues identified “grid cells” in the MEC, which fire in response to a triangular grid in the environment – that is, they keep track of the location of the organism within an imaginary grid and fire accordingly [2]. Thus, grid cells provide a direct neural equivalent of the GPS system used for Google Maps, helping an organism keep track of its own location. Place cells, meanwhile, code for specific spatial areas within this grid, encoding the spatial contexts of memories. More speculatively, an additional input to the hippocampus may come from the lateral entorhinal cortex (LEC), which might encode items to be contextualized as an object-spatial context pair in the hippocampus [4].

Although grid cells and place cells together provide a plausible mechanism both for the internal GPS and the memories of the memory palace, not all evidence supports the centrality of human hippocampal neurons in spatial cognition. To the contrary, a series of studies conducted by Kim and colleagues in 2015 seems to show that patients with hippocampal lesions demonstrated impaired episodic memory but intact performance on spatial tasks [5]. However, the apparently intact performance on spatial tasks might be explained by the specifics of their task design. One of the predictions of scene construction theory is that people with intact hippocampi experience “boundary extension” — that is, when recreating a cramped scene in a drawing (with cropping close to the subject of interest), people tend to expand the size of the background so that the subject fits comfortably. However, people with hippocampal damage are predicted to have impaired scene construction abilities, thus showing less boundary extension. Kim and colleagues found their patients to have levels of boundary extension indistinguishable from a control group, implying that the hippocampus might be key for episodic memory but not scene construction.

In a critique of the Kim and colleagues study, Maguire and colleagues argue that the researchers did not use good images for their tests, because the cropped areas were large compared to the close-ups used for the original studies demonstrating boundary extension [6]. Still, it seems that the hippocampus cannot be thought of as solely a spatial area – its true function may be more general. If our goal is to figure out a sentence of the form “the hippocampus does X, which supports Y and Z cognitive functions,” we should recognize that X is more likely to be a computational function than a direct behavioral task. By analogy, selective attention in the brain implements the computational function of choosing between bottom-up inputs based on the subject of attention, to support behaviors like visual search and object recognition. Perhaps things like “event memory” and “spatial navigation” are behaviors realized by some other computational function X.

In an attempt to answer what this X is, one computational model for the hippocampus casts it as a center for model-based planning, a kind of reinforcement learning that learns a model of the environment to predict future states and rewards [8]. Model-based learning is exactly the kind of learning that might be useful for navigating a maze, because it could be more useful to learn a mental map of the maze than to memorize a sequence of turns. And constructing a generative model of the world to make predictions sounds quite similar to constructing scenes from episodic or spatial memory, so it’s plausible that these are two ways of saying the same thing. To extend the statistical-analogy model further, Whittington and colleagues proposed the Tolman-Eichenbaum machine, a model of hippocampal and entorhinal neurons that successfully reproduces grid cells, border cells, and place cells through machine learning. Putting this result together with scene construction, we get the computational view that entorhinal cells provide a basis for describing structural knowledge, and inputs from this basis are combined within the hippocampus to form flexible cell assemblies, which then index cortical areas to construct scenes for spatial and event memory, as well as the imagination of fictitious scenes. In constructing these cell assemblies, it is efficient to maximize the use of existing learned structures – this would explain, for example, why memory palaces are so effective. Instead of learning a new set of random words, one can simply associate these words to an existing spatial structure, relying on the highly expressive components of existing structural knowledge.

In terms of its direct implications, the overlap between spatial and relational memory mechanisms gives new credence to Winston Churchill’s maxim: “First, we shape our buildings. Thereafter, they shape us.” More generally, spatial approaches to understanding relational memory – that is, seeing spatial cognition as a special case of relational memory – are converging with approaches in deep learning to reflect the fundamentally spatial nature of knowledge. In natural language processing, for example, all words can be encoded as n-dimensional vectors, in which spatial relationships reflect semantic relationships – for example, in the word2vec model, king – man + woman = queen [10]. Moreover, a recent study of large language models found that similar real-world locations were mapped to similar locations in embedding space, supporting the view that a generative model can learn cognitive maps [11]. Many deep learning models can be thought of as functions that seek to capture the most important dimensions of the underlying data. In the same way, the hippocampus seems to encode the direct analogy between physical space and informational space, making memory palaces an isomorphism from the space of words to encode to the space of a physical path.

References

[1] O’Keefe, J., & Dostrovsky, J. (1971). The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain research, 34(1), 171–175. https://doi.org/10.1016/0006-8993(71)90358-1

[2] Moser, M. B., Rowland, D. C., & Moser, E. I. (2015). Place cells, grid cells, and memory. Cold Spring Harbor perspectives in biology, 7(2), a021808.

[3] Hassabis, D., & Maguire, E. A. (2007). Deconstructing episodic memory with construction. Trends in cognitive sciences, 11(7), 299-306.

[4] Knierim, J. J. (2015). From the GPS to HM: Place cells, grid cells, and memory. Hippocampus.

[5] Kim, S., Dede, A. J., Hopkins, R. O., & Squire, L. R. (2015). Memory, scene construction, and the human hippocampus. Proceedings of the National Academy of Sciences, 112(15), 4767-4772.

[6] Maguire, E. A., Intraub, H., & Mullally, S. L. (2016). Scenes, spaces, and memory traces: what does the hippocampus do?. The Neuroscientist, 22(5), 432-439.

[7] Rubin, D. C., & Umanath, S. (2015). Event memory: A theory of memory for laboratory, autobiographical, and fictional events. Psychological Review, 122(1), 1.

[8] Vikbladh, O. M., Meager, M. R., King, J., Blackmon, K., Devinsky, O., Shohamy, D., … & Daw, N. D. (2019). Hippocampal contributions to model-based planning and spatial memory. Neuron, 102(3), 683-693.

[9] Whittington, J. C., Muller, T. H., Mark, S., Chen, G., Barry, C., Burgess, N., & Behrens, T. E. (2020). The Tolman-Eichenbaum machine: Unifying space and relational memory through generalization in the hippocampal formation. Cell, 183(5), 1249-1263.

[10] Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.

[11] Gurnee, W., & Tegmark, M. (2024). Language models represent space and time. arXiv preprint arXiv:2310.02207.