How old are you? For most people, this question brings up an automatic number. But how do you know how old you are? Unless you take a tally mark each time the sun rises – or, like most people, let the Gregorian calendar do the tallying for you – nothing in your memory will tell you how many years you’ve been alive. And it’s not only the length of years that gets warped in memory – even on the scale of a single day, some experiences seem to take longer times than others, even if they occupy the same number of hours. Rather than traversing a regular rhythm of city blocks, the elapsed time of memory may feel more like a drive down a desert highway, with long stretches of uneventful sameness demarcated by the occasional rock formation representing a death or a marriage. These significant events we might call “temporal landmarks.”

What explains the nature of this subjective experience of past time? Our ability to store and retrieve past events, called episodic memory, can be understood by looking to its neural correlates. One way to identify neural correlates for a behavior is to examine what kind of brain damage results in loss of that function. In 1957, Scoville and Milner found that lesions to parts of the medial temporal lobe (MTL), particularly the hippocampus, resulted in severe memory loss. In one famous example, a patient known by his initials H.M. experienced severe amnesia after large sections of his MTL (including most of the hippocampus on both sides) were removed to cure his epilepsy. Patient H.M., and a few patients with similar surgical operations, could not remember new events or events from the recent past, but had no problem remembering the distant past [1]. This study seemed to suggest that the hippocampus and adjacent regions played a critical role in long-term memory.

Notably, patients like H.M. did not lose all capacity for new memory formation. Later studies showed that other forms of memory, including skills like riding a bike or conditioning effects, seemed intact in patients with hippocampal damage [2]. By 1992, theories of hippocampal function generally proposed a distinction between declarative memory – conscious memory – and the kind of non-declarative or implicit memory that patients like H.M seemed to have no trouble with. Moreover, they proposed a distinction between long-term and short-term memory. In particular, patients with hippocampal damage were reasonably good at remembering the distant past, but struggled to form new memories, suggesting an inability to commit new events to long-term memory. Perhaps, then, the hippocampus plays a role in memory consolidation for declarative memories, with non-declarative and distant memories unaffected. Similar results were corroborated in studies of monkey and mice.

Having narrowed the subject of study to the hippocampus and MTL, how can their neuroanatomy explain the nature of our memory of the past? To answer this question, lesion studies are insufficient – a combination of neuroimaging and single-cell recordings is more useful. Inspired by such single-cell recordings of neurons, one mechanistic theory of the hippocampus proposes that certain cells in the hippocampus, called “concept cells,” encode concepts and associations in partially overlapping assemblies [3]. These cells encode information in a sparse manner, such that a small number of physically distant neurons in the hippocampus might encode the concept of “Ryan Gosling,” and another small group of neurons might represent the concept of the movie La La Land. An association is represented when the two assemblies overlap in some of their neurons. Crucially to the theory, the neurons are multimodally invariant – in other words, the same assembly of neurons responds to the concept of Ryan Gosling whether presented with a picture, his written name, or perhaps even his voice, for fans who would recognize it. This theory makes two strong predictions: first, that representations are sparse, and second, that representations represent concepts and not simply combinations of perceptual features. If true, the theory would explain a fundamental ability of humans to relate disparate concepts on the fly, via partially overlapping assemblies.

The first assumption to test is that memories can be sparse, rather than distributed. If memories were distributed, such that a consistent swath of neurons fired to each stimulus, then it would be difficult (if not impossible) to decode specific memories from hippocampal recordings. In 2009, Hassabis and colleagues found that spatial memories could be decoded from fMRI recordings of the human hippocampus, implying that the neuronal ensembles representing these place memories must display some degree of sparsity [4].

The next assumption to test is whether there are neurons that fire in response to concepts upon recognition, as opposed to conjunctions of features or something unrelated. To test this assumption, Quiroga and colleagues used epileptic patients. These patients have a hole drilled into their skulls so that surgeons can remove the part of the brain that causes the epilepsy. After the removal, the patients generally stay in the hospital for a few weeks so that doctors can monitor their brain function and clear them for discharge from the hospital. During this time, they typically implant electrodes into the brain so that they can monitor the functioning of other brain regions. These electrodes provide a convenient way for neuroscientists to get single-cell recordings from the brains of these patients without subverting ethical standards. Using such patients, Quiroga and colleagues began by displaying a wide variety of images to the patients, to see which elicited a response from the actively measured neurons in the MTL. They found that a small percentage of the measured neurons responded on a small number of trials. Given this result, they continued on the next day with these “recognized” images. For example, if someone showed a neural response to a picture of Jennifer Aniston, the researchers could test the multimodal invariance of that neuron showing them or playing them an audio clip of Aniston’s name. But could the neurons perhaps be imagining Jennifer Aniston in their heads, and responding to some combination of visual features in all cases? To make sure this was not the case, the researchers ran other tests studying pictures with modified visual features that would also be recognized as the same “concept.” Foe example, in one subject they found a hippocampal neuron that responded to a picture of Jodie Foster, but not a picture of Nicole Kidman. When presented with a morphed picture combining features of the two actresses, the neuron fired only when the person recognized the picture as a picture of Foster, but not as a picture of Kidman, implying that it fired in response to the conscious recognition of the actress as opposed to an automatic combination of visual features.

But how does the idea of sparse networks of partially overlapping assemblies explain the role of the hippocampus in episodic memory? What accounts for the richness of remembered experiences? The key mechanism might be reconstruction, in which concept cell assemblies encode a limited scaffold of concepts but enlist the hardware of the neocortex to reconstruct the rest of our event memory. In support of this view, a 2014 study by Tanaka and colleagues in mice found that deactivating certain hippocampal cells (the ones firing in response to a memory recall event) prevented the reactivation of the corresponding cortical regions [5], implying that hippocampal cells play a causal role in reactivating cortical representations that are associated with the corresponding images. While much more energy efficient than storing picture-perfect memories, this theory of episodic memory would also imply two strange facts about human memory: first, there would be no neural correlate for external time in episodic memory, which would instead be inferred based on the contents of memory. Second, our memory would be liable to be colored both by associated concepts and the limits of our visual capacity for reconstruction, which itself might be guided by our visual experience. While these are both reflected in our real-world experience of memory, a purely associative event memory would have trouble playing events in the correct order unless each followed inevitably from the previous one – though maybe events in the real-world are predictable enough for this to work via pointers from one “concept” to the next.

However, if the hippocampus indeed encodes long-term episodic memory, then how can memories from the distant past remain intact in patients like H.M.? Moreover, if the hippocampus is critical for forming rapid declarative associations, how do you explain a result that patients with MTL damage can still learn word pairs in working memory, but not in long-term memory [6]? Maybe working relational memory can occur somewhere in neocortex, but needs to be transferred to an abstraction in the hippocampus before being eventually consolidated in the neocortex – but this is really a speculation, because the mechanism by which concept assemblies become consolidated memories in neocortex is unclear, if it happens at all. As a further speculation, it could be that memories are stored on a spectrum from easily associated abstract representations (as hippocampal memories) to cemented realist representations (as cortical memories), while generally involving cells of both types, even though the former is required for the creation of the new cortical concept-memories.

While most tests have focused on existing concepts, how does one explain the creation of new concept assemblies? The neural implementation of this might rely on “boundary” neurons, which detect the presentation of new and relevant stimuli. In a paper that echoes our earlier questions about subjective timekeeping, a 2022 study by Zheng and colleagues wondered how humans structure our episodic memories. How do we experience discrete event memories, even as the flow of time is continuous? Using single-cell recordings, they found that certain cells in the MTL, called boundary cells, fired in response to shifts in a video scene, and that scene recognition – the ability to recognize if a scene had been seen before – increased after boundaries in proportion to the firing of these boundary cells [7]. On the other hand, for hard boundaries – cuts between a video from one story and a video from an unrelated story – the ability to determine the order in which two scenes across the boundary were shown was compromised by those boundaries. This might imply a tradeoff between remembering event order and remembering scene details – a person might remember many details but not remember the order in which they were presented.

Remembering details but not the order in which they were presented sounds a lot like what is traditionally called “semantic memory”: memory for knowledge. Is there a difference between semantic and episodic memory? According to concept cell theory, episodic memories construct time loosely via associations, rather than being strictly event-based – the important distinction, then, is not whether memories refer to events or ideas, but whether these memories refer to freely-associated concepts or cemented and hierarchical reconstructions [8]. On the other hand, this theory does not yet offer compelling explanations of how working relational memory should work without such rapid association formation in the hippocampus, or how event order can be recovered reliably – or to what extent concept cell assemblies are truly involved in episodic memory as opposed to just concepts that can theoretically represent episodic memory.

In all likelihood, what we refer to as episodic memory may not be a monolithic entity. Consider, for example, the memory of eating a toxic berry. In the near term, the person would probably associate this memory with the path they took to the berry patch, the specific berry involved, and the effects they experienced afterwards. Hippocampal cells might rapidly encode these associations. Over the course of repeated rehearsal, the same neocortical regions are invoked each time, and come to be associated as well. This would make it harder to form new associations with that memory, but it would also make it more permanent. At a computational level, this would be an interesting way to implement the creation of discrete and realistic experiences from continuous world input. In terms of our subjective experience, this would imply that our memory of the past is more of a reconstruction than a replay.

References

[1] Scoville & Milner (1957). Loss of recent memory after bilateral hippocampal lesions. Journal of Neurology, Neurosurgery & Psychiatry, 20, 11-21

[2] Squire, L. R. (1992). Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. Psychological review, 99(2), 195.

[3] Quiroga, R. Q. (2015). Neuronal codes for visual perception and memory. Neuropsychologia.

[4] Hassabis, D., Chu, C., Rees, G., Weiskopf, N., Molyneux, P. D., & Maguire, E. A. (2009). Decoding neuronal ensembles in the human hippocampus. Current biology : CB, 19(7), 546–554. https://doi.org/10.1016/j.cub.2009.02.033

[5] Tanaka, K. Z., Pevzner, A., Hamidi, A. B., Nakazawa, Y., Graham, J., & Wiltgen, B. J. (2014). Cortical representations are reinstated by the hippocampus during memory retrieval. Neuron, 84(2), 347-354.

[6] Squire, L. R. (2017). Memory for relations in the short term and the long term after medial temporal lobe damage. Hippocampus, 27(5), 608-612.

[7] Zheng, J., Schjetnan, A. G., Yebra, M., Gomes, B. A., Mosher, C. P., Kalia, S. K., … & Rutishauser, U. (2022). Neurons detect cognitive boundaries to structure episodic memories in humans. Nature neuroscience, 25(3), 358-368.

[8] Quian Quiroga, R. (2023). An integrative view of human hippocampal function: Differences with other species and capacity considerations. Hippocampus, 33(5), 616-634.