Consider the following question: Do polar bears have tails? What was your mind doing to answer that question? If you are like most people, you probably imagined an image of a polar bear in your mind and looked for the tail. But another way you could have answered the question is to consider what properties bears have in general (say, short tails), and apply that property to polar bears as well. As simple as this example is, the existence of these two ways of answering a question suggests that humans can both think logically and imagine pictures. One might ask similar questions for sounds, smells, and tastes, which would involve similar kinds of vivid recall.

But what exactly happens when your mind forms an image? Why, for example, do images seem to have colour, but not the same kind of colour as real life? Why do some people seem to have photographic memory – the ability to imagine any scene from your past with perfect accuracy – is this memory real or is it simply a vivid reconstruction? Answering these questions depends on a systematic understanding of mental imagery – that is, imagination – and its function in the context of other brain functions.

One of the earliest notable mentions of mental imagery in scientific literature comes from Francis Galton, who in 1880 sought to study the difference between mental imagery in scientists and non-scientists. His method was to have people fill out questionnaires about the degree of vividness in their mental images. Later studies – in fact, studies for the next hundred years – also relied either on questionnaires or on tasks like asking people to rotate a complex shape in their heads.

But all these studies came with a serious limitation: they treated the brain as a black box. For example, let’s say a person is slower to imagine rotating a shape through a larger angle compared to rotating through a smaller angle. Does that imply they are rotating the shape mentally, or do they implement some algorithm (such as matrix multiplication) to do the rotation? Even though these experiments were elegant in their setup, they failed to determine even the existence of imagination, because the results could be explained in other ways.

These conflicting explanations, known in psychology as the imagery debate, left two key questions: first, did we really have imagined images, or were they an illusion experienced by a logical mind? And if imagination existed, in what way was imagining different from seeing? Only by answering these questions could we know if sensory imagination existed as its own function.

Fortunately, the 1990s saw the rise of two important technologies in neuroscience: transcranial magnetic stimulation (TMS) and functional magnetic resonance imaging (fMRI), which turned mental imagery from an invisible phenomenon to a well-understood part of our psychology.

First, on the question of whether people truly have mental images, we have strong evidence suggesting that we do. In the visual domain, several dozen studies have found that the primary visual cortex is activated during visual imagination. Moreover, the specific areas activated parallel those used in visual perception. For example, areas specific to coarse features in vision light up when people imagine large images, and the same goes for areas for medium and fine features, and injuries to areas specific for identifying objects in the real world lead to parallel deficits in identifying objects in mental images. In other words, to the extent that we perceive real-world images as pictures (and not linguistic or logical entities), we also perceive mental images in this way. Similar results for the association between perception and imagination exist for the auditory system – although, strangely, existing studies have found no evidence of primary auditory cortex activation during auditory imagination.

So it seems likely that mental imagery co-opts the hardware of the senses for constructing imagined scenes. But it is clearly not the same, as otherwise we would regularly confuse the polar bears of our imagination with real polar bears. What is the difference between these two kinds of polar bears?

Intuitively, we might say that one polar bear comes from your mind, while the other polar bear comes from the real world. Indeed, mental imagery appears to originate from the frontal lobe or otherwise higher areas of the cortex, while visual stimuli originate from the retina. These two streams of input meet in the primary visual cortex.

But the story is not so simple – instead, what we imagine influences what we see. For example, when subjects are asked to imagine a colour before being presented with conflicting colours in the two eyes, they are more likely to see the imagined colour when it is presented. This suggests an alternative view of mental imagery, in line with the Bayesian brain hypothesis and active inference: our imagination represents our predictions about the world, which are updated in response to visual stimuli.

This prediction-based view of imagination also aligns with the active inference interpretation of disorders like schizophrenia. Schizophrenia is correlated with a 25% decrease in the size of the primary visual cortex, as well as unusually vivid mental images. Hallucinations can be thought of as the inability to distinguish mental images from real visual inputs. A smaller visual cortex might imply that visual input is treated as less reliable, making mental images take precedence in a schizophrenic person’s perception of the world. Notably, the neurotransmitters for the forward connections (sensory-to-prediction) and backwards connections (prediction-to-sensory) might be different. Should this explanation be correct, it would imply that adjusting the levels of particular neurotransmitters (for example, from the frontal lobe to the primary visual cortex) could be a way of treating such disorders.

Another implication of this neural approach to imagery is that we can decode what people are visualizing, based on recordings of their visual cortex. Although the resolution of our measuring instruments is still low, this would in principle enable us to create brain machine interfaces that allow us to communicate directly from our mental imagery to computers and other people. Imagine if, instead of tinkering with a CAD tool or meticulously editing a slideshow, you could merely imagine what you want and see it come to life? Moreover, the ability to decode these mental images implies the ability to encode them as well: what kinds of social interactions would be enabled if we could communicate directly through the images, sounds, tastes, and smells of our imagination?

Why, if mental imagery is so essential, do some people report weak or missing mental images? One answer comes from a picture of the brain as a prediction machine, in which a lack of mental imagery might correspond to weaker prior predictions – a difference in degree rather than in kind. Still, this condition, called aphantasia, is still not well understood.

Beyond its therapeutic and technological implications, mental imagery reveals something fundamental about the nature of our perception. Mental imagery is real, picture-like, and closely related to perception. Our reality is not just what we see, but also what we imagine.

References

Galton, F. (1880). Statistics of Mental Imagery. Mind, 5(19), 301–318. http://www.jstor.org/stable/2246391

Dance, C. J., Ipser, A., & Simner, J. (2022). The prevalence of aphantasia (imagery weakness) in the general population. Consciousness and Cognition, 97, 103243.

Zeman, A., Milton, F., Della Sala, S., Dewar, M., Frayling, T., Gaddum, J., … & Winlove, C. (2020). Phantasia–the psychological significance of lifelong visual imagery vividness extremes. Cortex, 130, 426-440.

Pearson, J. (2019). The human imagination: the cognitive neuroscience of visual mental imagery. Nature reviews neuroscience, 20(10), 624-634.

Keogh, R., & Pearson, J. (2018). The blind mind: No sensory visual imagery in aphantasia. Cortex, 105, 53-60.

Kirchhoff, M. D. (2018). Predictive processing, perceiving and imagining: Is to perceive to imagine, or something close to it? Philosophical Studies, 175(3), 751-767.

Bergmann, J., Genç, E., Kohler, A., Singer, W., & Pearson, J. (2016). Smaller primary visual cortex is associated with stronger, but less precise mental imagery. Cerebral Cortex, 26(9), 3838-3850.

Kosslyn, S. M., Ganis, G., & Thompson, W. L. (2001). Neural foundations of imagery. Nature reviews neuroscience, 2(9), 635-642.