Uncovering the Role of the Early Visual Cortex in Visual Mental Imagery

1. Introduction

Close your eyes and imagine an apple. Is this experience similar to actually seeing an apple or is it more like thinking about the idea of an apple? Answering this question, whether imagery relies on depictive or symbolic representations, has been referred to as the ‘imagery-debate’. This debate has dominated mental imagery research during the end of the last century and the beginning of this one (Nanay, 2021a; Sterelny, 1986). The development of cognitive neuroimaging promised a way to settle the debate once and for all: if mental images, like perceived images, are represented in the retinotopically organized early visual cortex (EVC), this would be knock-out evidence in favour of a depictive view. However, at this moment, more than three decades after the first neuroimaging studies on mental imagery, the role of the early visual cortex during visual mental imagery remains a topic of intense debate (Bartolomeo et al., 2020; Dijkstra et al., 2019; Pearson, 2019; Spagna et al., 2021).

In this paper, I argue that in order to make progress we need to move away from the question of whether the EVC is involved in mental imagery and instead move towards elucidating exactly how it is involved. I argue that many of the discrepancies in the literature can be explained by the unique methodological and conceptual challenges associated with the functional role of the EVC. I will start by explaining that the question of whether the EVC is ‘active’ or not during imagery is conceptually nonsensical in light of what we know about the functionality of the EVC. Instead, to elucidate the involvement of the EVC during mental imagery, its unique organization needs to be considered both in the experimental design as well as in the data analysis. I will discuss examples of studies that do this and explain what they tell us about how the EVC is involved in visual mental imagery. In light of the results of these kinds of studies, I propose that the current evidence suggests that the EVC represents fine-grained visual details during mental imagery, in the same way as it does this during perception. I will end by discussing how this account can explain contradictory evidence and highlight directions for future research.

2. Organizational principles of the early visual cortex

Decades of neurophysiological research has led to a highly detailed characterization of how the early visual cortex responds to external visual signals. Activity in EVC is retinotopically organized such that things that are spatially close together in the outside world elicit activity in neurons that are close together on the cortical sheet of the EVC. Furthermore, neurons in EVC have small receptive fields so that that each neuron only responds to signals coming from a very small portion of the visual field. Together, this means that activity in the EVC is highly sensitive to changes in low-level features of visual signals, such as their exact distribution of edges, location and orientation (Hubel & Wiesel, 1968). This is in contrast to neurons in more high-level visual areas which have much larger receptive fields and are therefore able to represent more ‘high-level’ features such as semantic meaning of a stimulus, irrespective of the specific low-level instantiation of that stimulus (Thorpe & Fabre-Thorpe, 2001). For example, the same neurons in high-level visual cortex might fire in response to a chair from many different viewpoints whereas the pattern of activity in EVC will be widely different in each specific instance.

The properties of the neurons in EVC have the consequence that any change in low-level features of visual signals, for example the exact location of a stimulus, will lead to a corresponding change in EVC activity (Figure 1A). While looking outside your window at a tree, any displacement of the branches or the leaves due to the wind will result in a change in activity in your early visual cortex. Furthermore, any movement of your eyes will change in which receptive fields the visual signals of the tree fall, again leading to corresponding changes in EVC activity. If you were to now test whether any given neuron in EVC was consistently active when you were looking at this tree, you would likely find very few neurons that showed consistent activity. Based on this, you could wrongfully conclude that the EVC was not involved in the perception of the tree. This is why most visual neuroscience experiments studying EVC are done with controlled repetitions of the exact same set of simple stimuli, presented in exactly the same location on the screen, with the participant’s head firmly fixated on a chin rest and taking dedicated measures to minimize eye-movements as much as possible. Under these controlled conditions, it is much more likely to observe consistent activity in EVC neurons.

3. The challenges of studying the early visual cortex during mental imagery

Now consider imagining a tree in your mind’s eye. How stable are the low-level features of this mental image? Are the locations of the branches fixed and clear from one moment to the next? Are you able to hold a clear and stable image without moving your eyes? A recent study showed that when asked to freely imagine a scene, most participants leave out many low-level details (Bigelow et al., 2023). Furthermore, some people might even experience mental images in a kind of separate space that is decoupled from the physical environment (Schwarzkopf, 2024). Herein lies one of the difficulties of studying the role of EVC in imagery: Given the sensitivity of activity in EVC to changes in low-level features, the fleeting and undefined nature of mental images makes it very hard to determine any consistent relationship. In order to properly study the role of EVC in imagery, it is therefore essential to have a firm handle on the low-level features of the mental images under investigation.

One way to control the low-level features of mental images is by instructing participants to imagine very simple stimuli. One of the earliest studies using this approach was by Klein and colleagues in 2004. In this study, participants were instructed to view and imagine horizontally and vertically oriented bow-tie stimuli while their brain activity was measured using functional magnetic resonance imaging (fMRI). Crucially, the exact spatial location and orientation of the to-be-imagined stimuli were clearly defined. The results revealed that, imagining a horizontal stimulus activated the horizontal meridian within the EVC while imagining a vertical stimulus activated the vertical meridian, in line with retinotopic involvement of EVC during imagery (Klein et al., 2004). Similar approaches have been used to show retinotopic encoding in EVC of imagined domino patterns (Thirion et al., 2006) and letters (Senden et al., 2019).

Another unique challenge of studying the EVC during imagery is that the way that EVC activity is instantiated during imagery is different to perception. Specifically, during imagery, activity in EVC is assumed to be instantiated via feedback connections from high-level areas (Dentico et al., 2014; Dijkstra, Zeidman, et al., 2017; Mechelli et al., 2004). Importantly, in contrast to feedforward connections, feedback connections tend to predominantly modulate neural activity, changing existing firing rates via gain control, but usually without driving neurons to fire action potentials directly (Aru et al., 2020; Larkum, 2013). In line with this, it has recently been proposed that imagery inhibits activity in irrelevant neuronal populations, rather than directly increasing activity in the populations representing the imagined stimulus (Pace et al., 2023). This is similar to the ‘sharpening’ process assumed to underlie attention (Bartsch et al., 2017) and expectation (Kok et al., 2012). Importantly, if imagery is instantiated through inhibition of irrelevant activity, overall activity levels in EVC might not increase and might even decrease compared to baseline (Figure 1B). This could explain the observation that aphantasia – the absence of visual mental imagery – has been associated with hyperactivity of the EVC (Keogh et al., 2020; Winlove et al., 2018). Importantly, however, if EVC is involved in imagery, the relative activity pattern should still be informative of which stimulus participants were imagining, even if general activation levels do not increase.

Testing whether neural activity patterns contain information about perceived or imagined stimuli is the main goal of ‘decoding’ techniques in cognitive neuroscience. In this context, decoding refers to the use of machine learning algorithms to describe how neural activity patterns relate to different stimuli or conditions (Robinson et al., 2023). In a standard imagery decoding experiment, a decoder is first ‘trained’ to capture the variation in brain activity patterns during the perception of different stimuli. This perception-decoder is then applied to the brain activity during imagery of the same stimuli, giving a ‘guess’ of which stimulus the participant was imagining. If the perception-classifier is able to accurately guess or ‘decode’ the imagined stimulus, it can be concluded that the pattern of neural activity during imagery is similar to that during perception in that brain area. Using this technique, significant cross-decoding between imagery and perception in EVC has been shown for gratings, letters, objects and shapes (Albers et al., 2013; Lee et al., 2012; Ragni et al., 2020; Senden et al., 2019).

Furthermore, to more directly test whether low-level visual features are encoded in EVC activity patterns, researchers have used so-called feature encoding models (St-Yves & Naselaris, 2018). These are models based on computer vision algorithms that describe complex images in terms of unique combinations of simple, low-level features. These models can in turn be combined with models that describe activity in different brain areas in response to the same low-level features to predict what the brain activity would be like for an entirely new set of complex images (Güçlü & van Gerven, 2015). For example, one study used a low-level feature encoding model to first determine how images of complex artworks could be described in terms of a combination of Gabor wavelets (Naselaris et al., 2015). Next, the activity in EVC during the perception of the same Gabor wavelets was measured to create a corresponding low-level feature model of EVC activity. Applying the transformation between Gabor wavelets and the artworks to EVC activity during the imagery of different artworks led to successful prediction of which artwork participants were imagining (Naselaris et al., 2015). Several other recent studies have used similar computer vision approaches to reveal low-level feature encoding of visual mental imagery in brain activity patterns (Koide-Majima et al., 2024; Shen et al., 2019).

Together, these studies highlight that in order to capture EVC involvement during visual mental imagery it is essential to use the right experimental and analytical approach. In this context, investigating whether the EVC is ‘active’ or not without specifying the exact way in which its activity relates to specific properties of the mental image is unlikely to reveal consistent involvement. For example, consider an experiment in which, on each trial, participants are asked to imagine a specific stimulus but are not asked to imagine that stimulus in a specific size or at a specific location. Due to the organization of the EVC, each of those imagery instances would be associated with different patterns of activity. Averaging over trials might therefore lead to the incorrect conclusion that imagery did not recruit the EVC. This can also explain why meta-analyses investigating activation of imagery of different stimuli over different studies are ill placed to find activation in EVC and are instead more likely to find consistent involvement of higher-level visual regions (Spagna et al., 2021).

4. The role of the early visual cortex in visual mental imagery

Considering the evidence until now, I propose that the EVC represents fine-grained visual details during imagery in similar ways as it does during perception, in line with a depictive view. However, importantly, that does not mean that all instances of visual mental imagery will activate the EVC. Consider for example a recent study in which participants were asked to imagine a person walking into a room and knocking a ball off a table (Bigelow et al., 2023). Afterwards, participants were asked several questions regarding the visual details of their mental imagery; i.e. ‘did you imagine the colour of the ball’, ‘the clothes the person was wearing’, etc. It turned out that most participants did not imagine the majority of basic visual details. The authors concluded that ‘while imagination may indeed be a good artist, it’s on a deadline, and stingy about paint’ (Bigelow et al., 2023). In other words, imagination can create vivid, detailed scenes, but that requires effort, so if it is not necessary, those details will be omitted.

This means that EVC activity should not always be observed during mental imagery, but should only be there when low-level details are indeed imagined. The idea that EVC activity depends on whether the imagery in question requires fine-grained details is in line with early observations that noted that the EVC is more likely to be activated in imagery tasks that required high-resolution discrimination judgements (Kosslyn & Thompson, 2003). Furthermore, the observation that the vividness of mental imagery correlates with perception-imagery cross-decoding accuracy in EVC (Dijkstra, Bosch, et al., 2017; Dijkstra & Fleming, 2023; Lee et al., 2012) is also in line with the idea that more detailed mental images are more likely to recruit EVC.

One line of research that has been taken as evidence against the involvement of EVC in visual mental imagery comes from neuropsychological observations of intact visual mental imagery after lesions to the EVC (Bartolomeo, 2002; Bartolomeo et al., 2020). One way to align these findings with the previously mentioned neuroimaging research is with the fact that imagery does not always contain fine-grained visual details (Pearson, 2020). This would mean that patients without a functioning EVC might still be able to generate mental images, but that these mental images would not contain the fine-grained visual details that require a functioning EVC. Future studies using controlled psychophysical paradigms in these patients are required to determine whether this is indeed the case.

Another potential argument against the involvement of EVC in imagery comes from recent studies that show that imagery can be decoded from EVC activity in aphantasia (Cabbai et al., 2024; Meng et al., 2023; Weber et al., 2023). However, the claim that EVC representations are necessary to imagine low-level features does not mean that these representations are also sufficient for conscious visual imagery. One possibility is that these low-level EVC representations need to be coupled with higher level representations in frontoparietal areas in order to reach awareness, as proposed by higher-order theories of consciousness (Brown et al., 2019; Fleming, 2020; Lau & Rosenthal, 2011). In line with this, a recent study found that there were no differences in visual regions between aphantasia and controls but that in aphantasia, these regions were functionally disconnected with fronto-parietal (Liu et al., 2023).

5. Conclusions and future directions

The question of whether visual mental imagery recruits the early visual cortex has been at the core of the imagery-debate for decades. Initial neuroimaging findings were inconclusive, with some studies finding reliable activity while other studies did not. In this paper, I have suggested that the inconsistency in the literature can be explained by the unique challenges associated with studying the EVC during visual mental imagery. The EVC is known for representing fine-grained visual details during perception. However, compared to perception, imagery tends to be associated with weaker and less consistent visual signals. Moreover, some instances of mental imagery might contain very few fine-grained visual details at all. These challenges require specific experimental and analytical approaches, such as using very simple stimuli or quantifying expected activity patterns with sophisticated models. The results of studies using these approaches consistently show that the EVC represents low-level visual details during mental imagery in similar ways as it does during perception.

A number of questions regarding the exact role of the EVC in visual mental imagery remain open. Firstly, the exact relationship between EVC representations and conscious imagery experience remains unclear. Several studies have shown a relationship between EVC involvement and subjective reports of imagery vividness (Dijkstra, Bosch, et al., 2017; Dijkstra & Fleming, 2023; Lee et al., 2012). However, recent studies suggest that EVC representations can be present in the complete absence of conscious visual imagery (Cabbai et al., 2024; Meng et al., 2023; Weber et al., 2023). Future research is necessary to investigate which factors determines whether imagery related EVC activity becomes conscious (Nanay, 2021b).

Relatedly, exactly how visual mental imagery representations are instantiated in EVC remains unclear. The current consensus is that this happens through (inhibitory) feedback connections from high-level areas (Dijkstra et al., 2019; Pearson, 2019). However, it is not clear exactly which high-level regions are referred to and exactly how activation flows through these different regions. Several studies have found increases in coupling between dorsolateral prefrontal and visual cortex during imagery (Dijkstra, Zeidman, et al., 2017; Mechelli et al., 2004). Other studies have shown increases in coupling between parietal and visual regions (Dentico et al., 2014). Lastly, research on scene imagery has focused on the role of medial frontal and hippocampal memory systems in generating mental images (Barry et al., 2018). Further research is necessary to fully elucidate the neural network involved in generating EVC activity during mental imagery.

Finally, given that imagery seems to be able to recruit the EVC in a similar way to perception, one important open question is how the brain keeps track of which visual signals reflect reality and which reflect imagination. One recent hypothesis is that the feeling of ‘realness’ is determined by whether or not the superficial layers of the EVC are activated (Bergmann et al., 2024). Initial results show that during mental imagery, only deep layers of EVC are involved. In contrast, superficial layers do get activated during seemingly ‘real’ illusory perception (Bergmann et al., 2024). Another, not mutually exclusive, hypothesis is that whether EVC signals are considered as real or not depends on whether they are strong enough to cross a ‘reality threshold’ (Dijkstra & Fleming, 2023). Further elucidating how the brain keeps apart EVC signals of imagery and perception has important implications for our understanding of when this line breaks down in hallucinations.

To conclude, developments in both experimental design and analytical approaches are converging on the idea that the EVC is recruited when low-level visual details are imagined. Future research is needed to further characterize exactly how activity in EVC is instantiated during imagery and how this activity leads to the unique conscious experience characteristic of our imagination. Answering these remaining questions will further increase our fundamental understanding of the neural correlates of consciousness as well as provide novel perspectives on understanding disorders of mental imagery.