Visual Maze Solving by Multimodal Large Language Models

1. Introduction

The ability of multimodal versions of large language models (LLMs), and their counterparts specifically designed to integrate vision and language (large vision-language models or LVLMs), to reason about spatial relations in images has been examined in a number of recent studies [1,2,3]. Xu et al. [4] developed a comprehensive benchmark involving visual perception, spatial reasoning, and commonsense visual knowledge, but they restricted evaluation to open-source LVLMs and their different configurations. The most recent visuospatial benchmark is OmniSpatial, which covers four major domains: dynamic reasoning, complex spatial logic, spatial interaction, and perspective-taking [5]. Several proprietary LLMs and open-source LVLMs were tested on OmniSpatial. Results showed that proprietary models exhibited strong performance in dynamic reasoning and spatial interaction tasks whereas open-source models lagged behind. By contrast, the performance of all tested models was near chance level in complex spatial logic and perspective-taking tasks. Jia et al. [5] interpreted these findings as evidence that proprietary models have extensive world knowledge and strong reasoning capabilities that enable them to achieve high test scores in motion comprehension, spatial relations analysis, and map-based comprehension.

Creating large benchmarks by scraping samples from the internet and using aggregate measures such as accuracy scores on particular tests may overestimate the real-world capabilities of LLMs [6,7]. Benchmark studies are often prone to data contamination, approximate retrieval, and exploitation of spurious correlations. An alternative strategy is to provide more in-depth analyses by testing for model consistency, robustness, and generalization [7]. Here, we focus on LLMs’ map-based comprehension and investigate how they find a path on a map. Specifically, a visual maze serves as the map. The maze-solving task is a type of navigational task that requires several cognitive components, including planning, identifying one’s current location in allocentric coordinates, and updating that location as one mentally moves along the maze [8]. Wang et al. [2] developed a Maze-Nav test as a component of the larger SpatialEval benchmark. The same test has also been incorporated into a spatial reasoning benchmark [9]. The Maze-Nav test consists of a very simple maze with only a few turns. Moreover, the correct path is already labeled in a different color from irrelevant corridors. The task is to provide a verbal description of the maze’s properties by answering questions such as “How many total turns are there in the provided path from the start to the exit? A) 4; B) 1; C) 9; D) 5.”

The multiple-choice questions used in the Maze-Nav test are examples of visual question answering (VQA), which is commonly used in many benchmarks of visuospatial reasoning. There are also other forms of answers, such as open-ended questions and image descriptions, but the key point is that in all cases the modality of the answer is purely verbal [4,5]. However, such outputs do not fully capture the complexities of visuospatial reasoning. Consider, for example, the problem of finding a path from location A to location B on a map. It is not enough to state that such a path exists; it is even more important to show the path on the map. The intelligent system needs to be able to take one visual representation of space and act upon it – that is, transform it into another spatial representation – in order to satisfy the stated goal, based on a set of rules that define which moves are allowable.

To address this limitation, we created a new test called VMaze-20, consisting of 20 visual mazes with labeled starting points and exits. In half of the mazes there is a free path from the start to the exit, and in the other half the path is blocked by walls. In each trial, the model receives a verbal prompt along with an image of the maze. The model’s task is to produce a new image of the same maze showing the path from the starting point to the exit, under the constraint that the path must follow the corridors (white spaces) and avoid obstacles, that is, walls (black lines). In addition, the model is required to explicitly state whether a path from the starting point to the exit exists or not. Thus, the model receives combined visual and verbal input and must provide a combined visual and verbal answer. Open-source models were unable to follow this protocol, so we restricted our attention to proprietary models similar to those studied by Jia et al. [5].

2. Methods

There are many maze generators available on the Internet, and we chose one that best suited our purpose (https://mazesforfun.com/maze-generator). The maze images ranged in size from 544 × 544 to 801 × 801 pixels. We generated 10 mazes in which the start and exit points were positioned at opposite corners of the maze (corner mazes) and 10 mazes in which the start and exit points were positioned on opposite sides of the midline vertical axis (midline mazes). The starting point was labeled with a red X and the exit point with a green Y. To examine whether LLMs can distinguish between a maze that has a valid path from the start to the exit and one that does not, we manually blocked the exits for 5 corner mazes and 5 midline mazes. Thus, we created four types of mazes: 5 corner mazes with an existing path from the start to the exit, 5 corner mazes without a path from the start to the exit, 5 midline mazes with a path, and 5 midline mazes without a path between the start and the exit (see Figure 1 for examples).

To examine whether the language of the prompt plays any role in solving the task, we decided to test LLMs using prompts in two different languages. First, we used prompts in English, because English encompasses almost 50% of all web pages on the internet, making it the largest training corpus available for LLMs. For comparison, we also used prompts in Croatian, which accounts for about 0.2% of web pages on the internet, so it represents a relatively small training corpus [10]. The prompt in English was as follows:

„The image I will show you contains a schematic depiction of a maze or labyrinth. The maze consists of black lines, which represent walls through which you cannot pass, and white surfaces, which represent corridors thought which you can move. Therefore, in the maze, allowable moves are thought white surfaces and it is forbidden to cross through the black lines. In this particular maze, there are also two letters: a red letter X and a green letter Y. Your task is to find the path or route from the letter X to letter Y and to draw the path on the image by using a blue line. The path must follow the rules described above. Sometimes it will not be possible to reach point Y from the point X because the path is blocked by walls (black lines). In that case, explicitly state that there is no valid path from point X to point Y. When creating the path, do not alter the appearance of the maze, that is, you should not move walls (black lines). At the end, explicitly state whether a possible route from letter X to letter Y exists or not, and estimate the degree in percentages of how sure you are that you successfully solved the task.”

The prompt in Croatian was a direct translation of the quoted text. We tested ChatGPT-4o, ChatGPT-5.2, Gemini-3, and Grok. Testing started on January 3 and ended on January 16, 2026. To assess the consistency of answers, each of the 20 mazes was presented 5 times so that each model was tested 100 times per language. The presentation was organized into 5 blocks of trials, and the order of maze presentation was randomized across blocks. Once testing was completed in one language, the procedure was repeated in the other language. Tests were run with the memory setting turned off and the reasoning ability set to the highest level. No feedback was provided about answer accuracy. Following the advice of [6], we provided open access to all answers from all tested models via the Open Science Framework, available at https://osf.io/vh7pk/overview.

3. Results

Answers were analyzed manually by visual inspection. There were two types of correct answers: either the LLM found the correct path by following the rules without making any errors, or it explicitly stated that there was no valid path from X to Y if the rules were followed. With the exception of ChatGPT-5.2, LLMs were generally unable to draw a valid path from X to Y. Contrary to what [5] reported about strong map-based comprehension in proprietary models, Grok and ChatGPT-4o failed to produce any correct path (when one existed) or to explicitly declare that a valid path did not exist. The accuracy of Gemini-3 was 3% with an English prompt, and 2% with a Croatian prompt. An example of its rare correct answer is shown in Figure 2a. By contrast, ChatGPT-5.2 achieved 99% accuracy with an English prompt, and 100% accuracy with a Croatian prompt. It made one omission with the English prompt, but this was likely due to a technical failure rather than a genuine inability to provide the answer. However, its answers were unusual, as paths were always adjacent to the walls, suggesting that it searched for a shortest path even though this was not required by the prompt (Figure 2b). When providing the answer, ChatGPT-5.2 explicitly reported that it employed the breadth-first search algorithm. It treated the maze as a graph with four nearest-neighbor links between white pixels (corridors), whereas black pixels were removed from the graph by thresholding. Sometimes it also reported using the A* search algorithm to find a path on a graph. It seems that ChatGPT-5.2 is not a pure LLM, but rather is augmented with symbolic knowledge about classical algorithms from computer science.

Although it is correct to represent a maze as a graph and apply pathfinding algorithms, this approach is inefficient for solving a visual maze. Such a representation requires traversing a large number of nodes, which consumes substantial computing time. Humans solve this task quite differently: they use object-based attention to mentally trace different paths until they find one that connects the starting point to the exit [11,12]. In addition, humans are more likely to move their attention along the center of the corridors, much like the output produced by Gemini-3. Unfortunately, Gemini-3 was inconsistent because its correct answer appeared only once per maze across several different mazes.

3.1. Error Analysis

Since LLMs committed so many errors in drawing paths, it is instructive to analyze them in more detail. To this end, we sorted errors into five distinct categories:

1)

visual hallucination,

2)

failure to connect X and Y,

3)

crossing the wall,

4)

walking on the wall,

5)

failure to provide an answer.

Visual hallucination occurs when the model returns an image of the maze that differs from the one provided in the prompt. An example is shown in Figure 3a, which is clearly a hallucination: the output is a rectangular maze where Y is not positioned in the top-left corner, whereas all input mazes were square and, in all corner mazes, Y was always positioned in the top-left corner. Visual hallucinations are a common problem observed in many LLMs [13]. Interestingly, our results suggest that not all models are equally prone to this type of error. Failure to connect X and Y refers to situations in which the model produces a path that does not start near X or end near Y. We used a city-block metric by dividing the maze into a 10 × 10 grid and checking whether the path starts in the same block as X and ends in the same block as Y. In the example in Figure 3b, the model seems to work backward, starting from Y but then fails to turn to X. Other examples of this error involve loops that do not pass through Y. Yet another example is shown in Figure 3c, where the path misses both X and Y. Figure 3c also illustrates crossing the wall, an error in which the model ignores the instruction to avoid black lines, which represent walls. Similarly, walking on the walls occurs when the path is drawn along the black lines (Figure 3d). Sometimes models fail to provide an answer; we also count this as an error, although it occurs rarely.

If an LLM made one or more of the described errors, the answer was classified as incorrect, even if the model verbally asserted that there is a valid path from X to Y. Each error category was counted independently. This means that an LLM could make multiple errors within the same trial, which occurred frequently, as shown in Figure 3c and Figure 3d. Table 1 reports the percentages of errors committed by all tested LLMs. It should also be noted that, in computing these percentages, the total number of trials was adjusted for each model, depending on whether it provided two answers per trial or failed to provide an answer.

In many trials, Grok produced two answers instead of one; that is, it returned two images with different paths drawn on the maze (95% of trials with the English prompt and 91% of trials with the Croatian prompt), although both answers were incorrect. Sometimes Grok failed to connect X and Y, but it was especially prone to crossing walls or walking on walls. In several trials, Gemini-3 refused to provide an answer (9% with the English prompt, and 2% with the Croatian prompt). It also produced one answer with a black screen and one answer without a path on the maze (both times with the Croatian prompt). Of all models tested, Gemini-3 showed the strongest tendency to hallucinate an answer (57% with the English prompt and 60% with the Croatian prompt). On the other hand, it showed less tendency to fail to connect X and Y. Interestingly, it never walked on walls, regardless of the language. It should also be noted that there were trials in which we had to repeat the prompt multiple times in order to obtain an answer. In some trials, Gemini-3 also provided verbal instructions on how to navigate through the maze, but these were also incorrect.

In three trials, ChatGPT-4o provided two answers instead of one. In one trial, the model refused to provide an answer. ChatGPT-4o hallucinated much more often when the prompt was in English than when it was in Croatian. It seems that a larger training corpus in English encourages ChatGPT-4o to produce more hallucinations. A similar although less pronounced tendency is also seen in Grok. Regardless of the language, ChatGPT-4o rarely failed to connect X and Y but, like Grok, showed a strong tendency to cross the walls and walk on the walls, suggesting that it lacks a proper internal representation of what is forbidden in the maze.

3.2. Metacognitive Judgments

Metacognition is the ability to monitor and evaluate one’s own performance. It is central to human intelligence, but how closely LLMs align their metacognitive judgments with their actual performance is still an open question [14,15]. We addressed this question in our task by asking LLMs to explicitly rate how confident they were that they had provided the correct answer on a scale from 50% to 100%. Interestingly, some models refused to provide metacognitive judgments and were therefore probed with additional prompts. For example, Grok provided metacognitive judgments on only three trials, and in all three cases it was 100% certain that the answer was correct. Similarly, Gemini-3 gave metacognitive judgments in fewer than 50% of the trials, and whenever it did, it was always 100% certain that the answer was correct.

ChatGPT-4o has often blended visual answers with metacognitive judgments. In 12 trials, it failed to provide a metacognitive judgment (of that 9 with the English prompt). In the remaining trials, its dominant response was that it was 100% certain the answer was correct, and in a smaller fraction of trials it reported 99% certainty. ChatGPT-5.2 provided metacognitive judgments in all trials, ranging from 93% to 99%. Thus, the only model that correctly solved the mazes was less confident in its performance than other models that grossly misjudged their abilities. This suggests that LLMs do not have an appropriate internal representation of their visuospatial abilities and instead rely on guessing when forced to provide metacognitive judgments.

4. Conclusions

We identified an important limitation in current practices for evaluating the performance of LLMs on visuospatial reasoning tasks. Instead of using the typical visual question-answering format, we first asked the models to determine whether it is possible to find a path in a provided image of a visual maze and then, if such a path exists, to generate a new image showing the path drawn on the maze. Grok, Gemini-3, and ChatGPT-4o were unable to do this and made a number of errors that are revealing about differences in their internal workings. Only ChatGPT-5.2 consistently produced correct answers, but it did so by relying on classical computer science algorithms, such as breadth-first search for a path on a graph.