BMRS: Bongard–Maximov Problems for Remote Sensing

1. Introduction

Timely and reliable information from remote sensing (RS) is essential for addressing environmental challenges and supporting evidence-based natural resource management. Remote sensing data are widely used in environmental monitoring [1], agriculture [2], land-cover mapping [3], urban analysis [4], disaster assessment [5], and natural resource inventory [6]. The volume and diversity of available remote sensing imagery (RSI) continue to increase due to advances in imaging technologies and the growing availability of compact satellite platforms [7,8]. At the same time, improvements in image-processing methods further enhance the quality and usability of RSI [9,10,11]. As a result, manual interpretation becomes increasingly difficult to scale, creating a growing demand for automated approaches to high-level semantic analysis of RS data. Consequently, vision-language models (VLMs) are routinely applied to RSI. By aligning visual and textual modalities within a shared representation space, VLMs can interpret image content and express their understanding in natural language, enabling tasks that require both visual understanding and language-based reasoning. However, general-purpose VLMs are predominantly trained on human-perspective (ground-level) photographs and often perform poorly on RS-specific applications [12]. This domain gap is largely explained by the distinctive properties of RSI, including large spatial extents, pronounced scale variation, and sparse or subtle visual cues [13]. Consequently, current research focuses on remote sensing vision-language models (RSVLMs), i.e., VLMs adapted specifically to RS data.

RSVLMs can be applied to established remote sensing tasks, such as scene classification, object detection, semantic segmentation, and change detection. Beyond these traditional computer vision tasks, modern conversational VLMs support a broad range of vision-language applications, including image captioning, cross-modal retrieval, and visual question answering (VQA), i.e., answering free-form natural-language questions about an input image. Among these tasks, VQA is particularly challenging because success depends not only on recognizing individual objects but also on understanding their relationships and broader scene context [12]. Although several remote sensing VQA benchmarks have been proposed [14,15], recent surveys emphasize that the range of available benchmarks for evaluating RSVLM reasoning capabilities remains limited [16].

Benchmarking is more developed in general-purpose VLMs than in RSVLMs. Existing benchmarks assess sensitivity to visual illusions [17], logical learning [18], mathematical reasoning [19], and analogy-based visual reasoning [20]. Among the most challenging evaluation paradigms are Bongard problems (BPs) [21]. A BP presents a Bongard matrix containing two sets of images, arranged on the left and right sides according to an underlying rule or concept that must be inferred and expressed in natural language (Figure 1). BPs are typically designed so that human observers can solve them within a few minutes. BPs are particularly appealing because they capture several fundamental aspects of human cognition [22,23]. First, they require few-shot concept learning, where a visual concept must be inferred from only a small number of right and left images. Second, they involve context-dependent reasoning, since the interpretation of an image depends on the other images present in the Bongard matrix. Third, they require analogy-making perception, in which structural relationships rather than individual visual features must be recognized and generalized. Originally proposed by Mikhail Bongard and colleagues in 1967 [24] and later popularized through the work of Hofstadter [25], BPs remain a concise, interpretable, and challenging benchmark for abstract visual reasoning.

To the best of our knowledge, no Bongard-problem benchmark has yet been developed for remote sensing imagery. To address this gap, we introduce the first remote sensing benchmark based on the Bongard problem paradigm. To acknowledge the contributions of Vadim Maximov to the development of the original Bongard problems, we name our benchmark BMRS: Bongard–Maximov Problems for Remote Sensing (see Appendix C).

The proposed benchmark enables systematic evaluation of both general-purpose VLMs and RSVLMs on reasoning capabilities that extend beyond domain-specific visual perception. The main contributions of this work are threefold. First, we introduce BMRS, the first remote-sensing benchmark based on the Bongard problem paradigm. The benchmark comprises 122 problems constructed from aerial and satellite imagery and is designed to evaluate few-shot concept learning, context-dependent reasoning, analogy making, and relational reasoning. Second, we establish a human baseline through a study involving 113 participants and evaluate a diverse set of proprietary general-purpose VLMs, open-source general-purpose VLMs, and specialized remote-sensing VLMs. Third, we provide a detailed analysis of model performance across multiple categories of Bongard problems, identifying strengths and limitations of current remote-sensing model adaptation strategies and highlighting key directions for future research.

2. Related Work

2.1. Visual Language Models

Vision-language models can be broadly categorized into contrastive and conversational models [12]. Contrastive VLMs, such as CLIP [26], learn a shared embedding space for images and text through cross-modal alignment. While effective for representation learning and retrieval, they cannot directly generate natural-language responses and are not designed for instruction-following or complex reasoning tasks [16].

Conversational VLMs, such as LLaVA [27], typically consist of three components: a visual encoder that extracts semantic information from images, a connector module that maps visual features into the language-model embedding space, and a large language model (LLM) that processes multimodal inputs and generates natural-language responses. Through multimodal alignment and instruction tuning, these models learn to connect visual content with language and follow user instructions. Unlike contrastive VLMs, conversational VLMs can explain their predictions and perform reasoning in natural language, making them suitable for tasks that require abstract visual inference, such as Bongard problems.

Despite the rapid progress of general-purpose VLMs, their performance on remote sensing tasks is often limited by the substantial domain gap between natural imagery and remote sensing imagery (RSI) [16]. Differences in viewpoint, spatial scale, scene composition, and semantic structure make RSI fundamentally different from the data on which most generic VLMs are trained. To address this challenge, recent research has focused on remote sensing vision-language models (RSVLMs), which are adapted to remote sensing data through domain-specific training.

However, the relatively limited size and diversity of remote sensing image-text datasets make training large multimodal models from scratch impractical. Consequently, most RSVLMs inherit architectures and pre-trained weights from successful general-purpose VLMs and are subsequently adapted using remote sensing datasets [16]. Many open-source RSVLMs are based on LLaVA-1.5, including VHM [28], GeoChat [29], RS-LLaVA [30], and SkySenseGPT [31]. LHRS-bot [32] shares only the visual encoder of LLaVA-1.5, while RSGPT [33] shares only its LLM component. More recent models explore alternative architectures, although relatively few remain open source, such as TeoChat [34] and EarthDial [35].

Benchmark results indicate that domain adaptation can substantially improve performance on remote sensing tasks. On GeoBench, RSVLMs based on LLaVA-1.5 often outperform their generic counterpart and, in some cases, surpass more advanced general-purpose VLMs. Nevertheless, RSVLMs continue to lag behind generic VLMs on tasks requiring complex reasoning, as evidenced by results on the CHOICE benchmark [36]. This suggests that current adaptation strategies effectively improve domain-specific perception but do not fully close the reasoning gap.

Recent work has therefore explored reinforcement learning (RL) as a means of enhancing reasoning capabilities in RSVLMs [37]. Examples include GeoVLM-R1 [37], which demonstrates substantial gains in visual reasoning, ViLaCD-R1 [38] for semantic change detection, and the RS-EoT framework [13], which integrates iterative reasoning with visual inspection. As RL-based training continues to advance, benchmark datasets must evolve accordingly to provide challenging and reliable evaluations of RSVLM capabilities.

2.2. RSVLM Benchmarks

The dominant evaluation paradigm for RSVLMs is Remote Sensing Visual Question Answering (RSVQA), in which a model is provided with a remote sensing image and a natural-language question and is expected to produce a short textual answer. Early RSVQA benchmarks were fragmented and limited in scope: most datasets focused on isolated capabilities, lacked support for multi-temporal or context-aware reasoning, and relied on rigid exact-match evaluation protocols [36]. To address these limitations, the CHOICE benchmark was introduced as a structured and hierarchical evaluation framework for RSVLMs. It organizes model capabilities into two primary dimensions – perception and reasoning – which are further decomposed into six sub-dimensions and 23 sub-tasks, enabling a more comprehensive assessment of model performance.

As RSVLM research has progressed, benchmarks have become increasingly challenging, often by adapting evaluation settings from general vision-language benchmarks to remote sensing. These include, for example, mathematical reasoning and multi-image understanding benchmarks. GeoMath [39] evaluates mathematical reasoning across geometry, logic, statistics, arithmetic, counting, and algebra. Multi-image benchmarks are designed to assess the ability to model relationships, analogies, and fine-grained visual differences that cannot be inferred from a single image. In remote sensing, representative examples include GeoBench [40] and VLRS-Bench [41], which use multiple images to evaluate tasks such as change detection.

Bongard problems also belong to this category of multi-image benchmarks. They are specifically designed to evaluate abstract visual reasoning and pattern induction from sets, requiring generalization across multiple images. However, to the best of our knowledge, Bongard-style benchmarks have not yet been adapted to the remote sensing domain.

2.3. Bongard Benchmarks

Several Bongard-based benchmarks have been introduced over time, including some that predate the emergence of VLMs. One of the earliest is Bongard-LOGO [22] benchmark, introduced in 2020. The authors proposed an automatic BP generator, thereby greatly expanding the original set of BPs. Bongard-HOI [23] later transferred the paradigm to real-world images replacing, however, the original geometric concepts with human-object interactions. The more recent Bongard-OpenWorld benchmark [42] extended this paradigm with diverse real world concepts. Developed in the era of VLMs and LLMs, it has showed that large commercial neural networks can perform slightly better than humans, although most modern neural models still perform substantially below average human levels.

Subsequent work [43] hypothesized that VLMs poor performance on the original BPs is caused by a domain gap, because models were trained on photographs, but evaluated on abstract drawings. To test this hypothesis, the authors introduced Bongard-RWR, in which the concepts of the original Bongard problems were recreated using photographs; this translation was achieved for 60 of the 100 original BPs. In a follow-up study, the authors proposed an AI-based augmentation method and applied it to Bongard-RWR, resulting in the substantially larger Bongard-RWR+ benchmark [44]. This augmentation included generation of image descriptions based on the original Bongard-RWR dataset, text-to-text paraphrasing, text-to-image rendering, manual quality filtering, and diversity-aware subset composition to create new problem instances.

Nevertheless, verifying open-form answers becomes difficult on large dataset, and the tasks remain challenging for models. Therefore, several simpler task formats have been proposed, typically involving selection of the correct answer from a set of options. Although models achieve higher results on domain-adopted Bongard tasks than on the original Bongard problems, these analogues of the original problems still prove difficult for current models across all task formats.

3. Materials and Methods

3.1. Remote Sensing Bongard Benchmark

Various Bongard-related task formats exist in the literature, whereas Bongard himself originally proposed two formats: free-form concept formulation and image-to-side classification [24]. We use the free-form concept formulation — identifying the rule that distinguishes the two sides — as the most challenging format, since it prevents cues and guessing.

The original set of problems included only geometric problems, and Bongard required them to satisfy the following conditions [24]:

1.

the problems include black-and-white images without halftones (i.e., line drawings);

2.

the information contained in the images themselves is sufficient to solve the problem (analytic reasoning);

3.

the problems are solvable by a human observer.

Relaxing the first requirement is the central idea of the present work: instead of line drawings, we consider real photographs. Similar extensions have been explored in [42,43], but, to the best of our knowledge, this is the first Bongard benchmark based on RSI. The second requirement has also been relaxed in recent work [23,42]. This change is motivated by the emergence of VLMs, which are trained on large-scale datasets and are expected to combine visual perception with the common sense and preconceived notions of a human observer. Accordingly, in addition to problems that require purely analytic reasoning, our benchmark also includes problems that require broader semantic or knowledge-based (synthetic) reasoning.

We consider the third requirement essential and therefore preserve it. In his book, Bongard gave an example of a formally valid but excessively difficult problem in which the difference between the total number of angles and the total number of shapes on each side serves as an organizing rule. To verify that our problems remain solvable for humans, we conducted human studies (Section 3.4).

The remote sensing images were selected to cover a wide range of acquisition heights and spatial resolutions, including imagery captured from satellites, aircraft, and unmanned aerial vehicles. Most images were acquired in nadir view, whereas a smaller subset was captured at an oblique angle. All images are RGB. The data sources include Google Earth and the VisDrone [45], AID [46], DOTA [47], MLRSNet [48], RRealHyperPDID [11], and WAID [49] datasets. The resulting benchmark contains 122 Bongard problems. Table 1 compares BMRS with previously published Bongard benchmarks.

3.1.1. Collection Methods

We employed two complementary collection strategies. First, most problems were constructed manually. For each problem, the underlying concept was defined by the authors, and then suitable images were selected through visual inspection. Some of these problems were designed as RS analogues of original Bongard tasks (Figure 2).

Second, several categories of problems were assembled semi-automatically using object annotations. This approach was used for presence/absence tasks, class-contrast tasks (e.g., vehicles versus boats), counting tasks, relative-position tasks within an image crop (e.g., upper versus lower image regions), and size-based tasks (e.g., large versus small objects). Examples of semi-automatically assembled problems are shown in Figure 3.

3.1.2. Classification of Problems

To categorize BMRS problems, we adopted the taxonomy of original Bongard problems proposed by [21]. In this taxonomy, each problem is assigned to one of five mutually exclusive rule categories:

1.

Size: the discriminative rule is based on object size;

2.

Number: the discriminative rule is based on object count;

3.

Spatial: the discriminative rule is based on the spatial arrangement of objects;

4.

Same: objects within each image are either identical or non-identical with respect to a shared property;

5.

Concept: all remaining cases.

We found this taxonomy largely applicable to RS, with one modification. Because the Concept category encompasses a broad range of rules, we further subdivided it into Shape, Semantic, and Presence subclasses:

• Shape: the discriminative rule depends on shape-related properties irrespective of object semantics. For example, the BMRS analogue of original problem 97 [24], p.246 distinguishes rectangular from circular shapes (Figure 2).
• Semantic: the discriminative rule is defined by object identity or category rather than by visual properties alone. For example, the left problem in Figure 3 contrasts watercraft and aircraft.
• Presence: the discriminative rule is based on the presence or absence of a particular object or object category. For example, the right task in Figure 3 contrasts images containing houses with images containing trees.

The class distributions of BMRS and the original Bongard dataset are compared in Table 2.

3.2. Vision–Language Models

The evaluated models were selected to cover three complementary categories: open-source general-purpose VLMs, open-source RSVLMs, and state-of-the-art proprietary VLMs. Model selection was guided by architectural diversity, availability of model weights and/or source code, feasibility of local deployment, prevalence in recent literature, and relevance to complex visual reasoning tasks.

Among general-purpose VLMs, we included LLaVA-1.5-7B [27] as a representative baseline because many RSVLMs are derived from its architecture and pre-trained weights. We additionally evaluated LLaVA-1.5-13B to assess the effect of parameter scaling within the same model family. To further analyze the impact of architectural improvements while keeping the model scale comparable, we included LLaVA-1.6-7B as an updated version of the LLaVA family. Finally, we evaluated LLaVA-1.6-34B to jointly assess the effect of architectural changes and substantially increased model scale. We further included the medium-scale Qwen-3-VL-32B-Instruct and the larger InternVL-3.5-38B, two recent open-source VLMs with strong multimodal capabilities and competitive performance across a wide range of general vision-language tasks, since models of these families are frequently used in other Bongard benchmarks [21,43,44].

The RSVLM group consisted of VHM [28], RS-LLaVA [30], RS-EoT [13], GeoChat [29], SkySenseGPT [31], EarthDial [35], LHRS-Bot [32] and TeoChat [34]. These models were specifically developed for remote sensing imagery and collectively represent several generations of RSVLM development, ranging from domain-adapted LLaVA variants to more recent architectures incorporating explicit reasoning mechanisms.

To provide a reference based on current proprietary systems, we additionally evaluated Gemini 3.1 Pro Preview and ChatGPT 5.5 Pro. Although their architectures and parameter counts are not publicly disclosed, these models represent the current state of the art in proprietary multimodal AI systems and substantially exceed the scale of most open-source models considered in this study.

All models were evaluated on BMRS using a unified experimental protocol. Open-source general-purpose VLMs were used in their original form, enabling an assessment of their out-of-the-box performance on remote sensing tasks. RSVLMs were evaluated using their publicly released checkpoints without fine-tuning. Proprietary models were accessed exclusively through their respective APIs and were evaluated in inference-only mode. Table 3 summarizes the evaluated models.

Inference of RSVLMs was performed locally. VHM, RS-LLaVA, and RS-EoT were evaluated on a system equipped with an Intel Xeon W-2133 CPU, 128 GB of RAM, and an NVIDIA Quadro GV100 GPU (32 GB VRAM). GeoChat, SkySenseGPT, EarthDial, LHRS-Bot and TeoChat were evaluated on a second system equipped with an Intel Xeon Platinum 8468, NVIDIA H100 GPU (80 GB VRAM) and 2 TB of RAM.

Open-source general-purpose models were evaluated on a computing cluster managed via the Slurm workload manager. All experiments were executed on a single compute node comprising 96 logical CPU cores Intel Xeon Platinum 8462Y and six NVIDIA A100 SXM4 GPUs (40 GB VRAM each), with a total system memory of approximately 1008 GB.

3.3. Prompting Strategies

Prompting strategy has a substantial impact on the performance of large language models across modalities [50]. Moreover, explicit decomposition of a task into subproblems is often critical for strong performance [51,52]. Similar findings have been reported for VLMs, where multi-stage prompting improves multi-step and spatial reasoning compared with single-query approaches and simple chains of thought formats [53].

To evaluate both end-to-end and decomposition-based reasoning in Bongard problems, we consider the prompting strategies of [43]: Direct, Descriptive, Descriptive-direct, Descriptive-iterative, Contrastive, Contrastive-direct, Contrastive-iterative. The Direct strategy presents the full Bongard matrix to directly predict the underlying rule.

The Descriptive family first generates image-level descriptions to infer the rule from these representations. Variants differ in how descriptions are produced and aggregated. Descriptive processes images independently. Descriptive-direct provides the full matrix during final inference. Descriptive-iterative incrementally refines a shared side description. Descriptive family encourages reasoning over textual abstractions rather than raw visual input.

The Contrastive family instead operates on pairwise image comparisons between left and right sides to derive a general rule. Contrastive analyzes each pair independently with final aggregation. Contrastive-direct provides the full matrix during final reasoning. Contrastive-iterative considers pairs sequentially while accumulating context. Contrastive family explicitly encourages relational reasoning between sides.

We implemented five of the strategies described above: Direct, Descriptive-iterative, Descriptive-direct, Contrastive-iterative, Contrastive-direct. The original Descriptive and Contrastive strategies were omitted because most RSVLMs do not support text-only interaction without image inputs. All of the implemented strategies were used with prompts provided in Bongard-RWR [43] with little changes. Modified prompts are listed in the Appendix A.

Besides implemented strategies we introduced the following Shuffle modification. Because iterative prompting strategies may be sensitive to the order in which images are presented, we evaluated this effect using TeoChat by running ten trials with randomly shuffled image orders. The influence of model stochasticity was excluded in this experiment, as TeoChat produced character-identical outputs when the image order was fixed. However, some evaluated models exhibited stochastic behavior; therefore, multiple runs were conducted for highly variable models.

3.4. Human Study Design

To assess the solvability of the developed problems, we evaluated human performance on the resulting dataset. The study involved 113 participants recruited from the academic community. All participants were native Russian speakers, and the study was conducted in Russian.

In a single session, each participant was asked to solve up to 20 problems randomly sampled from the dataset and ordered by difficulty. Problem difficulty was estimated during dataset development using a two-stage procedure. First, all problems were independently cross-reviewed by the authors and assigned to coarse difficulty categories based on the perceived complexity of the underlying concept and the expected effort required to identify it. Second, a pilot study involving laboratory staff who were not involved in the problem development was conducted to assess problem clarity and verify the proposed difficulty ordering. Based on participant feedback and observed solution patterns, ambiguous problems were revised or removed, and the final difficulty ranking used in the human study was established. Problem sets were constructed to preserve the class distribution of the full dataset (Table 2) while maintaining a comparable average difficulty across participants. At least 14 participants evaluated each problem.

We adopted the classical open-ended Bongard problem format: participants were presented with six left and six right images arranged as a matrix, together with a text field for entering their answer. After one minute, participants were given the option to select “I don’t know”. Neither the total session duration nor the time allotted to individual problems was restricted.

Before the study began, participants received written instructions and a worked example based on the original Bongard problem, including several acceptable answer formulations. For each problem, participants were asked to provide a detailed explanation of the distinguishing concept, describing both the left and right sides.

3.5. Answers Evaluation

To evaluate the corpus of more than 10 000 human and model-generated responses, we adopted the LLM-as-a-judge framework [21,54], using DeepSeek-v4-flash in “Think Max” mode [55] as the automated evaluator. This model was selected for three reasons: DeepSeek-family models were previously used as judges in Bongard-problem research [43]; DeepSeek-v4-flash represents one of the strongest currently available lightweight reasoning models; and no DeepSeek models were included among the evaluated BMRS solvers, eliminating potential bias arising from evaluating a model family with itself.

To establish a reliable ground truth, three experts manually graded all human responses and a representative subset of model outputs. The evaluation protocol consisted of both general assessment criteria and class-specific guidelines covering all seven BMRS categories (Appendix B). The resulting expert-annotated corpus was divided into three subsets: few-shot examples for prompt construction, a validation set, and a test set. To prevent data leakage, the validation and test sets contained disjoint subsets of Bongard problems.

The validation set was used to iteratively refine both the system prompt and class-specific prompts (Appendix B) until the judge achieved 100% agreement with expert annotations. The final evaluation on the held-out test set yielded an accuracy of 93%, computed as $Ac{c}_j={\displaystyle \frac{T_P+T_N}{T_A}}$, where $T_P$ and $T_N$ denote the numbers of true-positive and true-negative evaluations, respectively, and $T_A$ is the total number of evaluated responses.

4. Results

Performance was measured using accuracy, defined as the proportion of correct answers. Overall accuracy was computed as $Ac{c}_a={\displaystyle \frac{C_a}{T_a}}$, where $C_a$ is the number of correct answers and $T_a$ is the total number of answers. For individual problem categories, we report class-specific accuracy, $Ac{c}_t={\displaystyle \frac{C_t}{T_t}}$, where $C_t$ is the number of correct answers within a given category and $T_t$ is the total number of answers for that category.

4.1. Human Study Results

A total of 113 participants completed all assigned BMRS problems: to reduce fatigue, each participant was presented with a subset comprising up to 20 problems rather than the full benchmark. The mean participant age was 25.1 years, and the average completion time was 29.6 minutes.

Across 2 255 collected solutions, the overall human accuracy was 74.5%. Performance by problem category is summarized in Table 4. Participants achieved the highest accuracy on Semantic problems (85.4%), followed by Number (81.8%), Shape (79.7%), and Size (78.9%) problems. The lowest accuracy was observed for Spatial problems (53.3%), with Same problems also proving relatively challenging (57.7%). The most difficult individual BMRS problems were predominantly in the Spatial category. In particular, problems where a discriminative rule depended on object location within the image (e.g., objects appearing in the upper versus lower part of the scene) exhibited the lowest solution rates. Overall, the results indicate that humans reliably solve object- and concept-based BMRS problems, whereas spatial-relational reasoning is substantially more challenging. Note that we consider the benchmark to satisfy the solvability criterion, as every problem was solved correctly by at least one participant. Detailed results, including examples of the most challenging problems, are provided in Appendix D.

4.2. Prompting Strategies Evaluation

The performance of different prompting strategies is summarized in Table 5. Proprietary General-Large models were excluded from this analysis because preliminary experiments showed that explicit decomposition of Bongard problems provided little benefit compared with direct prompting.

For open-source general-purpose VLMs, Descriptive strategies consistently outperformed Contrastive and Direct prompting. In particular, Descriptive-direct achieved the highest accuracy for all evaluated models except LLaVA-1.5-13B, for which Descriptive-iterative performed slightly better. These results suggest that generating intermediate image descriptions before concept induction is beneficial for Bongard problem solving.

In contrast, no single prompting strategy emerged as universally optimal for RSVLMs. LHRS-Bot and VHM achieved their best performance with Descriptive-direct, whereas RS-EoT, EarthDial, and SkySenseGPT performed best with Contrastive-iterative. GeoChat achieved its highest accuracy with Contrastive-direct, while TeoChat performed best with Direct prompting. This variability indicates that the effectiveness of problem decomposition depends strongly on the underlying model architecture and training procedure.

Overall, Descriptive prompting appears most effective for general-purpose VLMs, whereas RSVLMs exhibit less consistent behavior and derive varying benefits from decomposition-based strategies.

4.2.1. Sensitivity to Image Ordering (Shuffle) in Iterative Prompting

To evaluate the robustness of iterative prompting strategies, we investigated their sensitivity to the presentation order of the images within the Bongard matrix for each BMRS problem. Because iterative strategies incrementally build or refine rules based on sequential images, the specific order within the Bongard matrix can significantly influence the final inference. We evaluated this effect using the TeoChat model by performing ten independent trials per problem, randomly shuffling the image order within the left and right sides of the Bongard matrix for each trial. Since TeoChat produces character-identical outputs under fixed inputs, any variance in performance across these runs is entirely attributable to the permutation of the images rather than model stochasticity.

The results, visualized across all 10 shuffles for both descriptive-iterative and contrastive-iterative (Figure 4) strategies, reveal a stark disparity in performance across the BMRS taxonomy. Both strategies show the highest success rates within the Semantic category, indicating that the considered model is primarily capable of distinguishing rules based on object identity and categorical differences. Moderate success is observed in the Presence and Shape categories. Conversely, problems requiring abstract relational reasoning – specifically those categorized under Spatial, Number, and Same – remain almost entirely unsolved across all permutations, highlighting a significant limitation of the tested model.

A key finding from the shuffle experiments is the extreme fragility of the model’s reasoning paths. As shown in the frequency plots (see bottom panels in Figure 4), very few problems are consistently solved across all 10 shuffles. The vast majority of solved tasks are only answered correctly in a fraction of the permutations (often fewer than half). This indicates that the model does not demonstrate robust reasoning in neither Descriptive-iterative nor Contrastive-iterative case; rather, the model relies on lucky sequences of images that happen to align with its inductive biases during the iterative refinement process.

Interestingly, a comparative analysis of the two prompting strategies reveals that the specific problems solved by the Descriptive-iterative strategy and the Contrastive-iterative strategy frequently do not overlap. While both struggle with the same overarching categories (e.g., Spatial, Number), the individual problems they manage to solve within the Semantic or Presence categories differ. This suggests that describing images independently before inference (Descriptive) versus iteratively comparing them (Contrastive) triggers distinct reasoning pathways within the model. The lack of overlap implies that these prompting strategies are complementary, and a hybrid approach could potentially yield a higher overall solve rate on the BMRS benchmark.

4.3. Comparison of Solution Results Across Problem Classes

Table 6 and Figure 5 summarize the performance of humans and VLMs on BMRS. Detailed results are provided in Appendix F. For RS and General-Mid models, the best-performing prompting strategy was used.

Among all evaluated systems, proprietary large-scale VLMs (General-Large) achieved the highest performance. ChatGPT achieved the best overall accuracy (89.3%), closely followed by Gemini (86.9%). Both models exceeded human performance (74.5%), demonstrating higher accuracy across all problem categories. Spatial reasoning remains challenging for General-Large VLMs, although their performance remained considerably above the human baseline.

In contrast, open-source general-purpose VLMs (General-Mid) exhibited substantially lower performance. Qwen-3-VL achieved the strongest result within this group (42.6%), followed by InternVL-3.5 (40.2%). The LLaVA family performed considerably worse, demonstrating accuracy ranging from 13.1% to 20.5%. Despite their lower overall performance, Qwen-3-VL and InternVL-3.5 consistently outperformed all evaluated RSVLMs.

The performance of RSVLMs varied considerably across models. LHRS-Bot achieved the highest overall accuracy among remote-sensing-specific models (20.5%), exceeding General-Mid LLaVA-v1.5-7B, LLaVA-1.5-13B, LLaVA-1.6-7B and matching LLaVA-1.6-34B despite having substantially fewer parameters. RS-EoT obtained the second-best RSVLM result (17.2%), demonstrating competitive performance on Semantic, Shape, Presence, and Spatial tasks, but performing poorly on Number, Same, and Size problems.

Among RSVLMs derived from the LLaVA-1.5 architecture, VHM achieved performance comparable to the corresponding general-purpose LLaVA baseline, whereas GeoChat, RS-LLaVA, and SkySenseGPT performed substantially worse. This observation suggests that domain adaptation alone does not guarantee improved abstract reasoning performance.

Across both human and model evaluations, Semantic problems were consistently the easiest category. Several RSVLMs, including VHM, EarthDial, and RS-EoT, achieved their strongest results on Semantic tasks, while LHRS-Bot surpassed all open-source general-purpose models except Qwen-3-VL and InternVL-3.5 in this category. In contrast, Spatial problems remained challenging for both humans and models, highlighting spatial-relational reasoning as a key area for improvement.

5. Discussion

We introduced BMRS, the first remote-sensing benchmark based on the Bongard-problem paradigm. Human participants achieved an average accuracy of 74.5% solving BMRS, which is comparable to the performance reported in previous studies on Bongard-style reasoning benchmarks – 65% [43] and 86% [21]. This suggests that the overall difficulty of BMRS is broadly consistent with that of existing Bongard benchmarks while extending the paradigm to remote sensing imagery.

Direct comparison with prior work is challenging because model architectures and capabilities evolve rapidly. Nevertheless, one useful reference point is LLaVA-v1.6 7B, which was evaluated both in our study and in [43]. Using the Direct prompting strategy, LLaVA-v1.6 7B achieved 13% accuracy on BMRS, compared with reported accuracies of 0% on the original Bongard problems, 5% on Bongard-HOI, 12% on Bongard-OpenWorld, and 0% on Bongard-RWR. This suggests that the transition from line drawings to human-perspective photographs [43], and to remote sensing imagery is not the primary obstacle for current open-source VLMs. Instead, their main limitation appears to be the underlying reasoning task itself.

This interpretation is further supported by the category-level results. Both general-purpose VLMs and RSVLMs achieved their strongest performance on Semantic problems, which primarily require object recognition and category-level knowledge. Notably, the best RSVLMs approached or exceeded the performance of substantially larger general-purpose models on Semantic problems. In addition, the General-Mid Qwen-3-VL model achieved 84.6% accuracy on Presence problems, indicating that reliable recognition of remote-sensing objects is already possible even without remote-sensing-specific adaptation. Taken together, these results suggest that the domain gap between natural and remote-sensing imagery is not the primary limitation for current VLMs. In contrast, performance dropped markedly on Spatial problems, indicating that spatial-relational reasoning remains a major challenge even when object recognition is reliable. These findings suggest that current RSVLM adaptation strategies are more successful at transferring domain-specific perceptual knowledge than at developing abstract reasoning capabilities, such as concept induction, analogy making, and relational reasoning.

The performance of ChatGPT and Gemini demonstrates that Bongard-style reasoning is achievable at, and even beyond, human-level accuracy in modern frontier VLMs. It is worth noting that, judging by the responses, at least ChatGPT appears to be familiar with Bongard benchmarks: “It looks like there might be an interesting challenge here, possibly related to the "Bongard-HOI" task, involving satellite images like those from BigEarth”. This may indicate that Bongard benchmarks were used during training. Nevertheless, the high performance of these models is attributable to their reliance on computational resources that are far beyond those available for most practical remote-sensing applications. Consequently, an important research direction is the development of efficient domain-specialized models that retain strong reasoning abilities while remaining computationally tractable.

Future work will focus on expanding BMRS in both scale and diversity. First, additional remote-sensing analogues of original Bongard problems will be developed, potentially with the assistance of generative AI to overcome the difficulty of locating suitable images in real RS imagery. Second, we plan to investigate AI-based augmentation strategies similar to those proposed in [44]. Finally, we intend to study the effect of input representation, comparing traditional Bongard matrices with multi-image interfaces that allow models to process individual images separately. Such experiments may clarify whether limitations arise primarily from reasoning itself or from difficulties in parsing the visual structure of the Bongard matrix.

6. Conclusions

We introduced BMRS, the first remote sensing benchmark designed around the Bongard problem paradigm for evaluating vision-language models. Unlike conventional remote sensing benchmarks that primarily assess perception, BMRS evaluates the ability to perform multi-image, context-dependent reasoning, concept induction, and analogy-making. The benchmark contains 122 problems spanning seven categories: Shape, Semantic, Presence, Spatial, Size, Number, and Same. In contrast to classical Bongard problem collections, BMRS introduces a Semantic category that requires recognizing and reasoning about object identity and category information in remote sensing imagery.

A human study involving 113 participants established a reference accuracy of 74.5%. We evaluated three groups of models: proprietary large-scale general-purpose VLMs, open-source general-purpose VLMs, and specialized remote sensing VLMs (RSVLMs). The strongest models, ChatGPT and Gemini, achieved 89.3% and 86.9%, respectively, exceeding average human performance. In contrast, the best open-source general-purpose model, Qwen-3-VL, achieved 42.6%, while the strongest RSVLM, LHRS-Bot, achieved 20.5%.

Analysis across problem categories revealed a consistent pattern. RSVLMs performed relatively well on Semantic tasks involving object recognition, but remained substantially weaker on tasks requiring relational and abstract reasoning. Additional prompting and shuffle experiments further demonstrated that, although current RSVLMs can acquire domain-specific perceptual knowledge, their reasoning capabilities remain fragile.

These findings suggest that adapting visual perception to remote sensing is considerably easier than transferring general reasoning abilities. Consequently, future RSVLM research should focus not only on improving domain-specific understanding but also on developing methods that preserve or enhance high-level reasoning capabilities. We hope that BMRS will provide a challenging benchmark for measuring progress toward this goal.

Appendix A.1. Common System Prompt

Table A1. Common system prompt used for all evaluated strategies.

Prompt A1: Common system prompt.
You are a vision understanding module designed to provide short, clear, and accurate answers. Your goal is to solve a Bongard problem consisting of a collage with six images on the left side and six images on the right side.
All left images share a common concept that none of the right images have, and all right images share a different common concept that none of the left images have. Your task is to identify both concepts.
The answer must consist of exactly two plain sentences: the first sentence describes the concept of the left side, and the second sentence describes the concept of the right side. Do not use markdown, bullet points, or any formatting. Keep each sentence short and clear.

Table A1. Common system prompt used for all evaluated strategies.

Prompt A1: Common system prompt.
You are a vision understanding module designed to provide short, clear, and accurate answers. Your goal is to solve a Bongard problem consisting of a collage with six images on the left side and six images on the right side.
All left images share a common concept that none of the right images have, and all right images share a different common concept that none of the left images have. Your task is to identify both concepts.
The answer must consist of exactly two plain sentences: the first sentence describes the concept of the left side, and the second sentence describes the concept of the right side. Do not use markdown, bullet points, or any formatting. Keep each sentence short and clear.

Appendix A.2. Direct Strategy

Table A2. Prompt used for the direct strategy.

Prompt A2: Direct strategy.
Here is the Bongard problem collage. Provide your answer as two plain sentences: first the concept for all left images, then the concept for all right images.

Table A2. Prompt used for the direct strategy.

Prompt A2: Direct strategy.
Here is the Bongard problem collage. Provide your answer as two plain sentences: first the concept for all left images, then the concept for all right images.

Appendix A.3. Descriptive-Direct Strategy

Table A3. Prompts used for the descriptive-direct strategy.

Prompt A3: Descriptive-direct strategy.
This is a single image from either the left or right side of a Bongard problem. Do not solve the problem yet. Describe this image in detail.
Focus on all visual features: objects, shapes, colors, spatial arrangements, textures, and any other distinctive properties. Be thorough, as these descriptions will later be used to identify a common concept.
Here is the Bongard problem collage. Below are the detailed descriptions of each left image and each right image. Based on this information, provide your answer.
Left class image descriptions: {}. Right class image descriptions: {}.

Table A3. Prompts used for the descriptive-direct strategy.

Prompt A3: Descriptive-direct strategy.
This is a single image from either the left or right side of a Bongard problem. Do not solve the problem yet. Describe this image in detail.
Focus on all visual features: objects, shapes, colors, spatial arrangements, textures, and any other distinctive properties. Be thorough, as these descriptions will later be used to identify a common concept.
Here is the Bongard problem collage. Below are the detailed descriptions of each left image and each right image. Based on this information, provide your answer.
Left class image descriptions: {}. Right class image descriptions: {}.

Appendix A.4. Contrastive-Direct Strategy

Table A4. Prompts used for the contrastive-direct strategy.

Prompt A4: Contrastive-direct strategy.
Here is a pair: one image from the left side of a Bongard problem and one from the right side. Do not solve the full problem yet. Instead, carefully examine both images and list all differences between them.
Focus on contrasting visual features: objects, shapes, colors, positions, textures, counts, orientations, or any other properties that distinguish the left image from the right image. Output your answer as a simple list of differences. Use plain text, no markdown.
Here is the full Bongard problem collage. Below are all the difference lists obtained from comparing left-right image pairs.
Based on this information, determine the common concept that unites all left images but no right images, and the common concept that unites all right images but no left images. Provide your answer.

Table A4. Prompts used for the contrastive-direct strategy.

Prompt A4: Contrastive-direct strategy.
Here is a pair: one image from the left side of a Bongard problem and one from the right side. Do not solve the full problem yet. Instead, carefully examine both images and list all differences between them.
Focus on contrasting visual features: objects, shapes, colors, positions, textures, counts, orientations, or any other properties that distinguish the left image from the right image. Output your answer as a simple list of differences. Use plain text, no markdown.
Here is the full Bongard problem collage. Below are all the difference lists obtained from comparing left-right image pairs.
Based on this information, determine the common concept that unites all left images but no right images, and the common concept that unites all right images but no left images. Provide your answer.

Appendix A.5. Contrastive-Iterative Strategy

Table A5. Prompts used for the contrastive-iterative strategy.

Prompt A5: Contrastive-iterative strategy.
Here is a pair: one image from the left side of a Bongard problem and one from the right side. Carefully examine both images and list all differences between them.
Focus on contrasting visual features: objects, shapes, colors, positions, textures, counts, orientations, or any other properties that distinguish the left image from the right image. Propose a candidate concept that distinguishes the left image from the right image.
Here is the next left-right pair. Your goal is to generalize the concept to fit all of the pairs you have seen. Carefully examine the new pair and find all differences between the images.
Focus on contrasting visual features: objects, shapes, colors, positions, textures, counts, orientations, or any other properties that distinguish the left image from the right image. Your previous candidate concept is: {}. Check whether the previous concept applies to the new pair.
If it is fully correct, output the same concept unchanged. If it is partially correct, refine the concept by removing or adjusting the failing parts, keeping only the aspects valid for all left images seen so far and false for all right images. If it is completely wrong, discard it and formulate a completely new concept based on all pairs seen now, including this one.
Output the updated concept as one detailed sentence covering all distinguishing features common to all processed pairs.
This is the last left-right pair. Based on all six pairs and your iterative refinement, provide your final answer.
Output exactly two plain sentences: the first describes the concept that holds for all left images but no right images, and the second describes the concept that holds for all right images but no left images. Your iterative concept is: {}.

Table A5. Prompts used for the contrastive-iterative strategy.

Prompt A5: Contrastive-iterative strategy.
Here is a pair: one image from the left side of a Bongard problem and one from the right side. Carefully examine both images and list all differences between them.
Focus on contrasting visual features: objects, shapes, colors, positions, textures, counts, orientations, or any other properties that distinguish the left image from the right image. Propose a candidate concept that distinguishes the left image from the right image.
Here is the next left-right pair. Your goal is to generalize the concept to fit all of the pairs you have seen. Carefully examine the new pair and find all differences between the images.
Focus on contrasting visual features: objects, shapes, colors, positions, textures, counts, orientations, or any other properties that distinguish the left image from the right image. Your previous candidate concept is: {}. Check whether the previous concept applies to the new pair.
If it is fully correct, output the same concept unchanged. If it is partially correct, refine the concept by removing or adjusting the failing parts, keeping only the aspects valid for all left images seen so far and false for all right images. If it is completely wrong, discard it and formulate a completely new concept based on all pairs seen now, including this one.
Output the updated concept as one detailed sentence covering all distinguishing features common to all processed pairs.
This is the last left-right pair. Based on all six pairs and your iterative refinement, provide your final answer.
Output exactly two plain sentences: the first describes the concept that holds for all left images but no right images, and the second describes the concept that holds for all right images but no left images. Your iterative concept is: {}.

Appendix A.6. Descriptive-Iterative Strategy

Table A6. Prompts used for the descriptive-iterative strategy.

Prompt A6: Descriptive-iterative strategy.
This is the first image from one side of a Bongard problem. Later you will see more images from this same side. Your goal is to formulate a general description of all images on this side for solving the problem later.
Do not solve the problem yet. Describe this image in detail. Focus on all visual features: objects, shapes, colors, spatial arrangements, textures, counts, orientations, and any other distinctive properties. Be thorough, as these descriptions will later be used to identify a common concept.
Here is the next image from the same side. Your goal is to generalize the class description to fit all images from this side that you have seen so far.
Carefully examine this new image and list all its relevant visual features: objects, shapes, colors, spatial arrangements, textures, counts, orientations, or any other distinctive properties. Your previous candidate description for this side is: {}. Check whether it applies to the new image.
If it is fully correct, output the same description unchanged. If it is partially correct, refine the description by removing or adjusting the failing parts, keeping only what is true for all images seen so far. If it is completely wrong, discard it and formulate a completely new description based on all images seen now, including this one.
This is the last image from this side. Based on all six images and your iterative refinement, provide the final description that unites all images on this side. Your iterative description is: [previous concept].
Here is the Bongard problem collage. Below are generalized descriptions of the left and right sides of images. Based on this information, solve the Bongard problem.
Left class image descriptions: {}. Right class image descriptions: {}.

Table A6. Prompts used for the descriptive-iterative strategy.

Prompt A6: Descriptive-iterative strategy.
This is the first image from one side of a Bongard problem. Later you will see more images from this same side. Your goal is to formulate a general description of all images on this side for solving the problem later.
Do not solve the problem yet. Describe this image in detail. Focus on all visual features: objects, shapes, colors, spatial arrangements, textures, counts, orientations, and any other distinctive properties. Be thorough, as these descriptions will later be used to identify a common concept.
Here is the next image from the same side. Your goal is to generalize the class description to fit all images from this side that you have seen so far.
Carefully examine this new image and list all its relevant visual features: objects, shapes, colors, spatial arrangements, textures, counts, orientations, or any other distinctive properties. Your previous candidate description for this side is: {}. Check whether it applies to the new image.
If it is fully correct, output the same description unchanged. If it is partially correct, refine the description by removing or adjusting the failing parts, keeping only what is true for all images seen so far. If it is completely wrong, discard it and formulate a completely new description based on all images seen now, including this one.
This is the last image from this side. Based on all six images and your iterative refinement, provide the final description that unites all images on this side. Your iterative description is: [previous concept].
Here is the Bongard problem collage. Below are generalized descriptions of the left and right sides of images. Based on this information, solve the Bongard problem.
Left class image descriptions: {}. Right class image descriptions: {}.

Appendix B.1. System Prompt for the LLM Judge

Table A7. System prompt used for the LLM judge.

Prompt B1: LLM judge system prompt.
You evaluate the user’s answer by comparing it with reference answers for a given task.
You see: reference answers, examples of correct and incorrect answers, and the user answer.
Each answer contains features of the right and left class. Identify the target property that distinguishes the right class from the left. This may describe semantics, form, presence, quantity, or size of objects.
The user answer may contain features of both right and left classes, or an explicit target property separating these classes. If two features are indicated, formulate the target property based on them.
The answer is correct if the target property matches the reference or correct examples, accounting for generalization, paraphrasing, synonyms, word order changes, or simplification. The answer is incorrect if the target property is missing or wrong.
Do not accept answers similar to or matching incorrect examples. Focus on meaning, not exact wording. Ignore minor differences in style, grammar, or phrasing.
Respond with only one word: correct or incorrect.

Table A7. System prompt used for the LLM judge.

Prompt B1: LLM judge system prompt.
You evaluate the user’s answer by comparing it with reference answers for a given task.
You see: reference answers, examples of correct and incorrect answers, and the user answer.
Each answer contains features of the right and left class. Identify the target property that distinguishes the right class from the left. This may describe semantics, form, presence, quantity, or size of objects.
The user answer may contain features of both right and left classes, or an explicit target property separating these classes. If two features are indicated, formulate the target property based on them.
The answer is correct if the target property matches the reference or correct examples, accounting for generalization, paraphrasing, synonyms, word order changes, or simplification. The answer is incorrect if the target property is missing or wrong.
Do not accept answers similar to or matching incorrect examples. Focus on meaning, not exact wording. Ignore minor differences in style, grammar, or phrasing.
Respond with only one word: correct or incorrect.

Appendix B.2. User Prompt Template

Table A8. User prompt template used for the LLM judge.

Prompt B2: LLM judge user prompt template.
Task-specific evaluation instruction:
{class_specific_prompt}
Reference answers:
Left: {reference_left}
Right: {reference_right}
Examples of correct answers:
Example 1:
Left: {correct_example_left}
Right: {correct_example_right}
Examples of incorrect answers:
Example 1:
Left: {incorrect_example_left}
Right: {incorrect_example_right}
Model answer:
{model_answer}

Table A8. User prompt template used for the LLM judge.

Prompt B2: LLM judge user prompt template.
Task-specific evaluation instruction:
{class_specific_prompt}
Reference answers:
Left: {reference_left}
Right: {reference_right}
Examples of correct answers:
Example 1:
Left: {correct_example_left}
Right: {correct_example_right}
Examples of incorrect answers:
Example 1:
Left: {incorrect_example_left}
Right: {incorrect_example_right}
Model answer:
{model_answer}

Appendix B.3. Class-Specific Evaluation Instructions

Table A9. Class-specific evaluation instruction for size-based tasks.

Prompt B3: Size.
You are given answers to a problem of the “size” type. Your task is to check whether each answer is correct.   A correct answer must explicitly mention one of the following differences between the objects in the left and right sets of images: the objects in one set are closer and the objects in the other set are farther; the objects in one set are bigger and the objects in the other set are smaller; or each set has a name, such as “barns” on the left and “estates” on the right, and these names reflect a real-world size difference.
For example, barns are typically smaller than estates, so stating that difference would be correct logic.
If an answer contains the phrase “I do not know” or any similar expression of uncertainty, it is incorrect.
Output your judgment as correct or incorrect for each answer, with a brief justification.

Table A9. Class-specific evaluation instruction for size-based tasks.

Prompt B3: Size.
You are given answers to a problem of the “size” type. Your task is to check whether each answer is correct.   A correct answer must explicitly mention one of the following differences between the objects in the left and right sets of images: the objects in one set are closer and the objects in the other set are farther; the objects in one set are bigger and the objects in the other set are smaller; or each set has a name, such as “barns” on the left and “estates” on the right, and these names reflect a real-world size difference.
For example, barns are typically smaller than estates, so stating that difference would be correct logic.
If an answer contains the phrase “I do not know” or any similar expression of uncertainty, it is incorrect.
Output your judgment as correct or incorrect for each answer, with a brief justification.

Table A10. Class-specific evaluation instruction for presence-based tasks.

Prompt B4: Presence.
In this task, the reference target property is based on the presence of an object of a certain class in one set of images and the absence of that same object in the other set. The property is not based on the shape of objects, their color, size, or spatial arrangement.
The target property can be rephrased in semantic terms while preserving meaning. For example, the property “images in the left class contain clouds, images in the right class do not contain clouds” can be rephrased as “the left class depicts cloudy weather, and the right class depicts clear weather,” because clear weather means the absence of clouds.
Analyze the correct answers to understand which semantic rephrasings have already been accepted as valid. Analyze the incorrect answers to understand which semantic formulations lead to errors.
If the answer to be checked is formulated in terms of presence or absence of an object, evaluate it according to the standard rules. If the answer is formulated in semantic terms, first check whether such semantics has appeared in examples of correct or incorrect answers. If it has, follow those examples. If the semantics is new and has not appeared before, analyze whether the named properties always imply the presence or absence of the target object. If they do, mark the answer as correct; otherwise, mark it as incorrect.
If the answer refers to “image” in the singular rather than “images” as a set, accept it as valid as long as the distinction between the two sets is preserved in meaning. The answer is also valid if it includes additional information such as a cause-and-effect relationship, but it is incorrect if it includes additional distinguishing features.
If the answer describes a characteristic of an object being “more” or “less” of something rather than simply the presence or absence of the target object, accept it only if no additional distinguishing features are provided beyond that quantitative difference. If the answer includes other differences beyond the more-or-less comparison, reject it as incorrect.

Table A10. Class-specific evaluation instruction for presence-based tasks.

Prompt B4: Presence.
In this task, the reference target property is based on the presence of an object of a certain class in one set of images and the absence of that same object in the other set. The property is not based on the shape of objects, their color, size, or spatial arrangement.
The target property can be rephrased in semantic terms while preserving meaning. For example, the property “images in the left class contain clouds, images in the right class do not contain clouds” can be rephrased as “the left class depicts cloudy weather, and the right class depicts clear weather,” because clear weather means the absence of clouds.
Analyze the correct answers to understand which semantic rephrasings have already been accepted as valid. Analyze the incorrect answers to understand which semantic formulations lead to errors.
If the answer to be checked is formulated in terms of presence or absence of an object, evaluate it according to the standard rules. If the answer is formulated in semantic terms, first check whether such semantics has appeared in examples of correct or incorrect answers. If it has, follow those examples. If the semantics is new and has not appeared before, analyze whether the named properties always imply the presence or absence of the target object. If they do, mark the answer as correct; otherwise, mark it as incorrect.
If the answer refers to “image” in the singular rather than “images” as a set, accept it as valid as long as the distinction between the two sets is preserved in meaning. The answer is also valid if it includes additional information such as a cause-and-effect relationship, but it is incorrect if it includes additional distinguishing features.
If the answer describes a characteristic of an object being “more” or “less” of something rather than simply the presence or absence of the target object, accept it only if no additional distinguishing features are provided beyond that quantitative difference. If the answer includes other differences beyond the more-or-less comparison, reject it as incorrect.

Table A11. Class-specific evaluation instruction for number-based tasks.

Prompt B5: Number.
You are given answers to a problem of the “number” type. Your task is to check whether each answer is correct.
A correct answer must explicitly mention one of the following differences between the objects in the left and right sets of images: every image in one set contains more or fewer specific objects than every image in the other set; or every image in one set contains a specific fixed number of objects, and every image in the other set contains a different specific fixed number of objects.
For example, an answer is correct if it states that left-set images each have more apples and right-set images each have fewer apples, or that the left set always has three cars and the right set always has seven cars.
If an answer contains “I do not know” or any similar expression of uncertainty, it is incorrect.

Table A11. Class-specific evaluation instruction for number-based tasks.

Prompt B5: Number.
You are given answers to a problem of the “number” type. Your task is to check whether each answer is correct.
A correct answer must explicitly mention one of the following differences between the objects in the left and right sets of images: every image in one set contains more or fewer specific objects than every image in the other set; or every image in one set contains a specific fixed number of objects, and every image in the other set contains a different specific fixed number of objects.
For example, an answer is correct if it states that left-set images each have more apples and right-set images each have fewer apples, or that the left set always has three cars and the right set always has seven cars.
If an answer contains “I do not know” or any similar expression of uncertainty, it is incorrect.

Table A12. Class-specific evaluation instruction for shape-based tasks.

Prompt B6: Shape.
In this task, the reference target property is based on the shape of objects in the image, regardless of their semantics.
The target property can be rephrased in terms of object semantics while preserving meaning. For example, the property “the boundary between land and water is closed or not closed” can be rephrased semantically as “island or peninsula.”
Analyze the correct answers to understand which semantic rephrasings have already been accepted as valid. Also analyze the incorrect answers to understand which semantic formulations are mistaken.
If the answer to be checked is formulated in terms of shape, evaluate it according to the standard rules. If the answer is formulated in terms of semantics, first check whether such semantics has appeared in examples of correct or incorrect answers. If it has, follow those examples. If the semantics is new and has not appeared before, analyze whether objects with such semantics always have the required shape difference. If they do, the answer is correct; otherwise, it is incorrect.

Table A12. Class-specific evaluation instruction for shape-based tasks.

Prompt B6: Shape.
In this task, the reference target property is based on the shape of objects in the image, regardless of their semantics.
The target property can be rephrased in terms of object semantics while preserving meaning. For example, the property “the boundary between land and water is closed or not closed” can be rephrased semantically as “island or peninsula.”
Analyze the correct answers to understand which semantic rephrasings have already been accepted as valid. Also analyze the incorrect answers to understand which semantic formulations are mistaken.
If the answer to be checked is formulated in terms of shape, evaluate it according to the standard rules. If the answer is formulated in terms of semantics, first check whether such semantics has appeared in examples of correct or incorrect answers. If it has, follow those examples. If the semantics is new and has not appeared before, analyze whether objects with such semantics always have the required shape difference. If they do, the answer is correct; otherwise, it is incorrect.

Table A13. Class-specific evaluation instruction for spatial-relation tasks.

Prompt B7: Spatial.
In this task, the reference target property is based exclusively on the position of objects in the image, the orientation of objects relative to the image frame, or the mutual arrangement of objects relative to each other. The property is not based on shape, color, size, quantity, or semantics.
Semantic rephrasings are allowed as long as they preserve the meaning of the spatial arrangement. For example, “left-hand traffic” means that cars move on the left side of the road, and “right-hand traffic” means that cars move on the right side.
As a general rule, if an answer does not refer to the location or orientation of objects at all, it is incorrect. If the answer refers to a different type of spatial relation, it is also incorrect. The answer is still valid if it includes additional information such as a cause-and-effect relationship of this spatial relation, but it is incorrect if it includes additional objects as a distinguishing feature.
For tasks about absolute location, the answer must clearly indicate that the difference between the classes lies in the location. It is not necessary to specify exact parts of the image for each class, but the answer must state what the difference is. For example, “on different sides of the picture” is acceptable, whereas “in different places” is not, because it does not specify the difference. The object can be referred to generically as an “object.” If the object is named specifically, it must be named correctly. Rephrasings are allowed, such as “forest” instead of “trees.”
For tasks about orientation, the answer must clearly indicate that the difference between the classes lies in orientation. It is not necessary to specify the exact orientation for each class, but the answer must state what the difference is. For example, “oriented horizontally and vertically” is acceptable, whereas “oriented differently” is not, because it does not specify the difference. The object can be referred to generically as an “object.” If the object is named specifically, it must be named correctly. Rephrasings are allowed, such as “forest” instead of “trees.”
For tasks about mutual arrangement of objects, the answer must clearly indicate that the difference lies in the relative arrangement, not in absolute positions. If both objects or object classes are correctly identified, it is sufficient to state that there is a difference in their mutual arrangement. If specific mutual arrangements are given, they must be correct. It is acceptable to name only one object, but then the answer must specifically state that the difference is in its position relative to the other object. If no object is named, the answer is not counted as correct.
When checking, first determine the type of the reference answer: absolute location, orientation, or mutual arrangement. Then evaluate the user’s answer according to these rules. Analyze the correct answers to understand which formulations have already been accepted as valid. Analyze the incorrect answers to understand common mistakes.

Table A13. Class-specific evaluation instruction for spatial-relation tasks.

Prompt B7: Spatial.
In this task, the reference target property is based exclusively on the position of objects in the image, the orientation of objects relative to the image frame, or the mutual arrangement of objects relative to each other. The property is not based on shape, color, size, quantity, or semantics.
Semantic rephrasings are allowed as long as they preserve the meaning of the spatial arrangement. For example, “left-hand traffic” means that cars move on the left side of the road, and “right-hand traffic” means that cars move on the right side.
As a general rule, if an answer does not refer to the location or orientation of objects at all, it is incorrect. If the answer refers to a different type of spatial relation, it is also incorrect. The answer is still valid if it includes additional information such as a cause-and-effect relationship of this spatial relation, but it is incorrect if it includes additional objects as a distinguishing feature.
For tasks about absolute location, the answer must clearly indicate that the difference between the classes lies in the location. It is not necessary to specify exact parts of the image for each class, but the answer must state what the difference is. For example, “on different sides of the picture” is acceptable, whereas “in different places” is not, because it does not specify the difference. The object can be referred to generically as an “object.” If the object is named specifically, it must be named correctly. Rephrasings are allowed, such as “forest” instead of “trees.”
For tasks about orientation, the answer must clearly indicate that the difference between the classes lies in orientation. It is not necessary to specify the exact orientation for each class, but the answer must state what the difference is. For example, “oriented horizontally and vertically” is acceptable, whereas “oriented differently” is not, because it does not specify the difference. The object can be referred to generically as an “object.” If the object is named specifically, it must be named correctly. Rephrasings are allowed, such as “forest” instead of “trees.”
For tasks about mutual arrangement of objects, the answer must clearly indicate that the difference lies in the relative arrangement, not in absolute positions. If both objects or object classes are correctly identified, it is sufficient to state that there is a difference in their mutual arrangement. If specific mutual arrangements are given, they must be correct. It is acceptable to name only one object, but then the answer must specifically state that the difference is in its position relative to the other object. If no object is named, the answer is not counted as correct.
When checking, first determine the type of the reference answer: absolute location, orientation, or mutual arrangement. Then evaluate the user’s answer according to these rules. Analyze the correct answers to understand which formulations have already been accepted as valid. Analyze the incorrect answers to understand common mistakes.

Table A14. Class-specific evaluation instruction for intra-class consistency tasks.

Prompt B8: Same.
For this task, the target property should be a shared visual characteristic that unifies objects in one class and sets them apart from the other class.
A correct answer must describe a visual attribute that is uniform across the images in one set and non-uniform in the other collage.
The key distinction should rely on intra-set consistency, for example all objects facing the same direction or all objects having the same color or shape, versus variability within the opposing set.

Table A14. Class-specific evaluation instruction for intra-class consistency tasks.

Prompt B8: Same.
For this task, the target property should be a shared visual characteristic that unifies objects in one class and sets them apart from the other class.
A correct answer must describe a visual attribute that is uniform across the images in one set and non-uniform in the other collage.
The key distinction should rely on intra-set consistency, for example all objects facing the same direction or all objects having the same color or shape, versus variability within the opposing set.

Table A15. Class-specific evaluation instruction for semantic tasks.

Prompt B9: Semantic.
A correct answer identifies a difference in object type, purpose, class, activity, or real-world meaning between the two sets of images.
The target property must be the object type itself, such as rivers versus roads, or a direct equivalent, such as the presence of water versus the absence of water.
An answer is incorrect if it replaces the object types with a broader category, such as “natural landscapes versus human-made infrastructure,” “rural versus urban,” or “ecologically rich versus degraded.”
An answer is also incorrect if it adds an unnecessary distinguishing attribute that is not guaranteed by the reference answer. For example, “winding rivers” is incorrect if winding is typical but not required, and “urban roads” is incorrect because roads can be rural.

Table A15. Class-specific evaluation instruction for semantic tasks.

Prompt B9: Semantic.
A correct answer identifies a difference in object type, purpose, class, activity, or real-world meaning between the two sets of images.
The target property must be the object type itself, such as rivers versus roads, or a direct equivalent, such as the presence of water versus the absence of water.
An answer is incorrect if it replaces the object types with a broader category, such as “natural landscapes versus human-made infrastructure,” “rural versus urban,” or “ecologically rich versus degraded.”
An answer is also incorrect if it adds an unnecessary distinguishing attribute that is not guaranteed by the reference answer. For example, “winding rivers” is incorrect if winding is typical but not required, and “urban roads” is incorrect because roads can be rural.

Appendix D.1. Most Challenging Problems for Humans

The nine most difficult BMRS problems all belong to the Spatial category: bb_m_97, bb_s_08, bb_s_14, bb_s_20, bb_s_15, bb_s_57, bb_m_98, bb_m_95, and bb_s_30, listed from most to least difficult.

Six of these problems (bb_m_97, bb_s_08, bb_s_14, bb_s_20, bb_s_15, and bb_s_57) belong to the Absolute Position subclass and require distinguishing object locations within the image (left versus right, or top versus bottom). Problems bb_m_98 and bb_s_30 belong to the Orientation subclass, while bb_m_95 belongs to the Relative Position subclass.

Representative examples of most difficult problems from Spatial subclasses are shown in Figure A5.

Figure A5. Representative examples of the three Spatial subclasses among the most challenging BMRS problems for human participants.

Figure A5. Representative examples of the three Spatial subclasses among the most challenging BMRS problems for human participants.

There were also several difficult problems in the Shape, Same, Presence, and Semantic categories for which fewer than 50% of participants provided correct solutions. Figure A6 presents the most difficult problem from each of these categories.

Notably, all problems in the Number and Size categories were solved correctly by more than half of the participants, indicating that these categories were comparatively easy for humans.

Figure A6. The most challenging non-Spatial BMRS problems for human participants, shown for each problem category.

Figure A6. The most challenging non-Spatial BMRS problems for human participants, shown for each problem category.

Appendix E.1. Examples of Incorrect Answers

Since understanding not only which problems models struggle with but also how they fail is valuable for further model improvement, we provide examples of incorrect responses.

Some responses did not constitute valid answers to the Bongard problems in terms of their wording. For instance, we observed repetitions of the prompt, descriptions of the images, isolated symbols, as well as incoherent phrases and sentence fragments. This issue was particularly frequent for EarthDial, which tended to produce descriptions of individual images rather than the underlying concept. Examples of such responses are shown in Table A16.

Among the responses that were valid answers to the Bongard problems but still incorrect, the most frequent errors involved either ignoring or hallucinating objects. The majority of such responses were formulated around semantics, presence, and visual attributes, with very few addressing object positions or counts. Examples of these errors are given in Table A17.

It may also be of interest to see what ChatGPT and Gemini answered on the problems bb_m_08, bb_m_36 and bb_m_113, showed in Figure A9. These answers are provided in Table A18 ans Table A19.

Table A16. Examples of models answers that are not valid answers to the Bongard problem.

EarthDial on bb_m_01 using Direct strategy.
port, harbor, water, lake

Table A16. Examples of models answers that are not valid answers to the Bongard problem.

EarthDial on bb_m_01 using Direct strategy.
port, harbor, water, lake

TeoChat on bb_m_28 using Contrastive-iterative strategy.

None of the left images have the concept None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right images have None of the left images have None of the right

Table A17. Examples of models answers that are valid answers to the Bongard problems but still incorrect. All the responses provided correspond to the problem bb_m_29, shown in Figure A8.

EarthDial on bb_m_29 using Contrastive-iterative strategy.
The left image is a roundabout, while the right image is a building.

Table A17. Examples of models answers that are valid answers to the Bongard problems but still incorrect. All the responses provided correspond to the problem bb_m_29, shown in Figure A8.

EarthDial on bb_m_29 using Contrastive-iterative strategy.
The left image is a roundabout, while the right image is a building.

LHRS-Bot on bb_m_29 using Contrastive-direct strategy.
The difference between the two sides of the problem is the arrangement of the buildings, roads, and trees.

VHM on bb_m_29 using Descriptive-direct strategy.
The left side images share the concept of roundabouts, while the right side images share the concept of residential areas.

SkySenseGPT on bb_m_29 using Descriptive-direct strategy.
The common concept that unites all left images but no right images is "1 intersection". The common concept that unites all right images but no left images is "1 roundabout".

Table A18. ChatGPT’s responses to the problems bb_m_08, bb_m_36, and bb_m_113 (shown in Figure A9).

ChatGPT on bb_m_08.
The left images focus on cars on paved lots or roads. The right images show vehicle depots or parking areas mixed with buildings, buses, or trailers.

Table A18. ChatGPT’s responses to the problems bb_m_08, bb_m_36, and bb_m_113 (shown in Figure A9).

ChatGPT on bb_m_08.
The left images focus on cars on paved lots or roads. The right images show vehicle depots or parking areas mixed with buildings, buses, or trailers.

ChatGPT on bb_m_36.
The left images show fields with clear crop rows or furrows. The right images show fields without clear crop rows or furrows.

ChatGPT on bb_m_113.
The left images show square or rectangular roof forms. The right images show circular or radial roof forms.

Table A19. Gemini’s responses to the problems bb_m_08, bb_m_36, and bb_m_113 (shown in Figure A9).

Gemini on bb_m_08.
The left images feature groups of multi-colored vehicles. The right images feature fleets of uniformly white vehicles

Table A19. Gemini’s responses to the problems bb_m_08, bb_m_36, and bb_m_113 (shown in Figure A9).

Gemini on bb_m_08.
The left images feature groups of multi-colored vehicles. The right images feature fleets of uniformly white vehicles

Gemini on bb_m_36.
The left images show agricultural fields with distinct parallel rows or furrows. The right images show agricultural fields with uniform textures lacking visible rows.

Gemini on bb_m_113.
The left images show buildings with square or rectangular roofs. The right images show buildings with circular roofs.