Caption-Grounded Structural Parsing for Compound Scientific Visuals

1. Introduction

In contemporary scientific communication, figures function as a primary medium for conveying complex ideas. While textual narratives—abstracts, methods, and results—provide comprehensive descriptions, it is often the figures that deliver immediate, intuitive access to the essence of a contribution. The exponential rise in global publication volume has been paralleled by large-scale infrastructures such as PubMed, Semantic Scholar, and ScienceDirect, which facilitate text-centric exploration. However, the automated processing of visual evidence, particularly compound figures, remains comparatively immature. Prior studies underscore that graphical depictions are more cognitively memorable and accessible than text alone [1,2], and many landmark contributions rely primarily on figures to convey their insights [3].

A major bottleneck in this context arises from the ubiquity of compound figures, constituting over 30% of scientific visual material [4]. Such figures typically combine subimages drawn from distinct experimental conditions, modalities, or conceptual settings. Unlike single-topic illustrations that can be assigned a unified semantic category, compound figures defy monolithic interpretation. This heterogeneity limits their utility in retrieval systems, knowledge extraction pipelines, and automated summarization frameworks. The challenge lies not only in delineating the subfigures visually, but also in preserving the semantic correspondence between each subfigure and its associated textual explanation.

Fortunately, most compound figures are accompanied by captions that explicitly describe their subcomponents. Captions employ symbolic indices to create an alignment between text and corresponding subimages. These indices serve a dual role: they encode spatial layout while simultaneously defining semantic groupings. Exploiting these cues offers a principled way to bridge visual regions and their textual references, suggesting that figure decomposition should be guided not solely by visual segmentation but also by semantic anchoring from captions.

Traditional solutions have approached this problem through heuristic rules and handcrafted layout assumptions [5,6,7], including line-segmentation, region-growing, or morphological filtering. While effective for regular grid-like arrangements, such heuristics collapse when facing irregular or free-form layouts. More critically, they ignore caption semantics, resulting in outputs that may be visually segmented but semantically fragmented.

Data-driven alternatives have attempted to improve robustness. For example, Tsutsui et al.[8] formulated subfigure localization as an object detection task. This approach successfully identifies distinct visual regions, yet it disregards the symbolic anchors linking them to captions, leaving a semantic gap between figure structure and textual interpretation. Similarly, Shi et al.[9] advanced the field by proposing grid-aware spatial clustering, but the assumption of layout regularity remains a critical limitation, especially when semantic groupings diverge from visual proximity.

To confront these challenges, we introduce SEMCLIP, a decomposition model that explicitly integrates semantic anchoring into figure parsing. Central to SEMCLIP is the definition of master images: mid-level semantic entities that either correspond to an individual subfigure or aggregate multiple subfigures unified by a shared label. By anchoring decomposition to detected symbolic labels, SEMCLIP ensures a tight coupling between segmented visual regions and caption-based descriptions.

Our framework adopts a two-phase pipeline. In the initial stage, a label detection module identifies caption-like annotations, capturing both positional information and latent semantic grouping. These anchors inform a hypothesized structural layout. In the subsequent stage, this structural prior is fused with localized visual signals—such as saliency cues, boundary continuity, and textural features—to yield master image segments. To address challenges of annotation sparsity and class imbalance, we propose a dual-path training paradigm that decouples optimization for detection and semantic grouping, thereby enhancing generalization across diverse figure layouts.

Unlike existing methods that are constrained by geometric regularity or detached from caption semantics, SEMCLIP dynamically adapts to both hierarchical and irregular structures. Its emphasis on caption-figure coherence guarantees that each decomposed unit is interpretable in both visual and textual domains.

The contributions of this work can be summarized as follows:

• We present SEMCLIP, a semantic-aware decomposition framework that partitions compound figures into caption-aligned master images.
• We introduce a cascaded two-stage design that first localizes symbolic label anchors and then integrates them with visual descriptors for semantic-aware segmentation.
• We develop a bifurcated training strategy that alleviates label imbalance and sparsity, resulting in robust decomposition across a wide range of layouts.

2. Related Work

2.1. Compound Figure Parsing in Scientific Literature

Figures serve as a central medium for presenting results, visualizing experimental conditions, and communicating methodological designs. Prior studies highlight that visual elements strongly enhance memorability and visibility of scholarly content [3]. Within this visual ecosystem, compound figures—constructed from multiple heterogeneous subpanels—represent more than 30% of all figures in research publications [4]. Although beneficial to human readers, this compositional complexity creates difficulties for automatic systems, which often assume a single semantic focus per figure.

Early research on compound figure parsing primarily relied on heuristic rules and handcrafted segmentation cues. Representative approaches attempted to separate subfigures by exploiting uniform backgrounds [5], or by leveraging edge and line detection signals [6], frequently combined with assumptions of spatial alignment or symmetry. Such methods achieved reasonable outcomes on clean, grid-structured layouts, but quickly deteriorated when confronted with irregular arrangements, overlapping components, or inconsistent spacing.

With the adoption of deep learning, the field gradually shifted from rule-driven segmentation to data-driven recognition. Tsutsui et al. [8] redefined the decomposition task as an object detection problem, applying YOLOv2 [10] trained on synthetic data to improve generalization. Shi et al. [9] proposed LADN, a detection framework explicitly incorporating grid layout priors. While these approaches advanced the state of the art, they remain limited in maintaining consistency with caption semantics, which is essential for grounded interpretation and scientific utility. In contrast, SEMCLIP places subfigure labels at the core of the parsing process, using them as semantic anchors to align decomposition with textual descriptions, thereby ensuring interpretability and supporting multimodal reasoning.

2.2. Object Detection as a Basis for Semantic Anchoring

Object detection, a cornerstone of computer vision, addresses the joint challenge of localizing and classifying meaningful objects. Traditional systems built on handcrafted descriptors such as SIFT [11] and HOG [12], coupled with sliding-window classifiers. Despite their innovation, these methods were limited by poor scalability and weak robustness in cluttered scenes.

The breakthrough arrived with deep convolutional networks. OverFeat [13] pioneered dense CNN-based detection, and the R-CNN family [14,15,16] achieved substantial accuracy gains by combining region proposals with dedicated classification modules. However, their two-stage designs incurred significant inference overhead. To remedy this, single-stage detectors emerged, epitomized by the YOLO series [10,17,18], which enabled real-time performance with competitive precision. Among them, YOLOv3 strikes a practical balance between speed and accuracy, making it especially suitable for large-scale processing. In SEMCLIP, this detection paradigm is adopted to identify symbolic annotations such as “(a)” and “(b)”, which then function as semantic anchors. To counter the heavy imbalance and sparsity of such labels in scientific images, we design a dual-branch optimization strategy that decouples detection from semantic association, improving robustness across diverse figure types.

2.3. Modeling Visual Relationships and Contextual Semantics

Understanding visual scenes extends beyond detecting individual entities; it requires modeling the relations among them. Visual Relationship Detection (VRD) tackles this by predicting triplets in the form (subject, predicate, object), thereby capturing structural or functional dependencies. Pioneering work [20] explored structured learning for relation triplets, while situation recognition [19] integrated contextual reasoning. Human-centric relational detection was emphasized by Gkioxari et al. [21], who designed bidirectional mechanisms to capture interaction-specific links.

Building on these insights, SEMCLIP interprets subfigure labels as relational pivots. By explicitly linking labels with their associated subregions, the model embeds a semantic relationship graph between visual components and their textual references. This design ensures that decomposition is not merely a segmentation task but also a process of semantic contextualization, preserving interpretive meaning across the figure as a whole.

2.4. Layout Analysis and Structural Parsing of Documents

Research in document layout analysis provides another relevant foundation. This line of work investigates how to segment and classify diverse document elements—paragraphs, charts, tables, or figures—especially in scanned or digital corpora. Recent approaches have advanced by integrating multimodal signals, combining visual cues, textual embeddings, and spatial configurations to construct coherent document structures.

Such multimodal layout parsers have achieved notable progress in tasks like chart interpretation, table recognition, and diagram segmentation, enabling models to capture logical and hierarchical document relationships. Inspired by this paradigm, SEMCLIP formulates compound figure parsing as a special case of layout interpretation. In this framing, segmentation is guided not only by visual boundaries but also by the structural signals encoded in subfigure labels and captions. This layout-aware stance provides SEMCLIP with the ability to reconcile spatial structure with semantic fidelity.

2.5. Cross-Modal Retrieval and Scientific Knowledge Mining

Finally, our work intersects with cross-modal retrieval and scientific knowledge extraction, both of which seek to align visual figures with textual semantics. Systems in this domain enable figure-to-text retrieval, automated captioning, and the population of knowledge graphs from scholarly literature. Central to these applications is precise subfigure segmentation and robust caption alignment.

SEMCLIP directly contributes to this ecosystem by generating decompositions that preserve semantic alignment between visual subfigures and their caption references. This fine-grained structuring not only facilitates more accurate multimodal representation learning and figure-based search, but also strengthens tasks such as evidence mining and claim verification in scientific discourse.

3. The Proposed Framework

This section introduces the design of our framework, SEMCLIP (Semantic Layout-aware Compound Image Parser), developed to decompose compound figures into mid-level semantic units termed master images. The primary difficulty arises from the fact that such master images do not always coincide with clear pixel-level boundaries nor consistent visual signatures. Hence, classical bottom-up segmentation, which depends mainly on visual similarity, is insufficient for capturing the underlying semantic organization.

To address this ill-posed segmentation scenario, SEMCLIP leverages external semantic cues—namely the symbolic subfigure labels (e.g., “(a)”, “(b)”)—as anchoring references. These labels, which are intrinsically tied to caption text, provide both spatial layout hints and semantic grouping signals. By explicitly incorporating these annotations, SEMCLIP grounds the decomposition process in semantically meaningful structure.

The framework is organized into a two-phase pipeline. The first stage establishes semantic layout priors through a Subfigure Label Detection Module, responsible for locating symbolic labels inside the figure. The second stage exploits this detected layout information to regulate the predictions of the Master Image Detection Module, thereby refining decomposition into coherent units. This hierarchical strategy mirrors how humans typically interpret figures: first identifying labeled cues that reveal structural layout, and then parsing the contained visuals into semantically interpretable groups.

3.1. Subfigure Label Detection for Semantic Layout Priors

The symbolic labels serve as the semantic nucleus of SEMCLIP. Notations like “(a)”, “(b)” encode both spatial positioning and links to descriptive caption spans. Nevertheless, reliably detecting them is non-trivial: severe label imbalance is common, where frequent indices like “(a)” dominate and rare ones like “(h)” are scarcely represented, making naïve classifiers biased.

To counteract this, SEMCLIP separates the task into two synergistic submodules: (1) a balanced label classifier that achieves high recognition fidelity under skewed data, and (2) a region proposal mechanism regularized by semantic consistency from the classifier.

Step 1: Balanced Label Classification with Synthetic Data.

We employ a ResNet-152 based classifier ${\mathcal{C}}_{\theta }$[22] trained on a combination of real and synthetically rendered labels, the latter introduced to even out the distribution of classes. Given training samples ${\left\{(x_i,y_i)\right\}}_{i=1}^N$ with $y_i\in \{1,\dots ,L\}$, the learning objective follows a cross-entropy formulation:

$${\mathcal{L}}_{\mathrm{cls}}=-\sum _{i=1}^{N}log\left(P_{\theta }(y_i\mid x_i)\right)$$

This balanced augmentation strategy ensures that the classifier maintains accuracy across both frequent and rare label categories, achieving robust performance even under long-tailed distributions.

Step 2: Classifier-regularized Region Proposals.

Standard region proposal networks rely solely on geometric overlap for bounding box regression. In contrast, we enforce semantic consistency by integrating classifier predictions into the proposal evaluation. For a candidate $b_j$ compared to its ground-truth $b^{*}$, we define:

$${\mathcal{L}}_{\mathrm{proposal}}={\mathcal{L}}_{\mathrm{IoU}}(b_j,b^{*})+\lambda ·{\mathbb{1}}_{\mathrm{label}-\mathrm{match}}·{\mathcal{L}}_{\mathrm{cls}}\left({\mathcal{C}}_{\theta }\left(b_j\right)\right)$$

where $\lambda$ tunes the weight, and ${\mathbb{1}}_{\mathrm{label}-\mathrm{match}}$ activates the semantic constraint when classifier output diverges from the expected label. This design penalizes geometrically valid but semantically inconsistent proposals, yielding a more reliable set of anchors.

By explicitly bifurcating classification and localization while also coupling them via semantic constraints, this stage constructs a strong semantic layout prior that guides subsequent decomposition.

Algorithm 1: SEMCLIP: Semantic Layout-aware Compound Image Parsing

3.2. Anchor-driven Master Image Segmentation

Once symbolic anchors are obtained, the system progresses to segmenting the figure into semantically consistent master images. This component is designed to approximate the human parsing process: leveraging anchors to infer the global structure, followed by refining local visual groupings.

Step 1: Encoding Layout Masks.

Detected anchors are encoded as a binary mask $M\in {\{0,1\}}^{H\times W}$ marking label positions, concatenated with the original image to form:

$$\tilde{I}=\mathrm{concat}(I,M)$$

thus incorporating both raw visual data and structural layout.

Step 2: Feature Extraction and Anchor Projection.

A CNN backbone produces a feature map $F\in {\mathbb{R}}^{C\times h\times w}$ from $\tilde{I}$. Using ROI pooling $\varphi$, we project anchor coordinates $\mathcal{A}=\{a_1,\dots ,a_K\}$ into latent descriptors:

$$z_k=\varphi (F,a_k),k=1,\dots ,K$$

Step 3: Inferring Latent Masks.

Each descriptor $z_k$ is passed through a decoder $\mathcal{D}$ to predict a soft segmentation mask:

$${\widehat{M}}_k=\mathcal{D}\left(z_k\right)$$

These masks are upsampled and binarized to obtain spatial support for candidate master images.

Step 4: Refining by Semantic Coherence.

To enforce intra-region uniformity, coherence is assessed by comparing pixel features to the region mean:

$$\mathrm{Coherence}\left(R_k\right)={\displaystyle \frac{1}{|R_k|}}\sum _{(x,y)\in R_k}{\parallel f(x,y)-{\mu }_k\parallel }_2^2$$

Regions with coherence below threshold $\tau$ are retained, filtering out noisy or semantically inconsistent masks.

3.3. Optional Extension: Caption-Guided Matching

To explicitly link subfigures to textual references, we integrate an optional Caption Embedding Alignment Module (CEAM). For each label $a_k$, its caption span $t_k$ is embedded via a language encoder ${\mathcal{L}}_{\mathrm{text}}$:

$$e_k^{\mathrm{text}}={\mathcal{L}}_{\mathrm{text}}\left(t_k\right)$$

while visual embedding $e_k^{\mathrm{img}}$ is obtained from pooled region features. Alignment is optimized through:

$${\mathcal{L}}_{\mathrm{align}}=\sum _k\left(1-cos(e_k^{\mathrm{img}},e_k^{\mathrm{text}})\right)$$

which encourages multimodal consistency and can serve as auxiliary supervision when dense caption labels are available.

3.4. Discussion and Design Benefits

The SEMCLIP framework provides several benefits. First, by treating subfigure labels as semantic anchors, segmentation is semantically grounded and each master image remains aligned with its caption reference. Second, label imbalance is effectively addressed by decoupling classification and localization, ensuring rare classes are recognized without degrading frequent ones. Third, the pipeline is inherently interpretable: each stage corresponds to intuitive human-like reasoning steps, from label spotting to structural grouping. This makes SEMCLIP both transparent and reliable. Finally, the modularity of the design allows easy integration of enhancements such as caption alignment or additional semantic filters, extending its utility to downstream tasks including figure retrieval, multimodal QA, and scientific content summarization.

4. Experiments

We now provide a thorough empirical study to validate the performance and reliability of the proposed SEMCLIP framework. Our evaluation encompasses the full pipeline, including subfigure label recognition, label detection, and master image segmentation. To gain deeper insights, we further carry out ablation experiments, domain-transfer assessments, and fine-grained breakdown analyses. This section details dataset construction, training methodology, quantitative benchmarks, qualitative case studies, and diagnostic error analysis.

4.1. Dataset Construction and Annotation Protocol

To enable rigorous testing, we curated a benchmark of 1000 compound figures sourced from prestigious publishers, namely The Royal Society of Chemistry (RSC), Springer Nature, and the American Chemical Society (ACS). The corpus spans a wide spectrum of disciplines and figure formats, ranging from microscopy imagery to chemical diagrams. This diversity captures heterogeneous layouts, including clean grids, irregular scatterings, and multimodal composites that combine plots with experimental visuals.

Annotations were carried out using Amazon Mechanical Turk (MTurk). Annotators decomposed each compound figure into semantically interpretable units, termed master images, and linked them with their corresponding symbolic labels. Each annotation underwent manual verification to guarantee accuracy in both text correctness and bounding box geometry. Moreover, every subfigure was assigned to one of five semantic categories—microscopy, graph, illustration, diffraction, and chemical structure—enabling richer downstream analysis.

From the entire collection, 794 figures were allocated for training while 198 were reserved for evaluation. This split was chosen to balance statistical sufficiency with layout coverage, ensuring that the test partition reflects both regular and non-standard figure structures.

4.2. Training Procedures and Data Augmentation Strategies

SEMCLIP is trained through a staged process covering three modules: subfigure label classification, label detection, and master image segmentation. Each stage is optimized under a specialized regime to maximize its specific role while remaining compatible with the overall pipeline.

1. Subfigure Label Classifier.

We start by training a ResNet-152 classifier [22]. The training set blends real subfigure crops with dynamically generated synthetic labels. Synthetic data are created by sampling background patches from figures and overlaying letters in varying fonts, sizes, and styles, rendered stochastically in both upper and lowercase. This augmentation yields balanced distributions across label classes and curtails overfitting. The classifier achieves nearly perfect recognition on validation labels and provides semantic supervision for subsequent detection.

2. Subfigure Label Detector.

Training proceeds in two stages. Initially, a region proposal network is optimized for 10,000 iterations to predict bounding boxes of candidate label regions. Thereafter, fine-tuning for an additional 3,000 iterations incorporates the pretrained classifier, ensuring that candidate regions not only match geometrically but also align semantically. During this phase, bounding boxes are filtered jointly by Intersection-over-Union (IoU) thresholds and classification agreement, thereby reducing false positives and producing reliable anchors.

3. Master Image Detector.

Finally, the master image module is trained for 12,000 iterations using gold-standard label annotations. Each input is augmented with a binary anchor mask derived from detected label positions, concatenated with the original image. This layout-aware representation enables the network to condition predictions on both appearance and structural cues, resulting in subfigure partitions that are semantically consistent.

4.3. Subfigure Label Detection Performance

We report subfigure label detection accuracy in terms of mean Average Precision (mAP), measured across labels “a” through “h”. Three systems are compared: YOLOv3 [18] as a baseline, SLDv1 (our detector without classifier refinement), and SLDv2 (our final detector with classifier regularization).

As shown in Table 1, YOLOv3 exhibits substantial variability across classes, especially failing on less frequent labels such as “h”. SLDv1 reduces this disparity by separating classification from localization. SLDv2 yields the best performance overall, raising the average mAP to 88.8%. This improvement stems directly from classifier-guided training, which suppresses semantically inconsistent predictions and compensates for label imbalance.

4.4. Master Image Detection Results

We evaluate master image segmentation by measuring True Positives (TP), False Positives (FP), and Average Precision (AP) under IoU threshold 0.5. Our method is benchmarked against the decomposition approach of Tsutsui et al. [8].

Table 2 shows that SEMCLIP significantly outperforms the baseline, with a 14% increase in AP and a drastic reduction in false positives. Even on irregular and overlapping layouts, our method produces compact, caption-aligned decompositions that maintain semantic validity.

4.5. Ablation Study: Impact of Anchor Mask Encoding

To evaluate the contribution of layout-aware encoding, we compared the full SEMCLIP system with a variant lacking anchor masks. The results are summarized in Table 3.

The performance drops notably when anchor masks are removed, demonstrating that spatial layout priors are indispensable for distinguishing between closely packed or irregular subfigures. This underscores the importance of explicitly encoding layout cues.

4.6. Cross-Domain Evaluation: Generalization Across Figure Types

We also examine robustness across heterogeneous figure categories. Table 4 reports detection AP and label accuracy across four figure types.

Despite substantial variation in content and style, SEMCLIP consistently delivers high accuracy across all domains. This suggests strong adaptability to both appearance differences and layout irregularities.

4.7. Qualitative Case Studies and Error Analysis

Finally, we analyze typical failure modes. Errors are most often caused by degraded or occluded labels, inconsistent placement of symbolic annotations, or interference from surrounding graphical elements. Some false positives arise from visually salient text such as legends or axis markers, which the detector confuses with subfigure labels. These errors suggest potential improvements via multimodal integration, e.g., caption grounding or OCR-based restoration of noisy labels.

The experimental evidence confirms that SEMCLIP achieves state-of-the-art performance in compound figure decomposition. Its integration of semantic anchors, classifier-informed detection, and layout priors yields significant gains across benchmarks. Ablation results demonstrate the necessity of each component, while cross-domain analysis establishes robustness across scientific disciplines. Altogether, SEMCLIP emerges as a reliable foundation for figure understanding and multimodal retrieval.

5. Conclusions and Future Directions

This work presented SEMCLIP, a semantic-driven framework designed for decomposing compound figures in scientific literature. In contrast to conventional heuristic or grid-based methods, SEMCLIP exploits symbolic subfigure labels as semantic anchors and integrates them with learned visual representations to achieve layout-guided decomposition. Through this anchor-centric design, the model is able to extract intermediate-level master images that simultaneously maintain structural integrity and semantic alignment with associated captions. By coupling a robust subfigure label detector with a layout-aware segmentation module, SEMCLIP effectively resolves the inherent ambiguity of compound figures and establishes a principled bridge between visual structure and textual semantics.

Empirical validation across a carefully constructed dataset demonstrates that SEMCLIP consistently surpasses existing baselines. Significant improvements are observed in both label detection and master image segmentation, particularly under challenging scenarios involving irregular layouts or skewed label distributions. The two-phase training paradigm—comprising classifier-regularized detection followed by anchor-guided segmentation—proves highly effective, enabling generalization across multiple categories such as microscopy images, illustrations, and structured graphs.

Nevertheless, several limitations remain, pointing toward fertile directions for future exploration. A primary concern is the sequential dependency inherent in our pipeline. Errors originating from the label detection stage can propagate downstream and compromise segmentation, e.g., undetected labels may result in entire regions being omitted from decomposition. Addressing this cascading error issue could involve integrating multi-hypothesis detection, confidence calibration, or redundancy-aware reasoning to mitigate the risk of early-stage failures.

Another limitation arises from reliance on fixed anchor priors during detection. Although anchor-based frameworks improve efficiency, predefined aspect ratios can misalign with real-world scientific figures that exhibit skewed, rotated, or irregular shapes. This misalignment can degrade boundary precision for master images. Future work may leverage adaptive anchor generation strategies or transition toward anchor-free paradigms, such as keypoint- or center-based detection models, which dynamically infer spatial priors directly from data distributions.

A further avenue lies in deepening the integration of caption semantics. While SEMCLIP already leverages subfigure labels as explicit semantic cues, the broader linguistic context encoded in captions—such as experimental conditions, cross-panel dependencies, or narrative hierarchies—is not yet fully exploited. Incorporating natural language understanding components that align figure layouts with caption discourse could unlock richer multimodal reasoning capabilities and improve scientific interpretability.

Finally, the principles introduced by SEMCLIP can be extended beyond figure parsing toward holistic document-level visual understanding. Potential applications include automated figure-caption alignment, diagram reconstruction, visual explanation of experimental workflows, and scientific visual question answering. Realizing these goals will require large-scale multimodal pretraining objectives, expanded domain diversity, and more scalable annotation pipelines. We regard SEMCLIP as an important step toward unified modeling of structured scientific visuals and anticipate that it will inspire further progress at the intersection of vision, language, and scientific AI.