HEADHUNTER: Training-Free Annotated Dataset Synthesis via Self-Guided Diffusion Transformer Attention Head Selection

1. Introduction

Dense visual understanding tasks such as semantic segmentation depend on pixel-level annotations, which remain among the most expensive and time-consuming labels to acquire. Annotating a single image can take up to 90 minutes [1], and in certain domains the relevant labeled data frequently does not even exist due to issues such as privacy or confidentiality [2,3]. To sidestep these limitations, a growing body of work generates image-label pairs directly from generative models. Early methods such as DatasetGAN [4] and BigDatasetGAN [5] train a shallow decoder on a small number of manually labeled images to synthesize pixel-level annotations from the feature space of a generative adversarial network (GAN) [6]. More recently, the emergence of text-to-image (T2I) diffusion models, which generate highly realistic images from natural language prompts, has exposed a localization cue in the cross-attention between text and image tokens [7]. DiffuMask exploits this T2I cross-attention to localize class regions and refine them into masks via a trained affinity network [8]; Dataset Diffusion extends this with self-attention and large language model (LLM)-generated prompts to improve mask quality [9]; DatasetDM trains a perception decoder over the diffusion latent space to emit annotations for a range of dense tasks [10]; and MosaicFusion aggregates and thresholds cross-attention maps across layers and diffusion timesteps to produce instance masks [11].

The aforementioned approaches, however, are all built on U-Net-based diffusion models. The current generation of T2I models has shifted toward the multimodal diffusion transformer (MM-DiT) architecture, which replaces U-Net’s single cross-attention pathway with joint self-attention over concatenated image and text tokens [12,13]. As such, an emerging line of work has begun to probe how MM-DiTs localize textual concepts. ConceptAttention [14] shows that linearly projecting image and concept tokens into attention layer output space yields spatial saliency maps; and Seg4Diff [15] identifies specific “semantic grounding expert” layers whose image-to-text (I2T) attention aligns text tokens with coherent image regions. While their MM-DiT layer selection protocols vary, both methods ultimately aggregate saliency across all attention heads to form the final coarse segmentation mask, yet we observe this to be particularly detrimental to fine boundary detail. Our observation is that MM-DiTs contain a rich bank of concept-sensitive heads; some are noisy, some respond to context, others to object parts, and some are clean target concept localizers. Rather than averaging them, we ask which single head best localizes a given concept (Figure 1).

Our method, HEADHUNTER, is a training-free, open-vocabulary MM-DiT segmentation framework that uses aggregated concept saliency as a self-guided proxy to score individual attention heads and emit the mask of the highest-agreeing head, achieving strong zero-shot segmentation performance (81.2 mIoU on PASCAL VOC2012 [16] and 73.2 mIoU on ImageNet-Segmentation [17]) and producing clean pseudo-labels for target concepts. We leverage this capability to apply HEADHUNTER to synthetic dataset generation, where the goal is not merely to segment existing images, but to create effective image-label pairs for training segmentation models. We propose a fully automatic annotated dataset synthesis pipeline: for a given concept, an LLM generates a diverse scene prompt; an MM-DiT synthesizes the image; HEADHUNTER extracts a pseudo-label for seed-based refinement; and a vision language model (VLM) performs QA. We evaluate the synthetic datasets generated by our method using the VOC2012 segmentation benchmark, achieving a competitive 64.9 mIoU relative to comparable synthetic dataset generators. To summarize, our contributions are:

• We show that aggregating MM-DiT attention heads discards fine boundary detail, and that selecting a single concept-aligned head yields sharper segmentation masks.
• We introduce HEADHUNTER, a training-free, open-vocabulary MM-DiT segmentation framework that selects this head using aggregate concept saliency as a self-guided proxy, achieving strong zero-shot segmentation performance.
• We propose a fully automatic annotated dataset synthesis pipeline built around HEADHUNTER, producing image-label pairs for target concepts without any human annotation, delivering competitive segmentation performance on the VOC2012 benchmark.

2. Related Work

2.1. Synthetic Dataset Generation for Semantic Segmentation

Manual pixel-level annotation remains a central bottleneck in semantic segmentation, and synthetic dataset generation seeks to address this challenge by generating image-label pairs directly from generative models. In addition to dataset diversity, the primary challenge of this line of research lies in the fact that the annotation mechanism used must localize the target object reliably enough to train segmentation models.

GANs. DatasetGAN [4] is one of the earliest examples of a synthetic dataset generator for semantic segmentation, training a lightweight decoder over GAN features from a small number of manually labeled images to synthesize additional images with dense labels. BigDatasetGAN [5] scales this idea to ImageNet-level class diversity [17] by annotating a small number of generated images per class and learning a feature interpreter over the generator’s internal representations. These methods demonstrate that the feature space of GANs can support dense annotation, but their masks depend on a supervised decoder trained from manually-annotated labels, while their outputs are tied to the GAN’s training distribution, and therefore, can only produce in-domain concepts.

Diffusion Models. The emergence of T2I diffusion models introduced an alternative pathway for synthetic dataset generation. Unlike GANs, text-conditioned diffusion models are pretrained on highly diverse image-text pairs (e.g., LAION-5B [18]) and can consequently render arbitrary scenes from natural language prompts for concepts from practically any domain. Additionally, the cross-attention between text tokens and image regions provides a spatial correspondence between each prompt word and the pixels it describes, a localization cue that can be used to form pseudo-labels. DiffuMask [8] leverages this capability to localize class-specific regions and synthesize image-label pairs, but observes that raw attention is too coarse for semantic segmentation and therefore trains an AffinityNet [19] to propagate and refine these activations into dense masks. Dataset Diffusion [9] uses an LLM for prompt design and constructs masks from a combination of cross-attention and self-attention maps, generating large VOC and COCO-style [20] synthetic datasets. DatasetDM [10] learns a perception decoder over the latent space of a pre-trained T2I diffusion model to enable the synthesis of images with paired dense annotations for perception tasks such as semantic segmentation and depth estimation. MosaicFusion [11] tackles large vocabulary instance segmentation by generating multiple objects in a composed image and obtaining instance masks through cross-attention maps aggregated across layers and timesteps, followed by thresholding and edge-aware refinement.

Our approach differs from these synthetic dataset generators in both architecture and annotation mechanism, as shown in Table 1. Rather than learning a decoder over generative features (DatasetGAN [4], BigDatasetGAN [5], DatasetDM [10]), or aggregating U-Net cross-attention maps into the final mask (DiffuMask [8], Dataset Diffusion [9], MosaicFusion [11]), we operate on a frozen MM-DiT and select the single attention head that best localizes a target concept, achieving an entirely training-free automatic annotated dataset synthesis pipeline.

2.2. Concept Localization in Diffusion Transformers

Many early diffusion-based concept localization and segmentation methods exploited the cross-attention maps of U-Net-based T2I diffusion models, where image features attend to text tokens through a relatively explicit conditioning pathway [21]. Current DiT-based models, however, replace the U-Net backbone with transformer blocks over latent image patches, exposing a sequence of image tokens processed by multi-head attention [22]. MM-DiTs extend this idea to T2I generation by jointly processing image and text tokens, allowing bidirectional information flow between modalities rather than a one-way cross-attention conditioning stream [12]. As a result, concept localization in MM-DiTs is relatively challenging, as text-image interaction is distributed across many layers, tokens, and heads, rather than concentrated in dedicated cross-attention modules.

Multimodal Attention. ConceptAttention [14] is one of the first works to show that MM-DiT representations can be used for dense visual localization. It introduces a separate concept token stream and computes dot-product similarity between concept and image patch representations in attention output space to produce spatial saliency maps for textual concepts. The concept tokens are updated alongside the image and prompt representations across MM-DiT layers, but are isolated through one-directional attention. This allows for concept token contextualization without altering image appearance, enabling open-vocabulary textual concepts to be queried on an image. Seg4Diff [15] instead analyzes the I2T attention of MM-DiTs and identifies particular “semantic grounding expert” layers where I2T attention aligns text tokens with spatially coherent image regions. By extracting these maps for classname tokens and reshaping them over latent image patches, class-specific masks can be generated for zero-shot open-vocabulary segmentation. It further proposes a lightweight LoRA fine-tuning strategy that uses annotated images to amplify semantic grouping, improving segmentation performance while also enhancing T2I synthesis quality.

Despite differing in their concept localization strategies, both methods aggregate saliency across all attention heads within selected MM-DiT layers to form segmentation masks. Our observation, as shown in Figure 1, is that semantic localization within these heads is not uniform, and that averaging dilutes stronger individual heads, degrading boundary precision and often merging target concepts with context. Our approach therefore uses aggregate MM-DiT concept saliency as a self-guided proxy to score individual attention heads and emit the mask of the single best head, yielding sharper pseudo-labels for target concepts without requiring any training or ground-truth (GT) labels.

3. Preliminaries

3.1. Rectified Flow Modelling Framework

MM-DiTs such as Stable Diffusion 3 [12] and FLUX.1 [13] are trained as rectified flow models [23], learning a velocity field along straight-line trajectories between data and noise distributions. Given an image latent $x_0~p_{data}$ and noise $ϵ~\mathcal{N}\left(0,I\right)$, the intermediate latent $x_t$ at time $t\in \left[\mathrm{0,1}\right]$ is given as

$$x_t=\left(1-t\right)x_0+tϵ.$$

(1)

The corresponding conditional velocity is $u_t=ϵ-x_0$. A velocity model $v_{\theta }$, conditioned on the encoded prompt $p$, is trained with the flow matching objective

$${\mathcal{L}}_{\mathrm{F}\mathrm{M}}={\mathbb{E}}_{\left(x_0,p\right),ϵ,t}{‖v_{\theta }\left(x_t,t,p\right)-\left(ϵ-x_0\right)‖}^2,t~\mathcal{U}\left(\mathrm{0,1}\right).$$

(2)

At inference time, an image is generated by integrating the learned ODE backward from $t=1$ to $t=0$ starting from $x_1~\mathcal{N}\left(0,I\right)$

$${\displaystyle \frac{d{x}_t}{dt}}=v_{\theta }\left(x_t,t,p\right),t:1\to 0.$$

(3)

3.2. Multimodal Attention Mechanism

Our method operates on intermediate attention layer representations computed by $v_{\theta }$ at selected points along the rectified flow trajectory. The noised image latent $x_t$ is patchified into an image token stream $X^0\in {\mathbb{R}}^{n_i\times d}$, and the prompt $p$ is encoded into a prompt token stream $P^0\in {\mathbb{R}}^{n_p\times d}$. At multimodal attention layer $\mathcal{l}$, the model maintains image and prompt streams $X^{\mathcal{l}}$ and $P^{\mathcal{l}}$. For attention head $h$, queries, keys, and values are produced using modality-specific projections

$$Q_m^{l,h}=M_m^{\mathcal{l}}{W}_{Q,m}^{l,h},K_m^{l,h}=M_m^{\mathcal{l}}{W}_{K,m}^{l,h},V_m^{l,h}=M_m^{\mathcal{l}}{W}_{V,m}^{l,h},m\in \left\{x,p\right\},$$

(4)

where $M_x^{\mathcal{l}}=X^{\mathcal{l}}$, $M_p^{\mathcal{l}}=P^{\mathcal{l}}$, and $W_{\cdot ,x}^{l,h}$, $W_{\cdot ,p}^{l,h}$ denote image and prompt projection matrices, respectively. We write scaled dot-product attention as

$$Attn=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V.$$

(5)

The image and prompt projections are concatenated along the token dimension and processed jointly by self-attention over the concatenated multimodal sequence before being split back into individual image and prompt outputs

$$O^{\mathcal{l},h}=Attn\left(\left[Q_x^{l,h};Q_p^{l,h}\right],\left[K_x^{l,h};K_p^{l,h}\right],\left[V_x^{l,h};V_p^{l,h}\right]\right),$$

(6)

$$O^{\mathcal{l},h}=\left[O_x^{l,h};O_p^{l,h}\right],$$

(7)

which are combined across heads and passed through output projection and residual updates to produce $X^{\mathcal{l}+1}$ and $P^{\mathcal{l}+1}$.

3.3. Saliency Maps from Multimodal Attention

Following [14], we augment the MM-DiT structure with a separate concept token stream $C^{\mathcal{l}}\in {\mathbb{R}}^{n_c\times d}$, where each of the $n_c$ rows corresponds to a queried textual concept (e.g., “dog”, “car”). The initial concept embeddings $C^0$ are obtained by encoding the concept strings using the same text encoder used for the prompts. At each $\mathcal{l}$, concept tokens are treated as text-like tokens and reuse the prompt projection matrices

$$Q_c^{l,h}=C^{\mathcal{l}}{W}_{Q,p}^{l,h},K_c^{l,h}=C^{\mathcal{l}}{W}_{K,p}^{l,h},V_c^{l,h}=C^{\mathcal{l}}{W}_{V,p}^{l,h}.$$

(8)

One-directional attention is performed to contextualize concept tokens without altering the original image or prompt streams

$$O_c^{\mathcal{l},h}=Attn\left(Q_c^{l,h},\left[K_x^{l,h};K_c^{l,h}\right],\left[V_x^{l,h};V_c^{l,h}\right]\right).$$

(9)

Concept localization is computed layer-wise in attention output space and then aggregated across selected layers $\mathcal{L}$ to produce a saliency map $S$

$$S^{\mathcal{l}}={\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}}_c\left({\overline{O}}_x^{\mathcal{l}}{\left({\overline{O}}_c^{\mathcal{l}}\right)}^T\right)\in {\mathbb{R}}^{n_i\times n_c},$$

(10)

$$S={\displaystyle \frac{1}{\left|\mathcal{L}\right|}}\sum _{\mathcal{l}\in \mathcal{L}}{S}^{\mathcal{l}},$$

(11)

where ${\overline{O}}_x^{\mathcal{l}}\in {\mathbb{R}}^{n_i\times d}$ and ${\overline{O}}_c^{\mathcal{l}}\in {\mathbb{R}}^{n_c\times d}$ denote the head-aggregated image and concept representations at layer $\mathcal{l}$. In Sec. 4.2, we instead retain the per-head output representations $O_x^{\mathcal{l},h}$ and $O_c^{\mathcal{l},h}$ before head aggregation and select a single layer-head saliency map that best localizes the target concept.

4. Method

We introduce HEADHUNTER, a training-free open-vocabulary MM-DiT segmentation framework that emits the mask of the single best concept-aligned attention head using aggregate saliency as a self-guided proxy. We integrate HEADHUNTER as the annotation mechanism within a fully automated annotated dataset synthesis pipeline, as shown in Figure 2.

4.1. Prompt and Concept Generation

The first stage of the dataset synthesis pipeline produces, for each target class, a set of generation prompts and associated metadata. Following prior work that uses LLMs for prompt design [9], we employ Qwen3.6-27B [24] as a prompt planner to map out these specifications. Given a target class, the planner returns a text prompt $p_i$, concept set for a target concept $c_i^{\star }$, and QA requirements for a VLM. Further details are given in Appendix A.

Prompt Diversity. To enhance the visual diversity of images generated by the T2I model, we instruct the LLM to vary text prompts along axes such as camera viewpoint, framing and scale, pose and orientation, background features, and object appearance. This differs from lexical prompt augmentation, where prompts are varied primarily through synonymous rewrites (e.g., “a red car driving on a city street” → “a red car racing through the city”). We empirically find that such synonymous variants often leave the visual scene relatively unchanged, and therefore explicitly disallow them to prevent the accumulation of visually redundant images.

Prompt Subcategories. Additionally, we instruct the LLM to form a series of subcategories for classes such as animals (e.g., dogs, cats) and vehicles (e.g., cars, boats) to improve their visual diversity following the findings of [8]. As shown in Figure 3, these classes are typically biased toward fairly standard representations in MM-DiTs, however, they exhibit large variability in the real-world (e.g., dog breeds → chihuahua, pug, etc.). We ablate the effects of prompt diversity to segmentation performance in Sec. 5.4.

Concept Sets. Each specification generated by the LLM carries two concept sets. The selection set $C_i^{select}$ contains the target and a generic “background” concept which acts as a catch-all competing channel; and the proxy set $C_i^{proxy}$, which additionally includes context nouns describing the scene (e.g., “grass”, “sky”). These scene-specific nouns give salient non-target image regions their own concept channels, preventing context activations from leaking into the target channel and yielding a cleaner proxy mask. We further examine the impact of this design choice in Sec. 5.4.

4.2. Self-Guided Attention Head Selection

Given a generated image and its concept lists $C_i^{select}$ and $C_i^{proxy}$, the second stage of the dataset synthesis pipeline produces a concept saliency map by selecting the single attention head that best localizes the target. Recall from Sec. 3.3 that per-head output representations $O_x^{\mathcal{l},h}$ and $O_c^{\mathcal{l},h}$ are formed by applying one-directional attention over concept queries and image and concept keys and values. For $C_i^{select}$, each row of $O_c^{\mathcal{l},h}$ corresponds to a single queried concept. Let $k_i^{\star }$ denote the index corresponding to the target concept $c_i^{\star }$. We compute saliency maps for candidate heads as

$$S_i^{\mathcal{l},h}={\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}}_k{\left(O_x^{\mathcal{l},h}{\left(O_c^{\mathcal{l},h}\right)}^T\right)}_{:,k_i^{\star }}\in {\left[\mathrm{0,1}\right]}^{n_i},$$

(12)

where the softmax is taken over the concept dimension and $n_i$ denotes the number of image tokens. Candidates are evaluated over layers $\mathcal{l}\in \mathcal{L}$ and heads $h\in \left\{1,...,H\right\}$, yielding $\left|\mathcal{L}\right|\times H$ total saliency maps. Further details on layer selection are provided in Appendix B. Each candidate map is min-max normalized over image tokens. To rank them without GT masks, we construct a self-guided proxy using $C_i^{proxy}$. Applying Eq. (11) gives a multi-channel proxy saliency map $S_i^{proxy}$, which is binarized to form $M_i^{proxy}$ by assigning each image token to its highest-scoring proxy concept and retaining only tokens assigned to the target

$$M_i^{proxy}\left(u\right)=\left[{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}_k\left(S_i^{proxy}\left(u,k\right)\right)=k_i^{\star }\right],$$

(13)

where $u$ indexes image tokens and $\left[\cdot \right]$ converts a Boolean condition into a binary mask. Each candidate head is then binarized at its own mean activation and scored by its agreement with $M_i^{proxy}$ using a two-class mIoU which rewards agreement on both the target and the background

$${\widehat{M}}_i^{\mathcal{l},h}=\left[S_i^{\mathcal{l},h}>{\overline{S}}_i^{\mathcal{l},h}\right],$$

(14)

$$A_i\left(\mathcal{l},h\right)={\displaystyle \frac{1}{2}}\left[\mathrm{I}\mathrm{o}\mathrm{U}\left({\widehat{M}}_i^{\mathcal{l},h},M_i^{proxy}\right)+\mathrm{I}\mathrm{o}\mathrm{U}\left(\neg {\widehat{M}}_i^{\mathcal{l},h},\neg M_i^{proxy}\right)\right].$$

(15)

Finally, we select the highest scoring head and emit the corresponding saliency map $S_i^{{\mathcal{l}}_i^{\star },h_i^{\star }}$ for thresholding and refinement

$$\left({\mathcal{l}}_i^{\star },h_i^{\star }\right)={\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}_{\mathcal{l}\in \mathcal{L},h\in \left\{1,...,H\right\}}\left(A_i\left(\mathcal{l},h\right)\right).$$

(16)

4.3. Mask Generation and Dataset QA

Mask Generation. The final stage of the dataset synthesis pipeline converts $S_i^{{\mathcal{l}}_i^{\star },h_i^{\star }}$ into a dense label and verifies whether the generated image-label pair is suitable for inclusion in the synthetic dataset. We first resize $S_i^{{\mathcal{l}}_i^{\star },h_i^{\star }}$ from $64\times 64$ to $512\times 512$ using bicubic upsampling and min-max normalize it, giving ${\tilde{S}}_i^{\star }\in {\left[\mathrm{0,1}\right]}^{512\times 512}$. The initial binary mask is then obtained by thresholding via

$$M_{i,\gamma }^0=\mathbb{I}\left[{\tilde{S}}_{i,\gamma }^{\star }\ge {\tau }_i\right],$$

(17)

where $\gamma$ indexes image pixels and ${\tau }_i$ is an Otsu threshold [25]. Because the raw thresholded mask can occasionally contain isolated activations, disconnected fragments, or small responses to surrounding context, we apply a lightweight cleanup step that anchors the mask around the dominant target support and removes isolated components, producing a cleaned mask $M_i$. To alleviate the coarseness inherited from the $64\times 64$ token grid, we use SAM2 [26] prompted by $M_i$ to improve boundary adherence, as shown in Figure 4. We first compute a bounding box enclosing the localized region. Next, we select a positive foreground point inside this region so that SAM2 can identify which object within the box should be segmented. The generated image and box-point prompt are then passed to SAM2, which returns a boundary-refined segmentation mask $M_i^{\star }$.

is upsampled, thresholded, and lightly cleaned to form a coarse binary mask $M_i$. We derive a box-point prompt from $M_i$ by enclosing the localized foreground support and selecting a positive point inside the mask. SAM2 then uses the generated image and box-point prompt to refine the object boundary, producing the final mask $M_i^{\star }$ shown in yellow.

Dataset QA. Because dataset synthesis is automatic, individual samples may contain undesirable flaws such as mask noise. Each image-label pair $\left(x_i,M_i^{\star }\right)$ is therefore verified by a VLM before being admitted into the final dataset. We use Qwen3-VL [27] for this purpose and instruct it to adhere to the QA requirements generated by the LLM in Sec. 4.1 to assess target concept presence in the generated image as well as mask localization correctness. For each sample we render a composite QA image containing $x_i$, $M_i^{\star }$, and a translucent mask overlay, and present it to the VLM together with the text prompt $p_i$, target concept $c_i^{\star }$, and mask policy data. Further details are provided in Appendix C. A sample is accepted only when the target object is present, the mask covers the visible target structure, and background regions are excluded. Masks are rejected when they omit aspects of a target structure, miss a target instance, select a distractor, or leak into background regions. The accepted synthetic dataset for class $\mathcal{y}$ is therefore

$${\mathcal{D}}_{\mathcal{y}}=\left\{\left(x_i,M_i^{\star }\right)|c_i^{\star }=\mathcal{y},q_i^{\mathrm{V}\mathrm{L}\mathrm{M}}=1\right\},$$

(18)

where $q_i^{\mathrm{V}\mathrm{L}\mathrm{M}}$ denotes the QA pass decision made by the VLM. We ablate the contribution of the QA stage to segmentation performance in Sec. 5.4.

5. Experiments

5.1. Experimental Setup

We perform two experiments in this study. First, we evaluate HEADHUNTER as a zero-shot segmenter against other interpretability methods using VOC2012 and ImageNet-Segmentation. We then assess the proposed dataset synthesis pipeline against comparable dataset generators for semantic segmentation, measuring the pipeline as a whole rather than the annotation mechanism in isolation, since the selected head seeds the refinement of Sec. 4.3 rather than producing the final mask.

Zero-Shot Segmentation. To evaluate HEADHUNTER as a zero-shot segmenter, we follow the protocol established in [14] on two datasets: VOC2012 single-class and ImageNet-Segmentation. For each image, we encode it into the MM-DiT latent space, diffuse it to the midpoint of the rectified flow trajectory by adding Gaussian noise, and extract the single best concept-aligned attention head. We use only the Otsu-thresholded binary mask ($M_{i,\gamma }^0$) for evaluation (i.e., no mask cleanup or SAM2 refinement per Sec. 4.3).

Dataset Synthesis. We generate synthetic training data with FLUX.1-dev [13] using 20 sampling steps and a resolution of $1024\times 1024$. For each VOC2012 class, Qwen3.6-27B [24] generates 250 prompts with corresponding concept sets. We use two fixed seeds per prompt to generate 500 images per class for a total of 10k training images. QA specifications generated by the LLM are used by Qwen3-VL [27] for dataset filtering, and samples that fail are automatically regenerated. Generating and verifying a single sample takes approximately 20 seconds on an NVIDIA RTX Pro 6000 GPU.

Downstream Training. We train DeepLabV3 [28] with ResNet-50 and ResNet-101 backbones [29] using the synthetic data generated by our pipeline. Generated images and masks are resized from $1024\times 1024$ to $512\times 512$ for training. All models are trained using the same fixed optimization, loss, and augmentation protocol, with full details provided in Appendix D. We evaluate segmentation performance on the full VOC2012 validation set of 1,449 images.

Evaluation Metrics. We report the standard metrics of accuracy, mean intersection over union (mIoU), and mean average precision (mAP) for all experiments.

5.2. Zero-Shot Segmentation

We first evaluate HEADHUNTER as a zero-shot segmenter to assess the thesis that individual attention heads localize target concepts better than the corresponding head-aggregated output. Table 2 reports results on both benchmarks described in Sec. 5.1 alongside several interpretability baselines. HEADHUNTER achieves 73.2 and 81.2 mIoU on ImageNet-Segmentation and VOC2012 respectively, outperforming its closest baseline, ConceptAttention [14], by up to 4.7 points. This further supports the central hypothesis of this work: semantic localization is not uniformly distributed across heads, and aggregation can dilute stronger target-localizing heads, impacting segmentation performance. The margin over gradient- and attention-based methods such as Grad-CAM [30] and DAAM [31] is consistent with their coarser localization, while HEADHUNTER remains competitive with or ahead of stronger self-supervised baselines such as DINO SA [39] and OVAM [38].

5.3. Synthetic Dataset Evaluation

We next evaluate the proposed dataset synthesis pipeline by training downstream semantic segmenters on the generated image-label pairs. Table 3 compares the mIoU of our dataset against comparable works DiffuMask [8] and Dataset Diffusion [9] on the VOC2012 validation set.

With a ResNet-101 backbone, our dataset reaches the highest result of 64.9 mIoU, exceeding Dataset Diffusion by 1.1 points at the same number of training images, and matching their best reported result of 64.8 mIoU, which they obtain with four times as much data. Using the ResNet-50 backbone we reach 60.7 mIoU, marginally below Dataset Diffusion while using 75% less training data, and exceeding DiffuMask by 3.3 points despite the latter using six times more images. For reference, the real VOC2012 training set attains 77.4 and 79.9 mIoU with the ResNet-50 and ResNet-101 backbones respectively, highlighting the domain gap between synthetic and real-world data.

Per-Class Analysis.Table 4 reports per-class IoU for both backbones, with ResNet-101 showing consistent gains over ResNet-50 for nearly all classes. Our dataset performs particularly well on animals and vehicles, exceeding 80 IoU for aeroplane, car, cat, and cow, and almost reaching 90 IoU for bird and bus. For many other categories we record results in the 70-80 IoU range, and even achieve a notable 56.3 IoU for television (Dataset Diffusion [9] reports a score nearly half this). The weakest results are indoor furniture and thin-structure classes such as bicycles, which are particularly challenging to localize due to the resolution ceiling of the $64\times 64$ token grid. The open area between the spokes and rim often falls below the grid resolution and so the saliency map fills these regions in. While boundary refinement can sharpen the outer silhouette, it cannot recover these internal gaps, inflating the predicted mask and reducing IoU.

Table 5 compares selected per-class results (classes common to both baselines) against DiffuMask and Dataset Diffusion at matched backbones. Against DiffuMask (ResNet-50), our dataset achieves higher IoU on 10 of the 13 classes, including the difficult furniture classes chair (+9.0) and sofa (+9.7), as well as cow (+13.7) and dog (+9.1). Against Dataset Diffusion (ResNet-101), we lead on 8 of 13 classes, with large margins on car (+17.0), bird (+16.6), and cow (+11.5); the remaining classes are competitive, with Dataset Diffusion narrowly ahead on a few (e.g., horse, sheep).

Qualitative Results.Figure 5 presents representative synthetic image-mask pairs produced by the pipeline across a range of VOC classes. The annotations align very closely with object boundaries and correctly separate distinct instances in multi-object scenes, such as individual boats, birds, taxis, people, and monitors. These examples illustrate that the proposed pipeline produces accurate, well-localized masks across diverse object types, supporting the quantitative results reported.

Segmentation Predictions.Figure 6 shows predictions made on VOC2012 validation images from DeepLabV3 (ResNet-101 backbone) trained entirely on our synthetic dataset. The predicted masks exhibit close alignment with the GT across a range of classes and remain relatively robust in complex scenes with multiple object instances (e.g., boats and people).

5.4. Ablation Studies

We separate the ablations into two groups. First, Figure 7, Figure 8, Figure 9 and Figure 10 isolate the design choices underpinning HEADHUNTER. Second, Table 6 assesses the contribution of each stage of the dataset synthesis pipeline to semantic segmentation performance.

Number of Selected Heads. Figure 7 reports zero-shot single-class mIoU (VOC2012) as we aggregate the top-h attention heads ranked by HEADHUNTER. Selecting a single head performs best at 81.2 mIoU. Aggregating a small number of heads remains competitive, indicating that the proxy identifies a useful set of concept-localizing heads. Performance drops most sharply when averaging across all 24 heads, falling to 76.9 mIoU and indicating that the localization signal is concentrated in a small subset of heads that aggregation dilutes.

Self-Guided Head Selection. HEADHUNTER selects the single best concept-localizing attention head without GT supervision. To assess its effectiveness, we compare the selected head against an oracle that, for each image, picks the candidate head with the highest true IoU against the GT mask. In zero-shot single-class segmentation (VOC2012) the selected head recovers 91% of oracle mIoU (Figure 8), confirming that proxy identifies near-optimal heads on the majority of images.

Selection Distribution. To evaluate whether HEADHUNTER simply discovers a single globally useful attention head, we examine zero-shot single-class head selection frequency on VOC2012. Figure 9 shows that selections are broadly distributed and that the most frequent head is only selected for 2.2% of all evaluated images. This indicates that HEADHUNTER does not collapse to a single fixed head, but instead performs image-dependent selection from a heterogeneous set of concept-sensitive heads.

Concept Sets. The proxy mask uses the target, background, and scene-specific context concepts, giving salient non-target regions their own channels. Restricting $C_i^{proxy}$ to a binary $\{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t},\mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}\}$ set drops selection quality from 81.2 to 69.3 mIoU, with the multi-concept proxy producing a better mask on 88% of images. As shown in Figure 10, the additional context channels draw context activations away from the target, yielding a cleaner proxy mask and a stronger reference for selection.

Table 6 reports targeted dataset synthesis pipeline ablations on selected classes, with the remainder held fixed. For each ablation, only the dog and car synthetic samples are regenerated under the modified configuration; the downstream segmenter is then retrained on the resulting synthetic dataset and evaluated on the full VOC2012 validation set. These two classes provide complementary diagnostics: dog stresses intra-class appearance variation, while car stresses structured scenes with frequent distractors.

Mask Refinement. Removing mask refinement and training with coarse binary masks ($M_i$) degrades both classes relatively uniformly, reducing dog IoU by 14.8 points and car by 12.1 points (overall mIoU falls by 4.6), reflecting its role as a general boundary sharpening step.

Prompt Diversity. Replacing the diverse LLM-curated prompts with generic ones produces the largest single-class drop in the ablation, with dog IoU falling 20.1 points. As discussed in Sec. 4.1, dog appearance varies widely across breeds, and generic prompts collapse this variability toward a few standardized representations (Figure 3), leaving the segmenter poorly exposed to the intra-class diversity it must generalize over. Car is less dramatically affected but still drops 12.6 points, with overall mIoU down 5.2.

QA Verification. Removing the VLM-based QA stage most affects car, with a 13.7 point drop, the largest effect of any component on this class. Car scenes commonly contain distractor objects such as other vehicles and pedestrians (irrespective of using negative prompts), and capture support structures (e.g., the road) in saliency maps, corrupting the candidate mask. The VLM is instructed to reject such incorrectly localized samples, and without it the resulting noisy labels degrade the segmenter. Dog is less affected, dropping 11.4 points, with overall mIoU down 5.2.

5.5. Limitations

Resolution Ceiling. Concept saliency is computed over a $64\times 64$ token grid, and structures finer than a single token are difficult to localize. Thin and perforated objects such as bicycles are most affected, since the gaps between spokes and rim fall below the grid resolution and are filled in by the saliency map. This is a property of the saliency resolution rather than the head selection, and it limits performance on thin-structure classes.

Concept Entanglement. For semantically related concepts, attention tends to activate jointly rather than selectively due to an inherited limitation of the saliency extraction mechanism [14]. Because HEADHUNTER selects among heterogeneous heads, it can only recover a clean target when a well-localized head exists, but it does not perform explicit concept disentanglement and cannot reliably separate concepts that the underlying saliency conflates (Figure 11).

Domain Gap. The synthetic-to-real domain gap is inherent to generated datasets and is shared across comparable works [8,9]. As reported in Sec. 5.3, it has a marked impact on segmentation performance, with our dataset sitting roughly 15 points below real-only training on VOC2012.

Compute Cost. Dataset generation is computationally demanding, taking up to 20 seconds per image on an RTX Pro 6000 GPU. While the annotation step itself is relatively fast (approximately 2 seconds), diffusion sampling over 20 steps takes roughly 16 seconds, followed by about 2 seconds for VLM-based QA. Producing the final synthetic dataset used in this study took roughly 4 full days.

6. Conclusions

We presented HEADHUNTER, a training-free, open-vocabulary segmentation framework that selects a single concept-aligned attention head from a frozen MM-DiT using aggregate concept saliency as a self-guided proxy. We showed that aggregating attention heads discards fine boundary detail, whereas selecting the single best localizing head yields sharper masks, and that this selection can be performed without GT supervision, recovering 91% of oracle mIoU and performing competitively in zero-shot segmentation (81.2 mIoU on VOC2012 single-class and 73.2 on ImageNet-Segmentation). Built around this annotation mechanism, our fully automated dataset synthesis pipeline produces image-label pairs for target concepts without human intervention, with segmenters trained on datasets it generates reaching a competitive 64.9 mIoU (VOC2012) relative to comparable dataset generators.

Several directions remain open for future work. The $64\times 64$ token grid resolution limits the localization of very thin structures, which higher resolution attention or learned upsampling could address. Additionally, closely related concepts tend to get entangled, resulting in inaccurate segmentation; a natural next step would be to pursue disentanglement. The pipeline is also bounded by the training distribution of the generative model (e.g., FLUX.1-dev); extending it to specialized domains such as medical or industrial imagery would likely require adaptation, for example LoRA fine-tuning, but would enable the synthesis of highly specialized datasets. Finally, reducing the per-sample generation cost through faster diffusion samplers, and scaling the evaluation to larger benchmarks such as COCO [20], would broaden the applicability of the approach.