CLIP-Mono3D: End-to-End Open-Vocabulary Monocular 3D Object Detection via Semantic-Geometric Similarity

1. Introduction

Monocular 3D object detection estimates the 3D bounding boxes of objects from a single RGB image, an approach that offers compelling cost and deployment advantages over LiDAR-based or multi-sensor fusion systems. This capability is particularly valuable in large-scale applications such as autonomous driving, robotics, and augmented reality, where hardware simplicity and scalability are paramount. However, despite its practical appeal, monocular 3D detection remains significantly more challenging than its 2D counterpart. This difficulty stems primarily from the inherently ill-posed nature of depth recovery from a single viewpoint and the scarcity of large-scale, semantically diverse 3D annotations.

Recent years have witnessed substantial progress driven by benchmark datasets such as KITTI [1], SUN RGB-D [2], ScanNet V2 [3], and nuScenes [4]. Yet, as summarized in Table 1, these datasets are largely constrained in semantic scope, typically encompassing only 9 to 23 object categories. Most of these categories are restricted to traffic-related agents, such as cars and pedestrians, or common indoor items. In contrast, modern 2D detection benchmarks like COCO and Objects365 cover hundreds of categories, enabling robust generalization across diverse visual concepts. This semantic gap severely limits the applicability of current 3D detectors in open-world environments, where systems must recognize and localize objects beyond a fixed, predefined set.

To bridge this gap, open-vocabulary 3D object detection has emerged as a promising research direction. The goal is to enable models to detect and localize objects described by arbitrary natural language prompts, including those unseen during training. Existing approaches, however, predominantly follow a two-stage paradigm. They first leverage a pre-trained 2D open-vocabulary detector to extract semantic proposals, which are then fed into a class-agnostic 3D detector for geometric regression, as illustrated in Figure 1(a). While effective, this design introduces several practical limitations: (i) it requires heavy reliance on external 2D detectors and their associated supervision; (ii) it involves multi-stage training pipelines that are difficult to optimize jointly; and (iii) many methods still depend on point cloud priors or depth estimators, undermining the simplicity of the monocular setting.

Recent attempts, such as OVMono3D [5], have sought to mitigate these issues by incorporating foundation models like SAM for segmentation priors or monocular depth estimators. Nevertheless, they remain fundamentally dependent on pre-trained 2D detectors and auxiliary data sources, preventing true end-to-end training from 3D supervision alone.

In this work, we present CLIP-Mono3D, a novel, end-to-end trainable framework for open-vocabulary monocular 3D object detection. Built upon the MonoDGP architecture [6], our method integrates semantic knowledge directly into the 3D detection pipeline by leveraging a pre-trained FG-CLIP visual-language encoder [7]. Unlike prior approaches, CLIP-Mono3D eliminates the need for an external 2D detector by fusing CLIP-derived visual-semantic features with geometric representations via cross-modal attention. By initializing detection queries using language embeddings, our model achieves zero-shot generalization to novel categories without additional 2D supervision, representing a significant step toward practical and generalizable monocular 3D perception.

To facilitate research in this under-explored direction, we further introduce OV-KITTI, a new benchmark that extends the original KITTI dataset with 40 additional object categories, including animals and everyday items. As shown in Table 1, OV-KITTI not only expands semantic coverage but also provides more diverse shape and scale priors, which help alleviate the depth ambiguity inherent in monocular setups. The dataset is carefully curated to ensure balanced distributions between base and novel categories, enabling fair evaluation of open-vocabulary generalization.

The significance of our work lies in three key contributions:

1.

Unified End-to-End Architecture: We propose CLIP-Mono3D, an end-to-end framework that unifies semantic and geometric reasoning. By introducing a Cross-Modal Semantic-Geometric Fusion module, we inject fine-grained semantic clues into geometric features via a lightweight residual connection, enhancing semantic awareness without disrupting pre-trained geometric cues.

2.

Language-Aware Query Initialization: We design a novel query initialization strategy that converts 2D semantic heatmaps into explicit 3D query positions. This mechanism significantly improves 3D center localization and recall for open-vocabulary objects compared to standard learned queries.

3.

OV-KITTI Benchmark and Evaluation: We introduce OV-KITTI, a large-scale benchmark with controlled semantic and size distributions. Extensive experiments on OV-KITTI, KITTI, and Argoverse demonstrate that CLIP-Mono3D achieves competitive performance in both closed- and open-vocabulary settings, paving the way for deployment in truly open-world scenarios.

2. Related Work

2.1. Monocular 3D Object Detection

Monocular 3D object detection localizes 3D bounding boxes from single RGB images, but its development lags behind point cloud and multi-view methods due to ill-posed depth estimation. Current approaches include geometry-based methods and depth-assisted techniques. Geometry-based methods leverage geometric consistency: some reduce the task to 2D detection with 3D-2D projection for center calculation [8,9], while others directly regress 3D boxes via keypoints [10,11]. Enhanced variants incorporate geometric priors like ground planes [12], shape awareness [13], uncertainty modeling [14], and height cues [15]. To address depth perception limitations, depth-assisted methods utilize pseudo-LiDAR representations [16,17], LiDAR-supervised depth estimation [18], frustum discretization for BEV mapping [19], and depth pre-training [20]. Transformer-based designs further integrate visual/depth features [21] and employ depth-guided decoding [22].

2.2. Open-Vocabulary Object Detection

Open-vocabulary object detection handles both seen and unseen categories to accommodate real-world diversity. With CLIP’s rise, 2D open-vocabulary detection has advanced through knowledge distillation from image-text pairs [23], pseudo-label generation [24,25], pretrained encoders (CLIP [26,27,28,29,30] or BERT [31,32,33]), and generative modeling [34,35]. Transformer-based methods dominate for their effective integration of text encoders.

2.3. Open-Vocabulary 3D Object Detection

Extending to 3D space, this task detects unseen categories with higher complexity. Early works used unsupervised clustering for open-set detection [36]. Recent CLIP-inspired approaches include pseudo-label methods where OV3DET [37] trains class-agnostic detectors with 2D-generated labels then aligns 3D, image and language features. Cross-modal alignment techniques combine 3D geometric and 2D semantic priors without 2D detectors [38], or enrich datasets via cross-domain alignment [39]. Knowledge propagation frameworks cyclically transfer 2D semantic knowledge to 3D and geometric cues to 2D [40]. Point cloud-language alignment further bridges modalities [41,42].

3. Method

3.1. Task Definition and Overall Architecture

Open-vocabulary monocular 3D object detection aims to estimate 3D bounding boxes $\mathcal{D}={\left\{{\mathcal{O}}_i\right\}}_{i=1}^N$ from a single RGB image $\mathcal{I}\in {\mathbb{R}}^{H\times W\times 3}$, guided by a set of arbitrary text prompts $\mathcal{T}=\{{\tau }_1,\dots ,{\tau }_K\}$. Each object ${\mathcal{O}}_i$ is parameterized by its 3D center $(x_i,y_i,z_i)$, dimensions $(l_i,w_i,h_i)$, orientation ${\theta }_i$, and a semantic similarity vector ${\mathbf{s}}_i\in {[0,1]}^K$. The similarity score for each prompt is computed as:

$${\mathbf{s}}_i\left[k\right]=\sigma \left({\varphi }_{\mathrm{img}}{\left({\mathbf{p}}_i\right)}^{\top }{\varphi }_{\mathrm{text}}\left({\tau }_k\right)\right),$$

(1)

where $\sigma$ denotes the sigmoid function, ${\varphi }_{\mathrm{img}}$ and ${\varphi }_{\mathrm{text}}$ represent the image and text encoders (e.g., CLIP), and ${\mathbf{p}}_i$ denotes the region-level features associated with the i-th object. This formulation enables the detection of novel categories by replacing fixed-set classification with open-ended semantic matching.

As illustrated in Figure 2, CLIP-Mono3D extends the MonoDGP architecture [6]. A ResNet-50 backbone first extracts multi-scale features ${\left\{{\mathbf{F}}_i\right\}}_{i=0}^3$, which are enhanced by a Region Segmentation Head (RSH) to produce ${\mathbf{F}}_V$. These features are fused with CLIP-derived semantics (Sec. Section 3.2) and processed by a depth predictor to generate ${\mathbf{F}}_D$. Dual transformer encoders independently process these streams, followed by a 2D visual decoder and a 3D depth-guided decoder. To provide explicit geometric priors, language-aware queries (Sec. Section 3.3) are initialized from CLIP similarity maps. Final predictions are refined via geometric depth correction and scored against text embeddings.

3.2. Cross-Modal Feature Fusion

To bridge the semantic gap between vision and language modalities, we introduce a lightweight cross-modal fusion mechanism that injects text-guided spatial priors into the visual backbone. Given an input image $\mathcal{I}$ and prompts $\mathcal{T}$, we extract dense visual features ${\mathbf{F}}_{\mathrm{img}}\in {\mathbb{R}}^{14\times 14\times d}$ and global textual embeddings ${\mathbf{F}}_{\mathrm{txt}}\in {\mathbb{R}}^{K\times d}$ using a frozen CLIP encoder. Freezing CLIP’s parameters is essential to preserve its zero-shot generalization and prevent the loss of rich semantic knowledge during 3D detection training.

Semantically relevant regions are identified by computing a spatial similarity map $\mathcal{S}\in {\mathbb{R}}^{14\times 14}$ through the aggregation of cosine similarities across all text tokens:

$$\mathcal{S}(u,v)={\displaystyle \frac{1}{K}}\sum _{k=1}^K{\displaystyle \frac{{\mathbf{F}}_{\mathrm{img}}^{(u,v)}·{\mathbf{F}}_{\mathrm{txt}}^{\left(k\right)}}{\parallel {\mathbf{F}}_{\mathrm{img}}^{(u,v)}\parallel \parallel {\mathbf{F}}_{\mathrm{txt}}^{\left(k\right)}\parallel }}.$$

(2)

This map functions as a soft attention mask, highlighting regions aligned with the linguistic input. Unlike post-hoc filtering methods, our approach integrates this signal early in the feature hierarchy, allowing semantic guidance to inform all downstream components.

The similarity map $\mathcal{S}$ is bilinearly upsampled to match the dimensions of the intermediate feature map ${\mathbf{F}}_V^{\left(2\right)}\in {\mathbb{R}}^{C\times H_2\times W_2}$. It is then processed by a convolutional module ${\Gamma }_{\mathrm{fuse}}$—comprising two $3\times 3$ convolutions with ReLU activations—to refine its spatial structure and match the channel dimensions. The final fused feature map is obtained via an additive residual connection:

$${\mathbf{F}}_V^{\mathrm{enh}\left(2\right)}={\mathbf{F}}_V^{\left(2\right)}+{\Gamma }_{\mathrm{fuse}}\left(\mathrm{Upsample}\left(\mathcal{S}\right)\right).$$

(3)

This residual design ensures that primary geometric and structural information remains intact, which is critical for accurate 3D localization. This early fusion strategy creates a "semantic spotlight" that benefits both the RSH and the subsequent query initialization stage (Sec. Section 3.3).

3.3. Language-Aware Query Initialization

Standard DETR-like architectures typically initialize object queries as content-agnostic learnable embeddings, which can lead to slow convergence in complex scenes. To address this, we propose a language-aware query initialization strategy that transforms 2D semantic probability maps into explicit 3D geometric anchors.

As shown in Figure 3, the initialization involves three steps. First, we interpret the CLIP similarity map $\mathcal{S}$ as a spatial probability distribution and identify $N_q$ locations $\mathcal{P}={\left\{(u_k,v_k)\right\}}_{k=1}^{N_q}$ with the highest activations. These serve as candidate 2D centers for potential 3D objects.

Second, we employ F.grid_sample to sample local descriptors ${\mathbf{P}}_{\mathrm{keypoint}}\in {\mathbb{R}}^{N_q\times C}$ from the enhanced feature map ${\mathbf{F}}_V^{\mathrm{enh}\left(2\right)}$ at coordinates $\mathcal{P}$. This differentiable operation ensures that each query is initialized with features specific to its corresponding object region.

Third, a global semantic prior is distilled from these descriptors via a two-layer MLP:

$${\mathbf{Q}}_{\mathrm{prior}}=W_2\delta \left(W_1·{\displaystyle \frac{1}{N_q}}\sum _{k=1}^{N_q}{\mathbf{P}}_{\mathrm{keypoint}}^{\left(k\right)}\right).$$

(4)

The base queries $\mathbf{Q}\in {\mathbb{R}}^{N_q\times 2d}$ consist of a content component ${\mathbf{Q}}_c$ and a positional component ${\mathbf{Q}}_p$. We use sine-cosine positional encodings of $\mathcal{P}$ to initialize ${\mathbf{Q}}_p$, while ${\mathbf{Q}}_c$ is enhanced by the global prior:

$${\mathbf{Q}}_{\mathrm{content}}={\mathbf{Q}}_c+{\mathbf{Q}}_{\mathrm{prior}},\mathbf{Q}=[{\mathbf{Q}}_{\mathrm{content}};{\mathbf{Q}}_p].$$

(5)

By grounding queries in semantically verified regions, we transform the detection process from exhaustive spatial searching to targeted localization. This "semantic priming" accelerates training convergence and reduces attention to background clutter. Importantly, the CLIP-derived prior provides a strong inductive bias for open-world concepts. During inference, if no text is provided, the system reverts to base queries (${\mathbf{Q}}_{\mathrm{prior}}=\mathbf{0}$) to maintain backward compatibility.

3.4. Open-Vocabulary Detection Head and Loss

The detection head enables open-vocabulary classification by computing semantic similarity between decoder outputs and projected text embeddings. For each output $\mathbf{h}\in {\mathbb{R}}^{d_h}$, we project it into the CLIP feature space: ${\mathbf{v}}^{\prime }=\mathrm{norm}\left({\mathbf{W}}_v\mathbf{h}\right)$, where ${\mathbf{W}}_v\in {\mathbb{R}}^{512\times d_h}$ and $\mathrm{norm}(·)$ denotes ${\ell }^2$ normalization. Similarly, text embeddings ${\mathbf{t}}_{\mathrm{CLIP}}$ are projected via a learnable matrix ${\mathbf{W}}_t$. This dual-projection design mitigates the domain gap between the detector’s internal representations and CLIP’s pre-trained embeddings.

The similarity score is computed as a temperature-scaled dot product, $s=\tau ·{\mathbf{v}}^{\prime \top }{\mathbf{t}}^{\prime }$, where the learnable temperature $\tau =exp\left(\gamma \right)$ controls the distribution concentration. To handle background regions, we introduce a learnable background embedding ${\mathbf{t}}_{bg}$ appended to the text embeddings.

Training follows a bipartite matching strategy using the Hungarian algorithm. The matching cost incorporates both geometric and semantic terms:

$$C_{\mathrm{match}}={\lambda }_g\left(\parallel \Delta {\mathbf{c}}_{3\mathrm{d}}{\parallel }_1+{\parallel \Delta {\mathbf{b}}_{2\mathrm{d}}\parallel }_1+(1-\mathrm{GIoU})\right)-{\lambda }_{s}s.$$

(6)

Integrating the similarity score s into the matching process ensures that predictions are assigned based on both spatial accuracy and semantic coherence, significantly enhancing generalization to unseen classes.

For matched pairs $\mathcal{M}$, we apply a contrastive loss:

$${\mathcal{L}}_{\mathrm{contrast}}=-{\displaystyle \frac{1}{\left|\mathcal{M}\right|}}\sum _{(i,j)\in \mathcal{M}}log{\displaystyle \frac{exp(s_{ij}/\tau )}{{\sum }_{k}exp(s_{ik}/\tau )}}.$$

(7)

The total objective combines this with geometric regression losses ${\mathcal{L}}_g$:

$${\mathcal{L}}_{\mathrm{total}}={\lambda }_c{\mathcal{L}}_{\mathrm{contrast}}+\sum {\lambda }_g{\mathcal{L}}_g.$$

(8)

This co-design ensures that gradients from the contrastive loss are applied to the most semantically relevant predictions, creating a robust optimization cycle that yields a model both geometrically precise and semantically aware.

4. OV-KITTI Benchmark

Most existing monocular 3D object detection models rely on the KITTI dataset for training and evaluation. However, this dataset presents several critical limitations. KITTI contains only nine object categories, and prior works predominantly evaluate models on the “Car” category due to the scarcity of other instances. In open-vocabulary or zero-shot learning settings, a common practice involves training on “Car” and “Cyclist” classes while attempting to transfer detection capabilities to the “Pedestrian” category. Nevertheless, as illustrated in Figure 4(c), the 3D bounding box dimensions of the “Car” category differ significantly from those of “Pedestrian” and “Cyclist”. This discrepancy leads to an inherent bias where the detection of “Pedestrian” largely relies on knowledge spillover from “Cyclist” rather than benefiting from the rich feature learning associated with the dominant “Car” category. We argue that this imbalance in both category support and scale distribution within KITTI hinders the development of generalizable 3D detection models.

To overcome these limitations, we introduce OV-KITTI, an augmented benchmark based on the KITTI dataset designed specifically for open-vocabulary monocular 3D detection. We enrich the original dataset with 40 additional object categories sourced from Objaverse [43]. These categories encompass a diverse set of animals and household items. The new objects are carefully selected to avoid semantic overlap with existing traffic participants in KITTI, such as cars and pedestrians, thereby enabling precise supervision and evaluation.

4.1. Dataset Construction

The construction of OV-KITTI follows a systematic multi-step pipeline:

(1) Bounding Box Design: For each new category, we define physically plausible size constraints for its 3D bounding box. During rendering, the actual box dimensions are randomly sampled within this predefined range to ensure variability. (2) Scene Integration: Using Blender, we render 3D meshes into real driving scenes from KITTI. Objects are placed on the ground plane with random rotation, scaling, and translation. We explicitly enforce constraints to ensure no physical overlap with existing objects in the scene. (3) Stereo Rendering: We generate stereo-consistent left and right views for each modified scene. Annotations are provided in the standard KITTI format to ensure seamless compatibility with existing detection frameworks. (4) Balance Control: Special care is taken to balance the 3D box size distributions between known and unknown classes, as shown in Figure 4(a) and (b). This step reduces potential size-related biases and supports a fair evaluation of model generalization capabilities.

4.2. Category Statistics

We divide the 40 new classes into known (training) and unknown (testing) categories. The known set comprises 32 classes, consisting of 19 household items and 13 animals, while the unknown set contains 8 classes, including 5 items and 3 animals, for zero-shot evaluation. This split ensures diversity and scale balance across training and testing phases. The full category list and instance counts are provided in Table 2. As shown in Figure 5, OV-KITTI contains high-quality renderings of diverse objects under varied scales and contexts. This provides a challenging yet realistic testbed for evaluating open-vocabulary 3D detection performance.

5. Experiments

5.1. Experimental Setup

5.1.1. Datasets and Metrics

We evaluate on KITTI and OV-KITTI. For KITTI, we use standard splits (3712 train, 3769 val). For OV-KITTI, we use 5936 training samples (32 classes) and 1545 test samples (8 classes). Original categories are excluded from open-vocabulary evaluation. We use AP at 40 recall as the metric: $A{P}_{3D}$ and $A{P}_{BEV}$ for closed-vocabulary; $A{P}_{15}$ per category for open-vocabulary.

5.1.2. Settings

We follow MonoDETR-style augmentation and FG-CLIP image preprocessing. The model is trained on a single NVIDIA RTX 4090D for 200 epochs with batch size 8 using PyTorch.

5.2. Closed-Vocabulary Results

As shown in Table 3, our method achieves state-of-the-art performance in the standard monocular 3D object detection task. On the KITTI val set, our approach attains 31.40% $A{P}_{3D}$ under the “easy” difficulty level, surpassing previous methods by a clear margin: +2.84 percentage points over MonoDETR [22] and +1.72 over MonoDGP [6]. The improvements are consistent across moderate and hard difficulty levels, demonstrating enhanced robustness in detecting occluded and distant objects. In BEV detection, our model also sets a new benchmark with 39.47% $A{P}_{BEV}$ (easy), outperforming both baselines by over 1.5 points.

These gains can be attributed to our improved 2D feature representation enriched with CLIP-derived semantics, which provides stronger cues for depth estimation and spatial localization. Unlike purely geometric reasoning methods, our integration of semantic context allows the detector to better disambiguate scale and position, especially in low-texture or cluttered scenes.

On the OV-KITTI dataset, while the focus shifts toward open-vocabulary generalization, we still report strong closed-vocabulary performance. Our $A{P}_{25}$ reaches 14.44%, exceeding MonoDGP by 1.11%, which indicates that our enhancements do not compromise detection accuracy on seen classes. This balance between specialization and generalization is crucial for real-world deployment, where systems must handle both known and emerging object categories.

As shown in Table 4, we compare the efficiency metrics with baseline methods. It is important to note that while our framework incorporates a frozen CLIP image encoder of approximately 150 M parameters to extract semantic features, these parameters do not require gradient updates. Consequently, our trainable parameter count of 43.48 M is nearly identical to that of MonoDGP at 43.33 M. This minimal increase of approximately 0.15 M confirms that the performance gains stem from our effective semantic-geometric alignment design rather than a brute-force increase in model capacity. Although the integration of the visual encoder introduces a marginal latency increase, the inference speed of 51ms per frame remains sufficient for real-time autonomous driving applications.

5.3. Open-Vocabulary Results

The true strength of our method lies in its ability to detect objects from previously unseen categories, a key challenge in open-vocabulary 3D detection. As shown in Table 5, our method achieves an overall 5.81% AP on the eight unseen categories in OV-KITTI, outperforming the fine-tuned variant OVMono3D^† by +1.21%.

A critical observation is that fine-tuning on the known training set provides substantial gains for baseline methods, particularly for geometrically complex categories. As shown in the “Gain from FT” rows, OVMono3D^† improves by a remarkable +7.02 points on “tiger” and +1.61 on “giraffe” compared to its non-fine-tuned version. This confirms that exposure to 3D shape priors during fine-tuning is essential for accurate localization of categories largely absent from standard 3D datasets.

Despite this significant boost from fine-tuning, our method still achieves superior performance across most categories. We outperform OVMono3D^† by +2.73 on “tiger” and +1.04 on “giraffe”, demonstrating that our direct integration of vision-language semantics enables better generalization to novel biological shapes, without relying on cascaded 2D detectors or fine-tuning heuristics. For smaller or less frequent objects like “bucket” and “dog”, our method also shows consistent improvements.

The only exception is “wardrobe” where OVMono3D^† retains a slight edge. We attribute this to its stronger pre-trained knowledge of household items and the relatively simple geometric structure of wardrobes, which are more easily captured by existing 2D open-vocabulary detectors. Overall, our results validate that grounding 3D detection directly in language semantics, rather than relying on fine-tuned 2D detectors, leads to superior generalization, especially for structurally diverse and underrepresented categories like animals.

5.4. Visualization of Cross-Modal Alignment

To elucidate how language priors guide the detection process, we visualize the CLIP-based similarity maps $\mathcal{S}$ and their influence on query initialization in Figure 6. As shown in the middle column, $\mathcal{S}$ effectively serves as a semantic attention mechanism, highlighting regions aligned with class-specific prompts such as “car” and “wheelchair”. Notably, for the rare class “wheelchair”, which lacks 3D annotations during training, the model identifies plausible candidates by leveraging semantic cues (e.g., human silhouettes with wheels), demonstrating robust open-vocabulary generalization.

The right column illustrates the top-$N_q$ keypoints selected for query initialization. These points concentrate on the object’s spatial extent, confirming that our language-aware sampling effectively localizes semantically meaningful regions. These keypoints provide informed feature priors that prime the object queries, enabling the decoder to focus on relevant content from the initial stages.

This visualization validates two key advantages: (1) the similarity map bridges language semantics and visual geometry for zero-shot grounding; (2) language-aware initialization mitigates monocular depth ambiguity by providing semantically grounded spatial anchors.

5.5. Ablation Studies

To evaluate the contribution of each proposed component, we conduct comprehensive ablation studies on the OV-KITTI benchmark. Unless otherwise specified, performance is measured by $A{P}_{25}$ for seen classes (CV) and $A{P}_{15}$ for unseen categories (OV).

Core Components and Query Initialization. As summarized in Table 6, each module consistently improves performance. While feature fusion enhances the base representation, the language-aware query initialization provides the most significant gain for open-vocabulary generalization (+1.12% OV AP). This suggests that grounding queries in semantic regions from the early decoding stages is more effective for novel objects than relying on generic learnable embeddings.

Regarding the specific initialization design in Table 7, we find that using 50 queries with a global prior aggregated via mean-pooling yields optimal results. This global context informs each query of the overall semantic landscape, preventing them from over-focusing on isolated, potentially noisy local regions. In contrast, max-pooling over-emphasizes the single most salient activation, leading to poorer generalization for diverse scenes. We also compare VLM backbones: FG-CLIP (Frozen) achieves the highest accuracy, while fine-tuning it via LoRA [47] leads to a performance drop. This indicates that the 3D dataset’s scale is insufficient to update the VLM without causing catastrophic forgetting of its pre-trained open-world semantic knowledge.

Feature Fusion Strategy and Domain Robustness.Table 8 explores fusion architectures. We find that additive residual connections at res2 outperform simple concatenation. This additive design acts as a calibrated semantic spotlight that preserves the geometric integrity of intermediate features, whereas concatenation may introduce noise that disrupts sensitive 3D regressions. Injecting semantic guidance early in the feature hierarchy (stage 2) is crucial for informing downstream depth estimation and query sampling.

To investigate whether the model overfits to synthetic artifacts, we conduct a data-mixing study in Table 9. We progressively introduce real-world KITTI objects into the training pipeline. Crucially, to ensure a fair comparison, the Seen AP is evaluated exclusively on the original synthetic classes. The results show that adding real-world instances with distinct lighting and textures leads to negligible fluctuations in Unseen AP. This stability confirms that CLIP-Mono3D learns generalized semantic concepts rather than low-level texture priors. The drop observed in Row (3) is likely due to the extreme class imbalance introduced by real-world Cars, which biases the optimization away from rare open-vocabulary classes.

5.6. Visualization

To provide intuitive insights into the performance of our CLIP-Mono3D model, we present qualitative visualizations of detection results on both the KITTI and OV-KITTI datasets, as shown in Figure 7 and Figure 8.

Figure 7 visualizes detection results on KITTI. Our method exhibits higher recall for distant vehicles compared to MonoDETR. While sharing a similar architecture with MonoDGP, the integration of CLIP-derived semantics improves 2D detection performance, which subsequently leads to more accurate distance regression. These results confirm that semantically enriched 2D features provide robust spatial priors for 3D localization, especially for long-range objects where geometric cues are often ambiguous.

Figure 8 compares CLIP-Mono3D with OVMono3D on the OV-KITTI benchmark. While OVMono3D occasionally yields more precise 2D boxes, our framework demonstrates superior 3D geometric reasoning. Specifically, our depth-guided transformer and language-aware queries enable more accurate depth estimation (row 2, “bench”), enhanced bounding box integrity for complex shapes (row 3, “giraffe”), and significantly reduced missed detections for small objects (row 4, “dog”), highlighting the robustness of our end-to-end semantic-geometric alignment in open-world scenarios.

5.7. Real-World Scenario Verification

To verify generalization in unconstrained real-world environments, we extend our evaluation to the standard KITTI and Argoverse datasets. For KITTI, we adopt a cross category split by training solely on Car and Pedestrian and evaluating zero shot performance on Cyclist. For Argoverse, we train on common categories and evaluate on seven distinct novel classes.

Settings. To adapt to the complex nature of the Argoverse dataset, we modified our query initialization strategy. Specifically, we shifted from the global mean aggregation used in OV-KITTI to a discrete spatial assignment paradigm. This ensures that queries are initialized at distinct spatial locations, effectively preventing multiple queries from collapsing onto a single salient object in complex scenes.

Results. As shown in Table 10, on the KITTI dataset, despite the extreme scarcity of training categories limiting the learning of generalized 3D shape priors, our end-to-end framework still achieves competitive performance. It is worth noting that the baseline OVMono3D utilizes an external open vocabulary 2D detector for strong 2D localization, whereas our method operates seamlessly without explicit 2D bounding box guidance. On the larger scale Argoverse dataset detailed in Table 11, our method demonstrates robust generalization. Supported by the updated initialization strategy, our model successfully guides geometric reasoning for novel objects, achieving promising results across various categories and obtaining a higher overall average precision compared to the baseline.

To further illustrate the practical performance, we provide qualitative comparisons in Figure 9. In the first column, our method successfully detects the occluded moped while maintaining the detection completeness of the surrounding objects. In the third column, our model manages to recall the challenging animal class in the distance without missed detections. We acknowledge that the geometric regression for highly irregular novel classes like animals still requires further improvement and exhibits certain deviations. This is primarily due to the unreasonable category distribution inherent in the dataset and the extreme difficulty of these highly crowded scenes.

6. Conclusion

We present CLIP-Mono3D, an end-to-end open-vocabulary monocular 3D detector that directly integrates vision-language semantics without relying on pretrained 2D detectors. To facilitate comprehensive evaluation, we introduce OV-KITTI, a new benchmark with balanced category distributions. Furthermore, we validate our framework on real-world datasets including KITTI and Argoverse. Extensive experiments demonstrate that our method achieves competitive performance in both closed- and open-vocabulary settings, showing robust generalization across diverse environments. We hope this work inspires further research in semantic-geometric fusion for 3D perception.