Task-Decomposed Language Supervision for End-to-End Autonomous Driving

1. Introduction

End-to-end models for autonomous driving build a Bird's-Eye-View (BEV) representation from multi-camera images and learn object detection, map perception, motion prediction, and ego planning within a single network. UniAD [1], VAD [2], and GenAD [3] showed that such integration reduces error accumulation relative to modular pipelines and improves planning performance. In particular, GenAD formulates future trajectories as a generation problem and achieves strong planning performance through instance-centric representations and latent-variable trajectory generation.

However, these models encode the scene primarily through geometric and motion cues, and do not explicitly represent high-level semantic reasoning such as "which object is dangerous" or "what is a safe driving behavior." To address this, research combining LLMs/VLMs has emerged, falling into two categories. (i) Inference-time LLM-driven methods (DriveVLM [4], LMDrive [5], etc.) are highly expressive but challenging for real-time deployment due to the cost of autoregressive generation. (ii) Training-time language-supervision methods (VLP [6], VLM-AD [7], DiMA [8], etc.) remove the LLM at inference and are therefore efficient, but they generally transfer language-derived knowledge through feature alignment, auxiliary semantic targets, or cross-modal distillation rather than directly coupling stage-specific reasoning states to the corresponding task queries.

The two families above expose a design tension that motivates this work. Inference-time methods keep the language model in the loop and can therefore reason explicitly about a scene, but the cost of autoregressive decoding is incompatible with the update rates required for closed-loop control, and the additional memory footprint of the language model competes with the perception stack for on-board resources. Training-time supervision methods remove this cost at inference by transferring language-derived knowledge to the driving model during training. Existing approaches employ various mechanisms, including feature alignment, auxiliary semantic targets, and cross-modal distillation at different levels of the driving stack. Although effective, these methods generally use language representations as supervision targets or auxiliary signals rather than directly incorporating stage-specific reasoning states into the task queries that execute the corresponding driving subtasks. This distinction motivates a closer coupling between the structure of language reasoning and the query-based task hierarchy of an E2E driving model.

This motivates two design choices in TaLS. First, rather than using language representations only as alignment targets or auxiliary supervision, we inject the representation of each reasoning stage directly into the query representation used by the corresponding driving component. Second, we preserve the sequential structure of the reasoning process—perception, motion, risk, and planning—so that later language states are conditioned on earlier stages before being delivered to the associated task queries. TaLS therefore focuses not merely on transferring language knowledge, but on coupling task-decomposed reasoning states with the query-based computation of the E2E driving stack.

This paper belongs to category (ii). While recent work also supervises across pipeline stages (e.g., VERDI [9]), TaLS differs in the specific supervision mechanism and in its analysis of that mechanism. We restructure the DriveLM QA into a four-stage chain (perception → motion → risk → planning) (TD-QA) and, during training, inject the LLM hidden state at each stage's answer position directly into the corresponding task decoder's queries as a gate-free residual. This aligns the task hierarchy of the E2E model (detection → prediction → planning) with the stages of language reasoning in a stage-aligned manner. After training, the autoregressive LLM is removed, and the inference graph retains only the BEV Q-Former and the BEV token augmentation modules. We further analyze, on the trained model, which planning gains persist in the retained weights after the language branch is removed.

Hidden-state injection versus language distillation. Conventional language distillation [10] matches the student's features to a teacher's output through an auxiliary loss: the language representation is a target that the driving network is regularized toward, and the choice of what to align (a pooled caption embedding, a similarity score) is made once, globally. Our stage-wise injection is mechanistically different. The language hidden state is not a loss target but an additive term placed directly on the decoder queries during the forward pass, so the driving loss back-propagates through the injected representation into the decoder weights. The decoder therefore learns to use a query that has already been shifted by the language reasoning, and at inference — when the injection is removed — the weights retain the behavior that was shaped under that shift. In distillation, the language signal supervises a representation; in our injection scheme, it directly participates in the forward computation and can shape the decoder weights through the driving loss. Our mechanism analysis (Section 3.5) therefore examines which performance gains persist after the language branch is removed.

TD-QA versus generic chain-of-thought supervision. A chain-of-thought (CoT) [11] trace supervises a single output — the planner, or a language head — with a linear sequence of reasoning steps, and the steps are consumed only through their effect on that one output. TD-QA differs in that the chain is aligned with the task hierarchy of the driving network at the corresponding subtask level and each stage is injected into the decoder that owns the corresponding sub-task: the object stage into the detection queries, the motion stage into the motion queries, and the risk and planning stages into the ego query. The decomposition is thus not only a prompting device but a wiring principle, which provides explicit stage-to-module traceability (each stage supervises an identifiable module) and lets the autoregressive chain condition later stages on earlier ones (the risk judgment is generated before, and therefore conditions, the planning stage). We quantify what this structure contributes, relative to undecomposed language supervision, in Section 4.4.

The contributions of this paper are as follows.

• TD-QA (task-decomposed language supervision). We propose a language supervision technique that decomposes driving QA into a four-stage reasoning chain aligned with the task hierarchy and injects the LLM hidden state of each stage into the corresponding task decoder during training.
• LLM-free inference. The 1.5B-scale LLM and the stage-wise injection are used only during training; at inference the autoregressive LLM is removed, so the model operates without autoregressive generation cost.
• Mechanism analysis. Using a prune ablation on the trained model, we analyze which L2 and collision gains depend on the retained BEV token path (Q-Former, SLG, Fusion) and which persist after this path is disabled.

2. Related Work

End-to-end autonomous driving. UniAD [1] unified multiple tasks in a query-based manner; VAD [2] improved efficiency with a vectorized scene representation; GenAD [3] treated future trajectories as a generation problem and raised planning performance via instance-centric interaction and latent-variable generation. This work adopts GenAD as the base model and adds language supervision on top of it.

Open-loop evaluation on nuScenes. The planning results discussed above are obtained under open-loop evaluation, in which the planned trajectory is compared against a recorded human trajectory without executing it. Two studies have shown that this protocol is easier to satisfy than it appears. Zhai et al. [12] reported that a multi-layer perceptron taking only the ego vehicle’s past trajectory and velocity as input, without any camera or LiDAR perception, attains planning scores comparable to contemporary end-to-end systems, indicating that the benchmark rewards extrapolation of ego motion. Li et al. [13] reached a similar conclusion and showed that supplying the ego status dominates the reported metrics; for this reason VAD [2] deactivates ego status for a fair comparison. A second and separate difficulty is that the reported numbers depend on how the metrics are aggregated over the predicted horizon: UniAD [1] evaluates the value at each endpoint, whereas ST-P3 [14] and VAD [2] average over all waypoints up to that horizon, so the same model can differ substantially between the two conventions. We therefore report both aggregation protocols explicitly (Section 3.7).

LLMs/VLMs for driving — inference-time-driven. DriveGPT4 [15], LMDrive [5], and DriveVLM(-Dual) [4] place the LLM/VLM inside the inference loop to gain explainability and generalization, but their real-time performance is limited by autoregressive generation cost. SimpleLLM4AD [16] similarly decomposes driving into a multi-stage QA chain (perception, prediction, planning, behavior) connected as a graph, but reasons stage-by-stage with the VLM at inference. In contrast, TaLS uses its stage-wise QA only during training and removes the autoregressive LLM at inference.

LLMs/VLMs for driving — training-time supervision. VLP [6] trains a vision-based planner jointly with a frozen language encoder, aligning agent-level BEV features and the ego planning query with text features derived from ground-truth labels through contrastive learning; VLM-AD [7] prompts a vision-language model to produce free-form reasoning and structured action annotations for each frame and supervises auxiliary heads attached to the ego feature with them; and DiMA [8] uses the vision-based planner as a tokenizer for a multi-modal language model and distills features from the language model’s planning head into the planning transformer under a set of surrogate tasks. In all three the language branch is discarded, or becomes optional, at inference. Closest to our setting, VERDI [9] aligns intermediate perception, prediction, and planning representations with VLM-generated reasoning features at the corresponding stages. These methods share the advantage of being LLM-free (or LLM-optional) at inference. TaLS also belongs to this category, but differs in two respects: (i) rather than aligning module outputs to generated explanation text or distilling a single global feature, TaLS injects the answer-end LLM hidden state of each reasoning stage directly into the corresponding decoder queries as a gate-free residual during training; and (ii) we provide a mechanism analysis examining which planning gains persist in the retained decoder weights and which depend on the inference-time BEV token path (Section 3.5, Section 4.4).

Bridging visual features and language models. Connecting a visual encoder to a frozen language model requires compressing dense features into a small number of tokens that the language model can consume. BLIP-2 [19] introduced the Querying Transformer, a set of learnable queries that cross-attend to image features and emit a fixed-length sequence in the embedding space of the language model. We adapt this design to the bird’s-eye-view domain (Section 3.2), using the resulting tokens both as the language-model prefix and as the input to the modules that remain at inference. DriveLM [20] provides the graph-structured driving question-answering annotations on nuScenes from which our four-stage chain is constructed.

3. Materials and Methods

3.1. Overview

We refer to the complete framework as TaLS; its core is the task-decomposed language supervision TD-QA (Section 3.3), coupled to the network by the stage-wise hidden-state injection (Section 3.4). TaLS consists of a GenAD-based E2E decoder and an LLM branch (Figure 1). In the LLM branch, the autoregressive LLM and the stage-wise injection are used only during training (for the inference-time graph, see Section 3.5). Six surround-view camera images are converted by a ResNet-50 [17] backbone and a BEV encoder [18] into BEV features B ∈ ℝ ^{H× W× C} (H=W=100, C=256). The E2E decoder follows the GenAD pipeline: Agent Decoder (object detection), Map Decoder (vectorized map), Motion Decoder (agent–ego interaction and agent/ego–map cross-attention), the generative trajectory module, and the Planner.

During training, BEV tokens—the BEV features compressed by the BEV Q-Former—are fed to a LoRA-based Qwen2.5-1.5B; the TD-QA reasoning chain is performed, and each stage's hidden state is injected into the corresponding decoder. The BEV tokens are also fed back into the model through two lightweight modules: SLG (Spatial Language Grounding; BEV tokens are back-projected onto the BEV grid using the Q-Former's cross-attention weights and added to the BEV as a gated residual) and Fusion (detection queries cross-attend to the BEV tokens and are augmented as a gated residual). As shown in the analysis of Section 3.5, although these two modules add only small residuals, they contribute to inference performance (especially L2) and are therefore retained in the inference graph.

3.2. BEV Q-Former

To compress the BEV features into a small number of tokens suitable for the language model input space, we adapt BLIP-2's Q-Former [19] to the BEV domain. N_q=32 learnable query tokens cross-attend, through a BERT structure (12 layers), to the spatially pooled and projected BEV tokens (25 × 25 = 625) to summarize the scene. The output is linearly projected to BEV tokens V ∈ ℝ ^N_q^{× d}_LLM in the LLM embedding dimension (d_LLM=1536).

V = QFormer(pool(B)) ∈ ℝ ^32×1536.

(1)

3.3. Task-Decomposed Question Answering (TD-QA)

We restructure the DriveLM [20] driving QA into a four-stage reasoning chain based on keywords.

Table 1. The four-stage TD-QA chain and its corresponding driving tasks.

Stage	Query	Task
Object/Scene	Describe the objects in the current scene and their locations	Detection
Motion	Describe the motion state of each object	Motion prediction
Risk	Identify dangerous situations or unsafe actions for the ego vehicle	Risk assessment
Planning	The ego vehicle's safe driving behavior	Planning

Table 1. The four-stage TD-QA chain and its corresponding driving tasks.

Stage	Query	Task
Object/Scene	Describe the objects in the current scene and their locations	Detection
Motion	Describe the motion state of each object	Motion prediction
Risk	Identify dangerous situations or unsafe actions for the ego vehicle	Risk assessment
Planning	The ego vehicle's safe driving behavior	Planning

The four stages of queries and answers are serialized into a single sequence, and the BEV tokens V are prepended as a prefix to the input of Qwen2.5-1.5B. Figure 2 illustrates the chain and the resulting injections for one training sample.

input = [ V ; Stage₁ ; … ; Stage₄ ].

(2)

Since Qwen uses decoder-only causal attention, the BEV token hidden states at the front of the sequence (positions 0–31) cannot attend to the subsequent text. Therefore, the hidden states that carry the language-reasoning content are those at the end position of each stage's answer (h_s below), and these constitute the supervisory signal of our method. A causal language-modeling loss $ℒ$_LM is computed only over the answer tokens (BEV tokens and query tokens are excluded):

$ℒ$_LM = − Σ_{t ∈ A} log P_θ ( x_t | x_<t , V ).

(3)

where A denotes the set of answer-token positions. Through this objective, the LLM learns the decomposed driving reasoning.

3.4. Stage-Wise Hidden-State Injection

At the last-token position of each stage s's answer, we extract the LLM final-layer hidden state h_s ∈ ℝ ^d_LLM, and after a linear projection, inject it as a residual into the queries of the corresponding task decoder. The injection is a gate-free direct addition (q ← q + W_s h_s), where h_s ∈ ℝ ^d_LLM and the stage-specific projection W_s maps ℝ ^d_LLM to the query dimension of the target decoder (256 in our setting), so that W_s h_s is dimensionally compatible with q.

Writing H for the final-layer hidden states of the language model and π_s for the position of the last answer token of stage s, the extracted state is given by (4) and the stage-wise injection by (5); h₁ is broadcast to all detection queries i, h₂ to all motion queries j, and the risk and planning states are both added to the single ego query, matching Figure 2c.

h_s = H[π_s] ∈ ℝ^d_LLM, s ∈ {1, 2, 3, 4},

(4)

q′_agent⁽ⁱ⁾ = q_agent⁽ⁱ⁾ + W₁ h₁, q′_motion^(j) = q_motion^(j) + W₂ h₂,
q′_ego = q_ego + W₃ h₃ + W₄ h₄.

(5)

Concretely, the perception-stage hidden state h₁ is added to the decoded object queries of the Agent Decoder; the motion-stage hidden state h₂ is added to the motion queries, which are not independent embeddings but are derived from the Agent Decoder output; and the planning-stage hidden state h₄ is added to the learned ego query, which is subsequently fed to the Motion Decoder together with the agent queries. This aligns the detection → motion → planning task flow with the corresponding language-reasoning stages. The risk-stage hidden state is injected into the same ego query, providing an additional safety-oriented context for planning.

Auxiliary feature distillation. In addition to the stage-wise injection, which is the core mechanism of TaLS, we adopt an auxiliary feature-distillation loss that aligns the detection queries with the BEV token features as processed by the language model, in the spirit of multi-modal distillation for driving [8]. The teacher is the set of LLM hidden states at the BEV token positions, V_LLM ∈ ℝ^32×1536, taken with a stop-gradient; the student is the final-layer query set of the Agent (detection) decoder, Q_det ∈ ℝ^N×256 with N = 300. Both are projected into a shared 128-dimensional space by small MLPs with layer normalization, and a cross-attention layer (the 32 teacher tokens as queries, the N detection queries as keys and values) produces one student representation per teacher token. Writing the ℓ₂-normalized teacher and student representations as z^L_i and z^V_i over the M = N_b × N_q = 32 N_b pairs in a batch, where N_b is the batch size, the loss is the InfoNCE objective (6):

$ℒ$_dist = −(1/M) Σ_i log [ exp(z^L_i · z^V_i / τ) / Σ_j exp(z^L_i · z^V_j / τ) ], τ = 0.2,

(6)

in which matching token indices form the positive pairs and all other pairs in the batch are negatives. Because the BEV token positions precede the text under causal attention (Section 3.3), this term aligns visual-token features rather than transferring language reasoning; the latter is carried by the stage-wise injection.

The overall training objective is given by (7):

$ℒ$ = $ℒ$_drive + λ_LM $ℒ$_LM + λ_dist $ℒ$_dist, $ℒ$_drive = $ℒ$_det + $ℒ$_map + $ℒ$_mot + $ℒ$_plan,

(7)

where the individual weights within $ℒ$_drive follow GenAD [3], λ_dist = 0.5, and λ_LM = 0.3. Which terms are active in each phase is specified in Section 3.6.

3.5. Train/Inference Asymmetry and Mechanism Analysis

Training. We perform the TD-QA chain and compute the driving task losses (detection, map, motion, planning) together with $ℒ$_LM. Since the BEV encoder and embeddings are frozen throughout, the only components that are both trained and retained at inference are the task decoders (and heads) and the BEV token modules. Because the per-stage h_s is added to the decoder queries while the task loss is minimized, its gradient propagates into the decoder weights during training.

Inference. Without QA annotations, the autoregressive LLM (Qwen2.5-1.5B) is not executed and the stage-wise hidden-state injection is omitted (LLM-free); however, the BEV Q-Former and the two BEV token augmentation modules (SLG, Fusion) are retained in the inference graph. The inference path is the GenAD E2E decoder plus the retained BEV token path (Q-Former, SLG, Fusion); only the autoregressive LLM and the training-only injection are removed. We emphasize that the language branch acts as a training-time representational supervisor rather than an inference-time generator: under teacher forcing it shapes the injected hidden states toward the DriveLM targets, and its free-form generation is never used at inference. This is consistent with the training-only design.

Mechanism analysis (measurements on the trained model).

Although the BEV token residual has a small relative magnitude (r_SLG = 1.9%, r_Fusion = 0.5%), disabling the path increases L2 from 0.517 to 0.544, confirming its functional importance for trajectory accuracy. In contrast, the collision rate remains below the GenAD baseline after pruning, indicating that part of the collision improvement persists independently of the retained BEV token path.

The two retained modules are defined in (8), where Φ_SLG back-projects the BEV tokens V onto the BEV grid using the Q-Former cross-attention weights, CA denotes cross-attention with the detection queries as queries and V as keys and values, and g_S, g_F are learned scalar gates. The prune ablation of Section 4.4 sets g_S = g_F = 0 at inference. The relative contribution of each module, defined in (9) where Δ denotes each module's residual, is r_SLG = 1.9% and r_Fusion = 0.5%.

B′ = B + g_S Φ_SLG(V), Q′_agent = Q_agent + g_F CA(Q_agent, V).

(8)

r_SLG = ‖ g_S Δ_SLG ‖ / ‖ B ‖, r_Fusion = ‖ g_F Δ_Fusion ‖ / ‖ Q_agent ‖.

(9)

3.6. Training Strategy

We adopt three-phase training. (1) Phase 1 (5 epochs, base learning rate 1.2×10⁻³): freeze the task decoders and train only the Q-Former and LoRA adapters with the language-modeling loss $ℒ$_LM to establish BEV–language alignment. (2) Phase 2 (5 epochs, 1.4×10⁻⁴): train the task decoders, heads, BEV token modules (SLG and Fusion), and stage-wise injection projections with the driving losses and the auxiliary feature-distillation loss $ℒ$_dist, while the language model is frozen ($ℒ$_LM = 0). (3) Phase 3 (10 epochs, 3×10⁻⁵): fine-tune all trainable components jointly with the driving losses, $ℒ$_LM (weighted by 0.3), and $ℒ$_dist.

3.7. Experimental Setup

Dataset and metrics. We evaluate on the nuScenes [21] validation set. Planning is reported with L2 error and collision rate (box collision) under two aggregation protocols: TemAvg (cumulative average over all waypoints within the 0–N s interval), the protocol used by our baseline GenAD (following VAD), and NoAvg (the value at the N-s time point alone). We take TemAvg as the primary metric since it matches our baseline, and report NoAvg alongside it. In both cases the reported “Avg.” is the arithmetic mean of the 1 s, 2 s, and 3 s values, computed before rounding. Planning results in Table 2 and Table 3 are reported to two decimal places for consistency with prior work, whereas the controlled ablation results in Table 6, Table 7 and Table 8 are reported to three decimal places to preserve the resolution of smaller within-model differences. Relative improvements quoted in the text are computed from the unrounded values. Because the per-timestep aggregation is sometimes labeled “ST-P3” [14] in code and its exact definition varies across prior works, we standardize on the explicit terms TemAvg/NoAvg to avoid ambiguity.

Planned trajectories consist of waypoints at 0.5 s intervals, so an N-second horizon contains 2N waypoints; the collision rate is computed by placing the ego box (1.85 m × 4.084 m) at each evaluated waypoint and testing intersection with the ground-truth occupancy of the surrounding agents at the corresponding time step.

Implementation. Backbone ResNet-50 [17], BEV 100×100, LLM Qwen2.5-1.5B-Instruct [22] (LoRA [23] r=16, α=32, target q/v_proj), 32 BEV Q-Former queries.

Experimental conditions. All models were trained and evaluated on a single NVIDIA A100-SXM4-80GB GPU for direct comparability. The three-phase curriculum used a per-GPU batch size of 16 in phase 1 and 8 in phases 2–3 without gradient accumulation, so the effective batch size equals the per-GPU value. Input images (1600 × 900) were scaled by 0.4 and padded to a multiple of 32, giving a 640 × 384 input. Optimization used AdamW (weight decay 0.01) with the per-phase learning rates detailed in Section 3.6. The curriculum ran for 5 + 5 + 10 = 20 epochs. Evaluation follows the nuScenes validation split with the TemAvg and NoAvg protocols defined above; inference throughput (Table 5) was measured with the standard per-frame benchmark on a single NVIDIA A100-SXM4-80GB at batch size 1 in fp32, timing the model forward only and excluding data loading. Following the fair-comparison setting of VAD [2] and GenAD [3], ego-status inputs were not provided to the planner in either the GenAD baseline or any TaLS variant; only the standard navigation command was used.

Model complexity. We report parameter counts and resource usage of TaLS to characterize its cost. At training time the model comprises the GenAD E2E backbone and decoders (47.9 M parameters), the BEV Q-Former (206.1 M), the frozen Qwen2.5-1.5B-Instruct base model (1.54 B parameters) with trainable LoRA adapters (2.18 M parameters), and the SLG, Fusion, and stage-wise injection projections (2.89 M, comprising 1.31 M for SLG and Fusion and 1.57 M for the injection projections). At inference the autoregressive language model, the LoRA adapters, and the injection projections are removed, leaving the E2E decoders and the retained BEV token path (BEV Q-Former, SLG, Fusion). A module-level count gives an upper bound of 255.3 M parameters for the retained inference graph, approximately 207 M above the E2E baseline. This count includes Q-Former sub-modules, such as the masked-language-modeling head and vocabulary embeddings, that are not used in the BEV token forward pass; we therefore treat it as an upper bound and use measured throughput as the primary inference-cost comparison. Peak GPU memory during training was approximately 29.4 GB (phase 3, batch size 8, 640 × 384 input). Total training time was approximately 31 GPU-hours on the single A100, estimated from per-iteration timing rather than wall-clock timestamps. Because the autoregressive language model is discarded after training, the inference graph is that of the E2E decoder plus the retained BEV token path; the residual cost of that path is characterized directly by the prune analysis (Section 4.4) and the throughput measurement (Table 5).

4. Results

4.1. Planning

Table 2. Open-loop planning results on the nuScenes validation set under the TemAvg protocol (primary), which averages the metric over all waypoints up to the N-s horizon. L2 in m and collision rate in %; lower is better. Numbers for prior methods are quoted from the corresponding papers, whereas the GenAD baseline was trained and evaluated in the same environment as TaLS; the controlled comparison is therefore between the GenAD baseline and TaLS, and its TemAvg averages match the values reported in [3]. For VLM-AD, we report the VAD-Tiny-based Q1+Q2 configuration. Bold marks the best value in each column; ties are left unmarked.

Method	L2 (m)				Collision (%)
	1s	2s	3s	Avg.	1s	2s	3s	Avg.
VAD-Tiny [2]	0.46	0.76	1.12	0.78	0.21	0.35	0.58	0.38
VAD-Base [2]	0.41	0.70	1.05	0.72	0.07	0.17	0.41	0.22
VLP-VAD [6]	0.30	0.53	0.84	0.55	0.01	0.07	0.38	0.15
VLM-AD [7]	0.30	0.54	0.80	0.55	0.11	0.15	0.38	0.21
GenAD [3]	0.27	0.49	0.81	0.52	0.14	0.18	0.25	0.19
TaLS (Ours)	0.27	0.49	0.80	0.52	0.06	0.12	0.21	0.13

Table 2. Open-loop planning results on the nuScenes validation set under the TemAvg protocol (primary), which averages the metric over all waypoints up to the N-s horizon. L2 in m and collision rate in %; lower is better. Numbers for prior methods are quoted from the corresponding papers, whereas the GenAD baseline was trained and evaluated in the same environment as TaLS; the controlled comparison is therefore between the GenAD baseline and TaLS, and its TemAvg averages match the values reported in [3]. For VLM-AD, we report the VAD-Tiny-based Q1+Q2 configuration. Bold marks the best value in each column; ties are left unmarked.

Method	L2 (m)				Collision (%)
	1s	2s	3s	Avg.	1s	2s	3s	Avg.
VAD-Tiny [2]	0.46	0.76	1.12	0.78	0.21	0.35	0.58	0.38
VAD-Base [2]	0.41	0.70	1.05	0.72	0.07	0.17	0.41	0.22
VLP-VAD [6]	0.30	0.53	0.84	0.55	0.01	0.07	0.38	0.15
VLM-AD [7]	0.30	0.54	0.80	0.55	0.11	0.15	0.38	0.21
GenAD [3]	0.27	0.49	0.81	0.52	0.14	0.18	0.25	0.19
TaLS (Ours)	0.27	0.49	0.80	0.52	0.06	0.12	0.21	0.13

Table 3. Open-loop planning results on the nuScenes validation set under the NoAvg protocol (secondary), which reports the value at the N-s time point alone. Conventions and sources are as in Table 2.

Method	L2 (m)				Collision (%)
	1s	2s	3s	Avg.	1s	2s	3s	Avg.
ST-P3 [14]	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71
UniAD [1]	0.48	0.96	1.65	1.03	0.05	0.17	0.71	0.31
VAD-Tiny [2]	0.60	1.23	2.06	1.30	0.31	0.53	1.33	0.72
VAD-Base [2]	0.54	1.15	1.98	1.22	0.04	0.39	1.17	0.53
GenAD	0.35	0.86	1.67	0.96	0.14	0.29	0.53	0.32
TaLS (Ours)	0.35	0.85	1.64	0.95	0.06	0.29	0.47	0.27

Table 3. Open-loop planning results on the nuScenes validation set under the NoAvg protocol (secondary), which reports the value at the N-s time point alone. Conventions and sources are as in Table 2.

Method	L2 (m)				Collision (%)
	1s	2s	3s	Avg.	1s	2s	3s	Avg.
ST-P3 [14]	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71
UniAD [1]	0.48	0.96	1.65	1.03	0.05	0.17	0.71	0.31
VAD-Tiny [2]	0.60	1.23	2.06	1.30	0.31	0.53	1.33	0.72
VAD-Base [2]	0.54	1.15	1.98	1.22	0.04	0.39	1.17	0.53
GenAD	0.35	0.86	1.67	0.96	0.14	0.29	0.53	0.32
TaLS (Ours)	0.35	0.85	1.64	0.95	0.06	0.29	0.47	0.27

Table 2 and Table 3 report open-loop planning results under the two aggregation protocols. Compared with our GenAD reproduction, TaLS reduces the collision rate under both protocols (TemAvg 0.19 → 0.13, −31.3% relative; NoAvg 0.32 → 0.27) while maintaining comparable L2 performance (TemAvg 0.52 in both cases; NoAvg 0.96 → 0.95). Relative to prior methods, TaLS attains the lowest collision rate at the 3 s horizon and on average within each protocol block. We stress that the quoted numbers for prior methods were produced in their own environments and, as the spread among the quoted methods shows, absolute values are sensitive to implementation and evaluation details; the controlled comparison in this paper is therefore between our GenAD reproduction and TaLS. The retraining-free, deterministic analysis (Section 4.4) provides complementary evidence about the mechanism. Figure 3 shows qualitative trajectories for representative scenes spanning turns and curved road segments, including intersections with pedestrian crossings and surrounding agents; across these scenes TaLS produces smooth, map-consistent trajectories that follow the ground-truth path closely.

4.2. Perception

Table 4. Perception results on nuScenes (val). Higher is better; best in bold.

Method	Detection mAP	NDS	Map mAP
GenAD	0.280	0.399	0.477
TaLS	0.281	0.400	0.476

Table 4. Perception results on nuScenes (val). Higher is better; best in bold.

Method	Detection mAP	NDS	Map mAP
GenAD	0.280	0.399	0.477
TaLS	0.281	0.400	0.476

TaLS maintains perception performance comparable to the GenAD baseline, indicating that the planning improvement is achieved without materially affecting the underlying perception tasks.

4.3. Efficiency

Table 5. Inference efficiency.

Method	Inference graph	FPS	Relative
GenAD	E2E only	13.2	—
TaLS	+ Q-Former/SLG/Fusion	11.0	−16.7%

Table 5. Inference efficiency.

Method	Inference graph	FPS	Relative
GenAD	E2E only	13.2	—
TaLS	+ Q-Former/SLG/Fusion	11.0	−16.7%

At inference, the autoregressive LLM is completely removed, eliminating the associated generation cost. The retained BEV token modules (Q-Former/SLG/Fusion) operate at 11.0 FPS compared with 13.2 FPS for GenAD, corresponding to a 16.7% throughput trade-off. As shown in Section 4.4, these retained modules contribute to trajectory accuracy and therefore form a functional part of the inference architecture.

4.4. Ablation (Planning)

BEV token path prune (no retraining). On the trained model, we fixed the SLG and Fusion gates to 0 (output-equivalent to removing the modules) and evaluated at inference.

Table 6. Prune ablation on the trained model (nuScenes val), primary TemAvg protocol. Lower is better; best in bold.

Method	L2 (m)				Collision (%)
	1s	2s	3s	Avg.	1s	2s	3s	Avg.
GenAD	0.265	0.488	0.808	0.520	0.137	0.181	0.254	0.190
TaLS (pruned)	0.282	0.514	0.837	0.544	0.107	0.147	0.228	0.161
TaLS	0.265	0.487	0.800	0.517	0.059	0.122	0.212	0.131

Table 6. Prune ablation on the trained model (nuScenes val), primary TemAvg protocol. Lower is better; best in bold.

Method	L2 (m)				Collision (%)
	1s	2s	3s	Avg.	1s	2s	3s	Avg.
GenAD	0.265	0.488	0.808	0.520	0.137	0.181	0.254	0.190
TaLS (pruned)	0.282	0.514	0.837	0.544	0.107	0.147	0.228	0.161
TaLS	0.265	0.487	0.800	0.517	0.059	0.122	0.212	0.131

Disabling the inference-time BEV token path (SLG and Fusion, gate → 0) increases L2 from 0.517 to 0.544, with the same trend observed across all prediction horizons, confirming the functional importance of this path for trajectory accuracy despite its small residual magnitude (gate ≈ 0.01). The relative contributions of the two modules are r_SLG = 1.9% and r_Fusion = 0.5%. In contrast, the collision rate remains below the GenAD baseline after pruning: it is reduced by 15.7% relative to GenAD (0.190 → 0.161), compared with 31.3% for the full model (0.190 → 0.131), and remains lower at every horizon (0.107/0.147/0.228 versus 0.137/0.181/0.254). These results show that the retained BEV token path is particularly important for L2, whereas part of the collision improvement persists after the path is disabled, consistent with partial encoding in the retained decoder weights. Figure 4 visualizes this ablation.

Language supervision versus BEV token modules. To test whether the planning gains stem from the language supervision itself or merely from the additional BEV token modules, we trained a variant of TaLS without language supervision, denoted TaLS w/o language. It is identical to TaLS in architecture, training schedule, data, initialization, and budget, except that the language branch is disabled: the autoregressive LLM forward pass, the LoRA adapters, the language-modeling loss, the auxiliary feature-distillation loss, and the stage-wise hidden-state injection are all removed (equivalently, every training sample is treated as having no QA), while the BEV Q-Former, SLG, and Fusion modules are retained. Because Phase 1 of the full model is defined by the language-modeling loss, that objective is unavailable in this variant; Phase 1 here instead updates the Q-Former alone with a self-supervised reconstruction loss (weight 0.1) that pulls the BEV tokens towards the pooled BEV features, with all other components frozen. The Q-Former is therefore carried into Phase 2 without BEV–language alignment, which follows from removing language supervision rather than from an incidental difference in setup. The phase lengths, optimizer settings, and learning rates in Phases 2 and 3 follow the full-model schedule, while all language-dependent modules and losses remain disabled. Note that this ablation removes the stage-wise injection and the auxiliary distillation loss jointly, so it does not isolate the contribution of the distillation term on its own. Table 7 compares GenAD, TaLS w/o language, and TaLS under this common setting.

Table 7. Ablation of language supervision on nuScenes (val). All TaLS ablation variants use the same data, initialization, total training budget, and compute environment; component-specific training differences are described above. GenAD is included as the baseline. Results are reported under the primary TemAvg protocol; collision rate in %, lower is better.

Method	L2 (m)				Collision (%)
	1s	2s	3s	Avg.	1s	2s	3s	Avg.
GenAD	0.265	0.488	0.808	0.520	0.137	0.181	0.254	0.190
TaLS w/o language	0.262	0.484	0.805	0.517	0.098	0.156	0.270	0.175
TaLS w/o decomposition	0.267	0.490	0.802	0.520	0.078	0.132	0.221	0.144
TaLS	0.265	0.487	0.800	0.517	0.059	0.122	0.212	0.131

Table 7. Ablation of language supervision on nuScenes (val). All TaLS ablation variants use the same data, initialization, total training budget, and compute environment; component-specific training differences are described above. GenAD is included as the baseline. Results are reported under the primary TemAvg protocol; collision rate in %, lower is better.

Method	L2 (m)				Collision (%)
	1s	2s	3s	Avg.	1s	2s	3s	Avg.
GenAD	0.265	0.488	0.808	0.520	0.137	0.181	0.254	0.190
TaLS w/o language	0.262	0.484	0.805	0.517	0.098	0.156	0.270	0.175
TaLS w/o decomposition	0.267	0.490	0.802	0.520	0.078	0.132	0.221	0.144
TaLS	0.265	0.487	0.800	0.517	0.059	0.122	0.212	0.131

On the primary TemAvg protocol, the collision rate improves progressively from 0.190 for GenAD to 0.175 with the BEV token modules and to 0.131 with the full TaLS model, indicating that the language-supervised components make the major contribution to collision reduction. In contrast, the L2 improvement is already observed with the BEV token modules (0.520 → 0.517) and is preserved in the full model. These results highlight complementary roles of the two components: language supervision mainly benefits the open-loop collision metric, whereas the retained BEV token path supports trajectory accuracy.

Decomposed versus undecomposed language supervision. The control above shows that language supervision helps, but not whether the benefit follows from the task decomposition or simply from exposing the model to driving-related language during training. We therefore train a non-decomposed variant, denoted TaLS w/o decomposition, that is identical to the full model in every other respect — BEV token modules, LLM and LoRA adapters, language-modeling and distillation losses, number of injection sites, training budget, data, and schedule — but removes the four-stage structure: the four DriveLM answers are concatenated into a single generic question–answer pair without stage markers, and the resulting single answer-end hidden state is broadcast to all query types instead of four stage-aligned states being injected into their corresponding queries. The language content and the token budget are thus held fixed and only the structure is removed.

Under the primary TemAvg protocol, the collision rate decreases progressively along the ablation ladder: 0.190 for GenAD, 0.175 with the BEV token modules, 0.144 with non-decomposed language supervision, and 0.131 with the full TaLS model. The additional improvement from 0.144 to 0.131 shows that stage-aligned decomposition provides a further benefit beyond generic language supervision. In addition, the decomposition provides explicit stage-to-module traceability and a query-aligned injection design, in which each reasoning stage supervises the module associated with the corresponding subtask.

Three-phase curriculum versus single-stage training. The full model is trained with a three-phase curriculum: the LoRA adapters and BEV Q-Former are trained first, then the decoders together with the SLG, Fusion, and injection modules while the LLM is frozen, and finally all trainable components are fine-tuned jointly. To test whether this schedule is necessary, we train a single-stage variant that is architecturally identical to the full model — the same four-stage decomposition and aligned injection — but jointly optimizes all trainable components from the pretrained GenAD initialization for the same total budget of 20 epochs. To keep the comparison fair, the single-stage schedule uses fixed module-wise learning rates chosen to match the nominal learning-rate scales of the curriculum: a base rate of 1×10⁻⁴ for the pretrained decoders and heads, with the BEV Q-Former at 1.2×10⁻³, the LoRA adapters at 3.6×10⁻⁴, and the SLG, Fusion, and injection projections at 1.5×10⁻⁴ (effective batch size 8); the BEV encoder and image backbone remain frozen as in the main model.

Table 8. Training-procedure ablation on nuScenes (val), primary TemAvg protocol. The single-stage variant is architecturally identical to the full model and uses the same 20-epoch budget and module-wise learning rates chosen to match the curriculum's nominal scales, but is trained in a single joint phase. Collision rate in %; lower is better.

Method	L2 (m)				Collision (%)
	1s	2s	3s	Avg.	1s	2s	3s	Avg.
GenAD	0.265	0.488	0.808	0.520	0.137	0.181	0.254	0.190
TaLS (Single-stage)	0.259	0.476	0.788	0.508	0.273	0.303	0.436	0.338
TaLS (3-phase)	0.265	0.487	0.800	0.517	0.059	0.122	0.212	0.131

Table 8. Training-procedure ablation on nuScenes (val), primary TemAvg protocol. The single-stage variant is architecturally identical to the full model and uses the same 20-epoch budget and module-wise learning rates chosen to match the curriculum's nominal scales, but is trained in a single joint phase. Collision rate in %; lower is better.

Method	L2 (m)				Collision (%)
	1s	2s	3s	Avg.	1s	2s	3s	Avg.
GenAD	0.265	0.488	0.808	0.520	0.137	0.181	0.254	0.190
TaLS (Single-stage)	0.259	0.476	0.788	0.508	0.273	0.303	0.436	0.338
TaLS (3-phase)	0.265	0.487	0.800	0.517	0.059	0.122	0.212	0.131

Table 8 demonstrates the importance of the three-phase training strategy. While trajectory accuracy remains comparable across the two schedules (TemAvg L2 0.508 for single-stage training and 0.517 for the three-phase curriculum), the curriculum substantially improves the collision metric, reducing the average collision rate from 0.338 to 0.131, with consistent improvements across all three prediction horizons. This result indicates that the staged optimization strategy plays an important role in realizing the full collision benefit of the architecture and effectively transferring language supervision to the driving model. Together with the language ablation in Table 7, these results show that both language supervision and the training curriculum contribute to the collision-rate improvement.

These observations complement the prune analysis: the L2 gain depends on the inference-time BEV token path, whereas the persistence of part of the collision gain after pruning is consistent with partial encoding in the retained decoder weights.

5. Conclusions

This paper proposed TaLS, a training-only language-supervision framework that decomposes driving QA into a reasoning chain aligned with the task hierarchy and injects the language representation of each stage into the corresponding task decoder during training. TaLS improves open-loop planning performance over a strong baseline by substantially reducing the collision rate while maintaining comparable L2 performance, with the autoregressive LLM removed at inference. A central finding is that the mechanism of improvement differs across metrics (Section 3.5 and Section 4.4): the collision rate improvement is retained even when the BEV token path is disabled, which is consistent with partial encoding in the retained decoder weights, whereas the L2 performance depends on the BEV token path retained at inference. Control ablations (Section 4.4, Table 7) further show that language supervision provides a substantial collision-rate improvement, while the stage-aligned decomposition yields an additional reduction over non-decomposed supervision (0.144 → 0.131). Together, these results support the complementary roles of language supervision, task-aligned decomposition, and the retained BEV token path in improving planning performance. Future work will extend TaLS to closed-loop evaluation settings and investigate content-level controls that further characterize the role of reasoning semantics in stage-wise supervision.