LaneLM: Lane Detection as Language Modeling

1. Introduction

Lane detection (LD), the process of recognizing lanes as approximated curves, is a foundational task in the realm of autonomous driving, whereby the detector aims to perceive the accurate shape of each lane in an image captured by a front-mounted camera. In the context of autonomous driving or even human-driven vehicles equipped with advanced driver assistance systems (ADAS), accurate lane detection helps keep the vehicle within its proper lane.

Our objective is to develop an interactive lane detection model similar to SAM [1] (Segment Anything Model). Firstly, when encountering complex road conditions where the model fails to recognize, it can accept human feedback in natural language to weight the model’s spatial attention or can be prompted by a parallel large visual language model. Secondly, autonomous driving requires a large number of manual annotations for training. Before manual annotation, images are auto-annotated by a pretrained LD model, which often underperforms in corner cases that follow a long-tail distribution. If annotators could directly indicate several keypoints of the lane to the model and the model could autoregress complete high-quality lane based on these semantic cues, it would significantly reduce annotation costs. However, existing lane detection models neither possess such interactive capabilities nor can they be prompted. To address these issues, we train the LD model using the language model training method to explore the improvements gained by the model receiving a few prompts.

Existing lane detection methods, whether based on line-anchor regression, segmentation, or keypoint detection, rely on convolutional neural networks (CNN) and spatial attention mechanisms. Firstly, the locality of CNNs makes it difficult to capture the long-range dependencies of lanes across the entire image. Secondly, traditional methods lack the interactive capabilities of language models and cannot specify the position of spatial attention weights in the feature map in corner cases. To address these issues, we convert visual and semantic information into a token sequence, utilizing a cross-attention mechanism, in which queries are language tokens and keys and values are visual tokens. Therefore, the model can determine the location where attention is applied on the feature map.

Inspired by [1,2], thanks to Lane2Seq [3], this study proposes a promptable framework, called LaneLM, to LD that differs from the mainstream approaches. Similar to pioneer work [3,4], our method is based on two intuitions: First, if the detector knows where lanes are, we just need to teach it to read them out so that we can refine the location of keypoints in this context. If an initial set of keypoints, which are discretized into tokens, is given as prompt for each lane , the detector should generate holistic sequence of the lane respectively in a manner of causal language modeling. Second, since language models are few-shot learners[5], why not use a small number of prompts to enhance the LD performance in corner cases? Besides, we can utilize the instruction-following ability of language models to make LD models follow simple instructions. Furthermore, LaneLM avoids customized heads, relying instead on direct keypoint token classification.

In summary, we make the following contributions:

• It is the first attempt to treat the LD task as visual question answering (VQA). Our work paves the way toward unification for language modeling and lane detection.
• We develop an interactive lane detection framework that can accept prompts with semantic information to guide the visual modality.
• With a small number of keypoint prompts (e.g. 4 adjacent points, about 7% of the total length of each lane), LaneLM easily outperforms the previous SOTA method by 1.09%.

Figure 1. Corner cases in lane detection. There are many challenging scenarios in the lane detection task, such as at night, with severe occlusions, resulting in no visual clues for visual task.

Figure 1. Corner cases in lane detection. There are many challenging scenarios in the lane detection task, such as at night, with severe occlusions, resulting in no visual clues for visual task.

2. Related Works

2.1. Traditional Lane Detection Methods

The mainstream approaches can be classified into the following categories:

1) Anchor-based methods. The network structure designing is similar to object detection but [6,7,8,9] use line-anchor instead of anchor box as lane prior and regress offsets between sampled points and predefined anchor points. At inference time, Non-Maximum Suppression (NMS) is applied as post-processing. These methods take both speed and accuracy into account. Nevertheless, these methods struggle to detect lanes with complex topology, the predefined line anchor with a fixed shape fails to serve as a correct reference.

2) Segmentation-based methods [10,11,12] pixel-wisely predict each lane and assign different lanes to different channels, respectively, so that they are flexible enough to detect lanes with complex topology. Segmentation-based methods can be unified with other segmentation tasks. For example, a single model can be used to segment lanes and parking slots simultaneously. However, these methods, which require heuristic post-processing to decode the segmentation map, are not accurate or fast enough compared with anchor-based methods.

3) Keypoint-based methods. [13,14,15] treat each lane as a set of keypoints and then binarily predict foreground points and the background. Compared with segmentation-based methods, keypoint-based methods reduce the complexity of foreground classification. They can handle complex topological structures while also considering real-time performance. However, the post-processing of keypoint-based methods still remains an issue. Researchers have developed complex clustering and filtering methods to distinguish lanes in the same image.

4) Row-wise methods [16,17,18,19], which rasterize lanes in the image and row-wisely predict the grid occupation for each lane, are simple yet effective. These methods develop the row-anchor representation for lane detection instead of a line-anchor. However, side lanes of the Y-shape, often nearly horizontal, make lane classification difficult. These methods can achieve a high inference speed and easily surpass the methods described above in terms of real-time performance while getting comparable results.

5) Curve-based methods [20,21,22,23] formulate each lane into a parametric equation, such as polynomials or Bézier curves, predicting the parameters. Benefiting from holistic representation of the lane, inherently eliminate occlusions. These methods predict lanes in an end-to-end manner, require no post-processing but their performance still lag behind other methods by a large margin.

The current SOTA CLRerNet [9], based on the previous SOTA CLRNet [6], has introduced the LaneIoU loss for training, taking into account the topological structure of the lanes. CLRNet [6] integrates high-level features with low-level features that contain more spatial information, and achieves impressive results through the application of a spatial attention mechanism. Previous SOTA GANet [13] use DCN [24] to associate keypoints in the same lane with their belonged start points. In summary, traditional LD methods focus only on the spatial information and topological structure of lanes.

2.2. Autoregressive Modeling for Vision

In light of the aforementioned methods, spatial information of lane perception in 2D space is well-exploited. We aim to borrow semantic-aware capability from Natural Language Processing (NLP) so that our model can capture contextual dependency of input sequences and produce high-quality output. . Autoregressive (AR) model has found great success in motion prediction [25,26], autonomous driving [27], NLP [5,28,29] and visual tracking [30,31]. Apart from Lane2Seq [3], little work in lane detection focuses on this promising paradigm that utilizes temporal or semantic information, and none of these traditional LD methods are interactive.

Pix2Seq [4] quantizes bounding boxes into language tokens to describe the objects. Therefore, the model can learn to ground the language tokens on visual observation. Following Pix2Seq, ARTrack [30,31] firstly adopts this framework in visual tracking. Flamingo [28] explores the few-shot learning capability of visual language models and it is visually-conditioned AR language model, i.e. a visual encoder is used to perceive the information in images while a language model is utilized to refine the visual information at the semantic level in the few-shot context and employ a connector to bridge the visual encoder and the language decoder. The same architecture can also be found in many VQA models [29,32,33,34].

3. Proposed LaneLM

LaneLM has neither complex trick nor sophisticated structure. First, we tokenize the float keypoints and convert them into word embeddings. Then, we use a visual encoder to transform the visual information into visual embeddings. Next, a language decoder (shown in Figure 4) is designed for prompting. We simply use cross-attention layers to condition the language model [35] on visual inputs.

3.1. Lane Representation

A lane is a sequence of xy coordinates represented as tuples of float numbers. First, we quantize the float numbers in these tuples into integer tokens. Then, we integrate the x and y coordinates information into a single embedding, transforming it into a contextual representation that can be processed by a transformer decoder.

Lane tokenization. In order to represent lanes, following [6,8], we use equally-spaced 2D-points as lane representation. Each lane L is expressed as a sequence of keypoints, i.e.

$$L=\{(x_1,y_1),(x_2,y_2)...,(x_T,y_T)\}$$

(1)

in which y coordinate is equally and vertically sampled at regression time step t, i.e. $y_t={\displaystyle \frac{H}{T}}·t$, where H is image height. Inspired by row-wise methods mentioned above, we transfer continuous keypoints of lane into discrete counterparts. Then we can rasterize H pixel rows into integer space T, which is also the vocabulary size for $y_t$. Specifically, the keypoint at time step t is composed of two tokens, i.e. $(x_t,t)$. For each column, $x_t$ is quantized to $[0,n_{bins})$, in which $n_{bins}$ is the vocabulary size for $x_t$.

Figure 2 shows the row-wise location for a lane. We process the $n_{bins}$-classes classification in each row and T-classes classification at time step t in each column.

It should be noted that 0 and T are padding tokens for $x,y$ coordinates respectively (i.e. if there is no lane at time step t, let $x_t=0$ and $y_t=T$). Here, T also refers to the number of rollouts.Then we use the quantized term to index a learnable vocabulary to get the token counterpart, which allows the model to depict the location of keypoints.

Keypoint token encoding. If $a_t^i=(x_t^i,y_t^i)$ in the i-th lane at time step t is given, we use two simple embedding layers $E_x$ and $E_y$ to get the token embeddings $e\in {\mathbb{R}}^D$, where D is the size of each vector. Therefore, each integer tuple $a_t^i$ is mapped to a high-dimensional vector.

$$e_t=E_y\left(y_t\right)+E_x\left(x_t\right)+{\mathrm{PE}}_{keypoint}\left(t\right)$$

(2)

where ${\mathrm{PE}}_{keypoint}(·)$ is positional encoding in language modeling [35] to provide transformer model with information about the position of each token in the sequence. Eq. 2 encodes the information of the x-coordinate and y-coordinate of tuple $a_t^i$ into embeddings, which are then fused through addition, followed by the incorporation of positional information. The sequence composed of these embeddings conditioning on visual features is subsequently processed by a language decoder. In the output layer, each embedding will go through the linear head and predicts the next token by classification:

$$a_{t+1}=\mathrm{argmax}\left(\mathrm{softmax}\left(Linear\left(e_t\right)\right)\right)$$

(3)

Therefore, we embed a integer tuple lane sequence L (see Eq. 1) to its contextual representation

$$\mathbf{h}=\{e_1,e_2,...,e_T\}$$

(4)

3.2. Probabilistic Rollouts

Probability factorization. Following [25], let $a_t^n=(x_t^n,y_t^n)$ represent the target keypoints for the n-th lane at time t. We cast LD as a next-token prediction task, formulated as a conditional probability $P(A_1,...A_T|{\mathbf{X}}_{\mathbf{v}})$, where $A_t=\{a_t^1,...,a_t^N\}$ represents the set of target keypoints for all lanes at time t and ${\mathbf{X}}_{\mathbf{v}}\in {\mathbb{R}}^{B\times C\times H\times W}$ is the visual context, RGB image inputs from camera. Thus, we factorize the distribution over future sequences as a product of conditionals:

$$p_{\theta }(A_1,...A_T|{\mathbf{X}}_{\mathbf{v}})=\prod _{t=1}^T{p}_{\theta }\left(A_t\right|A_{<t},{\mathbf{X}}_{\mathbf{v}})$$

(5)

$$p_{\theta }\left(A_t\right|A_{<t},{\mathbf{X}}_{\mathbf{v}})=\prod _{n=1}^N{p}_{\theta }\left(a_t^n\right|A_{<t},{\mathbf{X}}_{\mathbf{v}})$$

(6)

Similar to [25], Eq. 6 represents the fact that we treat each lane as conditionally independent for its keypoint at time t, given the previous keypoints and visual context.

n-lanes parallelism at time step t. In this formulation, we duplicate the visual context n times for each lane in an image, where duplication is repeated in batch dimension in our implementation to calculate $A_t$ in parallel because they are assumed to be conditionally independent as described in Eq. 6.

3.3. Network Architecture

In Section 3.1, we obtained the embedding sequence for the keypoint modality. Similarly, we first encode the image into an embedding sequence using a visual encoder. Then, the information from both the visual modality and the keypoint modality is fused within the AR language decoder. Ultimately, each output embedding can be utilized by the classification head to predict the next token id. We use an encoder-decoder structure for learning and inference. Such a network architecture is widely used in vision-language model [28,29,36] and language model [37].

Visual encoder. Our visual encoder has to extract visual features from images and then transform them into embedding sequences. The pyramid feature extractor f shown in Figure 3 adopts the standard ResNet[38] and DLA[39] as visual feature extractor. And we add a FPN [40] neck to provide integrated multi-scale features:

$$\{{\mathcal{F}}_0,{\mathcal{F}}_1,{\mathcal{F}}_2\}=f\left({\mathbf{X}}_{\mathbf{v}}\right)$$

(7)

where $C_i$, $H_i$ and $W_i$ is the channel, the height and the width of the i-th level feature ${\mathcal{F}}_i\in {\mathbb{R}}^{B\times C_i\times H_i\times W_i}$ extracted from CNN and FPN [40].This structure leverages ConvNet’s pyramidal feature hierarchy and demonstrates its efficiency in [6,13,16,41]. Then we split ${\mathcal{F}}_i$ into fixed-size patches, linearly embed each of them, add position embeddings:

$${\mathcal{L}}_i=E_v\left({\mathcal{F}}_i\right)+{\mathrm{PE}}_{vision}(H_i,W_i)+\mathrm{LE}\left(i\right)$$

(8)

where $\mathrm{LE}(·)$ is level embedding that embeds level information into vectors, ${\mathrm{E}}_{\mathrm{v}}(·)$ is patch embedding in ViT[42], ${\mathrm{PE}}_{vision}(·)$ is its standard positional encoding layer that retains the positional information of each patches and the result value ${\mathcal{L}}_i\in {\mathbb{R}}^{B\times N_i\times D}$ represents the token sequence extracted from level feature ${\mathcal{F}}_i$, in which $N_i$ is the number of patches and we linearly embed them into D-dimensional visual embeddings to aligned with keypoint embeddings $e\in {\mathbb{R}}^D$.

Since ViTAdapter [43] proposes a pretraining-free adapter and achieves SOTA result in dense prediction and more and more visual-language model [28,29] use a frozen pretrained visual encoder and with how well the spatial feature in LD has been exploited, there is good reason to believe that we can directly adopt a frozen backbone from pretrained to reach comparable performance.

Language decoder. Our language decoder causally models keypoint sequence and conditions each embeddings e on visual sequences $\mathcal{L}$. As shown in Figure 4, we use a transformer decoder for target sequence generation. This decoder consists of 3 layers of LaneLM blocks. For every LaneLM block, we insert new cross-attention layers between OPT[35] causal attention blocks, trained from scratch. The visual feature sequence ${\mathcal{L}}_i$, a sequence of flatten 2D-patch embeddings will serve as keys and values in cross-attention layer of block i while queries are derived from keypoint prompts:

$${\mathbf{h}}_{\mathbf{i}}=\mathrm{CrossAtt}(\mathrm{Q}=\mathrm{CausalSelfAtt}\left({\mathbf{h}}_{\mathbf{i}-\mathbf{1}}\right),\mathrm{K}={\mathcal{L}}_i,\mathrm{V}={\mathcal{L}}_i)$$

(9)

in which we denote ${\mathbf{h}}_{\mathbf{i}}$ as hidden state output from block i and specially, ${\mathbf{h}}_{\mathbf{0}}=[e_1,e_2,...,e_N]$ is original keypoint embedding sequence input. ${\mathbf{h}}_{\mathbf{0}}$ should go through word embedding layer and add position embedding as OPT[35] does before feeding into block i.

Our LaneLM layer can avoid information leak. cross-attention ensures that the outputs for a certain position in a sequence is based only on the known query tokens at previous positions and not on future positions. Besides, each query embedding is conditioned on the whole visual information. Theorem A1 demonstrates the feasibility of the network architecture.

Decoupled head. In training phase, every embedding (except for the last embedding of each sequence) output from language decoder will go through x head and y head to predict the next token id of x and y as illustrated in Eq. 3. Considering that predicting ordered y sequence is a simple task, we use two decoupled classification heads, x head, y head, to predict the next token of x and y, respectively, instead of predicting $xy$ tokens like [25]. Such decoupled prediction strategy greatly reduces complexity of classification from $O\left(\right|X\left|\right|Y\left|\right)$ to $O\left(\max \right(\left|X\right|,\left|Y\right|\left)\right)$. To further reduce computational overhead, these heads share the same hidden state input because the embedding $e_t$ contains the information of both x-coordinate and y-coordinate (see Eq. 2).

Figure 4. To condition the LM on visual inputs, we insert cross-attention layers between causal attention layers. The keys and values are obtained from the visual features $\mathcal{L}$ while queries are derived from keypoint prompts $L_q$

Figure 4. To condition the LM on visual inputs, we insert cross-attention layers between causal attention layers. The keys and values are obtained from the visual features $\mathcal{L}$ while queries are derived from keypoint prompts $L_q$

3.4. Training and Inference

Visual question answering. If the detector knows where lanes are, we just need to teach it to read them out so that we can refine the location of keypoints in this context. For instance, during training, each image is pre-annotated by other lane detection models. The output labels are converted into pseudo labels using the method described in Section 3.1. These pseudo labels are then used as lane priors and fed into the language decoder as queries to extract spatial information from the keys and values in its cross-attention modules. In this way, the model gains an approximate understanding of the lane locations. Our goal is to refine these locations or just read them out if pseudo labels are accurate enough. Moreover, These pseudo-labels can be regarded as questions in VQA, while their corresponding ground truth serves as the answers to these questions. In this way, the LD task is transformed into a VQA task.

Self-supervised labels. Following [28,29], we design a muti-turn conversation between LaneLM and a pretrained teacher. For an input image ${\mathbf{X}}_{\mathbf{v}}$, we consider the pretrained detecor [6], which provides pseudo keypoint labels $L_q$ as init queries for laneLM. Then we generate multi-turn conversation data $(L_q^1,L_{gt}^1;...;L_q^N,L_{gt}^N)$, where N, the max number of turns is also the max number of lanes in each image and $L_{gt}^1$ is the first lane of ground truth in the image. We organize them as a sequence by treating ground truth as the answer of pseudo label. Therefore, the multi-modal self-supervised label S for image ${\mathbf{X}}_{\mathbf{v}}$ can be expressed as follows:

$$S=(L_q^1\circ L_{gt}^1,...,L_q^N\circ L_{gt}^N,{\mathbf{X}}_{\mathbf{v}})$$

(10)

where ∘ means concatenating two sequences. We adopt the bipartite matching to find the matching that minimizes the distance of the start points between the query sequence $L_q^i$ and the answer $L_{gt}^j$, and then take the matched pair $(L_q^i,L_{gt}^j)$ as the self-supervised label for each lane.

Loss. We only adopt standard loss in the decoder-only language models. We train our model by minimizing negative log-likelihoods of keypoint tokens conditioned on the visual inputs ${\mathbf{X}}_{\mathbf{v}}$ with cross entropy loss at each time stamp t:

$$\mathrm{maximize}\sum _{t=1}^T(\log P\left(x_t\right|x_{<t},{\mathbf{X}}_{\mathbf{v}})+\log P\left(y_t\right|y_{<t},{\mathbf{X}}_{\mathbf{v}}))$$

(11)

where T is the length of the sequence.

Inference. We sample tokens from the model likelihood $P\left(x_t\right|x_{<t},{\mathbf{X}}_{\mathbf{v}})$ and $P\left(y_t\right|y_{<t},{\mathbf{X}}_{\mathbf{v}})$ using the argmax sampling as illustrated in Eq. 3.The same as language models, we apply the standard greedy search with fixed length and EOS (the End of Sequence token) stop criteria to generate $xy$ tokens at the same time (i.e. we stop prediction when EOS token is predicted or the current sequence reaches the max length). After obtaining the discrete tokens, we de-quantize them to get continuous coordinates.

To speed up the inference, in addition to adopting the parallel strategy outlined in Section 3.2, each level of visual sequence ${\mathcal{L}}_i$ is cached into the decoder’s corresponding cross-attention layer. Further more, we adopt the KV-cache strategy in each causal self-attention layer only at inference time.

Prompting strategy. We have the following three prompting strategies. (1) A regression network is employed to provide the two initial keypoints, for each lane. LaneLM is responsible for completing the remaining keypoints. The regression network (we use CLRNet [6]) only gives start points for each lane rather than the holistic lane, which is easier than the LD task. Keypoint-based methods [13,14,15] use the similar start point regression. However, they are struggling to design keypoint decoding strategy while we leverage contextual representation of lane and just rollout the keypoints. (2) Pseudo labels in Section 3.4 are given as questions, our language decoder leverage the few-shot capability to generate the answers and refine the read input or just read them out. (3) We simulate the annotation process performed by annotators by introducing a certain level of noise to the ground truth (randomly shifting the x-coordinates by -5 to 5 pixels) to demonstrate LaneLM’s capability of interaction. (4) We give some simple instruction tokens to LaneLM to accomplish some post-processing tasks.

3.5. Hallucination Removal

Language models are prone to hallucination, generating plausible but non-factual content [44]. As shown in Figure 6(a), like most language models, LaneLM may exhibit hallucination on side lanes. In order to mitigate such hallucination, we have developed a threshold filtering strategy HR (Hallucination Removal) based on clustering at the 85th percentile, in which points with offsets of adjacent x-coordinates exceeding twice the 85th percentile, along with their subsequent points, will be filtered out. Figure 6(a) shows the mitigation of hallucination after filtering by HR.

4. Experiments

4.1. Implementation Details

Model variants. We have three model variants and their trainable parameters are shown in Figure 5. With the different hidden size of the decoder, the visual encoders of LaneLM-128, LaneLM-256, and LaneLM-512 are ResNet18, ResNet34 [38] and DLA34 [39].

Hyperparameters. We generate 40 points ($T=40$) for CULane [45], 56 for TuSimple [46] and LLAMAS [47]. We only apply HR for lanes whose valid number of points is greater than $\mathcal{N}=10$ in Alg. 1. On CULane [45] and LLAMAS [47], the max number of lanes $N=4$, while on TuSimple [46]$N=5$. The padding value for x-offsets is 0, while T for y. EOS is a special token that represents the end of the sentence. $(x,y)$ is an EOS token when $x=0$ or $y=T$. In our additional study, we use EOS token as cmd to control the output and LaneLM stops its inference until the second EOS token is predicted.

Algorithm 1: HR(·)

Training environment. All experiments are carried out with a single NVDIA GeForece RTX 4090, 128 batch size, 800 $n_{bins}$ and 100 training epochs and we report the best results in Table 3, Table 2 and Table 4. All visual encoders are frozen in the training phase and pretrained from [6].

4.2. Real-Time Performance and Computational Overhead

Figure 5 shows that the trainable parameter of LaneLM is extremely low. We aim to make LaneLM like an adapter, bridging the visual and language modalities. Similar to multimodal models, there is no need for pre-training the visual encoder. LaneLM is easy to plugin into any LD models.

Table 1 reports FPS (Frames Per Second) of LaneLM. We did not measure the inference time for the visual encoder f in Eq. 7 because it is frozen and shared with the teacher model CLRNet [6], which can also be replaced with other pretrained models. We set batch size to 1 and measure the max memory allocated during inference using the corresponding function in pytorch. T=80 indicates that we rollout 80 keypoints for each lane. The impact of sequence length on memory overhead is minimal when the batch size is relatively small.

The real-time performance of LaneLM can be attributed to our cache strategies. We cache the visual sequence of each level feature in each cross-attention layer in language decoder and adopt KV-cache at inference time. All lanes in the same image can be batch processed after padding. Lane2Seq [3] is an AR LD model, which can not reach real-time performance.

Figure 5. The comparison of trainable parameters in LaneLM with some backbones.

Figure 5. The comparison of trainable parameters in LaneLM with some backbones.

4.3. Datasets

Our experiments are conducted on three lane detection benchmark datasets: CULane [45] and TuSimple [46] and LLAMAS [47].

CULane [45] is a large-scale dataset that poses significant challenges for lane detection. It encompasses nine challenging categories, including crowded scenes, nighttime conditions, crossroads, etc. It has 100,000 images. Each image in CULane has a resolution of 1640 × 590 pixels.

Tusimple [46] lane detection benchmark is among the most widely used datasets in lane detection. It consists solely of highway scenes and contains 3268 images for training, 358 for validation, and 2782 for testing. These images have a resolution of 1280 × 720 pixels.

LLAMAS [47] is a large lane detection dataset that has more than 100,000 images. The annotations in LLAMAS are automatically generated using high-definition maps. The images in LLAMAS are all from highway scenarios. For evaluation purposes, the CULane’s F1 metric is utilized. The annotations of the testing set are not publicly available.

4.4. Evaluation Metric

For CULane [45] and LLAMAS [47], we use the F1@50 measure as the evaluation metric. We compute intersection-over-union (IoU) of two lanes, which is defined as the IoU of the two lane masks within 30 pixels and used to determine whether a sample is true positive (TP), false positive (FP), or false negative (FN), between prediction lane and ground truth. Predicted lanes with an IoU greater than a threshold (50%) are regarded as true positives (TP). Further, F1 score is calculated as follows:

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}\mathrm{F}1={\displaystyle \frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(12)

In particular, since there are no lanes in the cross-road scenes, the false negative will be adopted as the evaluation metric.

For Tusimple [46], we report three official metrics: false positive rate, false negative rate and accuracy:

$$\mathrm{accuracy}={\displaystyle \frac{{\sum }_{clip}{C}_{clip}}{{\sum }_{clip}{S}_{clip}}}$$

(13)

where $S_{clip}$ is the total number of ground truth lane points and $C_{clip}$ is the number of correct lane points in prediction of a clip. A predicted lane is considered correct if more than 85% of its predicted lane points are within 20 pixels of the ground truth. In addition, we also report the F1 score on TuSimple.

4.5. Benchmark Results

Performance on TuSimple. Table 2 presents the performance comparison with SOTA methods on TuSimple. * indicates that LaneLM only uses prompting strategy (1) from Section 3.4. The performance difference on this dataset is very small because TuSimple is a dataset of small scale and it contains single scene only. Our prompting strategy has a limited effect on performance. The result in this dataset seems to be saturated. Therefore, The prediction can easily overlap with the ground truth. It is hard to get a significant improvement on TuSimple.

Performance on CULane. Table 3 reports the main results on CULane. Our model receives two adjacent keypoints output from CLRNet [6] as init prompts for each lane and rollouts the remaining keypoints in the * version. To explore the interactive capability of LaneLM, (2-kp) denotes that the holistic lane predicted from CLRNet is given and two adjacent ground truth keypoints with ramdom shift at the commencement of each lane are also supplied to enhance model performance.

We observed that with the increase in the number of keypoint prompts, F1-score has significantly improved in most corner cases, which can be attributed to the powerful language modeling capability. The results on CULane demonstrate that our model has surpassed the SOTA [9] after prompting and is particularly suitable for application in the annotation of corner cases. More results can be found in Figure A1

Analysis on Limitations. (1) In the * version, LaneLM underperforms CLRNet because, in Eq. 10, LaneLM actually predict pseudo-labels from CLRNet i.e. the knowledge of this part in LaneLM is distilled from the CLRNet. (2) LaneLM with fewer keypoint prompts is worse than the * version because, in the training sequence, a sudden jump occurs at the junction between the pseudo-label and the ground truth (see Eq. 10), which disrupts the contextual semantic information and confuses the model. It has been observed that the model often hallucinates on the side lanes, indicating that the model struggles to cope with abrupt changes in semantic information.

Performance on LLAMAS. The result on the LLAMAS is shown in Table 4. LaneLM-512 outperforms PolyLaneNet [20] by 7.05 and LaneATT [8] by 2.08. The training strategy is slightly different with CULane and TuSimple. We directly use $L_{gt}$ as self-supervised label S and $Lq$ is not used during training, which is different with Eq. 10. Thus, we can avoid knowledge distilling from the teacher model.

4.6. Ablation Study

We have maintained a minimalist design for the model. LaneLM has neither complex trick nor sophisticated structure. To investigate the impact of pure language modeling architectures on LD tasks, we did not design complex spatial attention mechanisms.

Table 5 illustrates ablation study on CULane and shows effects of each module and prompting strategy in our method, where cmd refers to the command token. We use LaneLM-512 (0-kp) with DLA34 encoder but without HR (Hallucination Removal), as our baseline to conduct our experiments. Experiments have proven that, compared with other methods which add the attention module to LD model, following the prompting protocol for inference can improve the accuracy of the model by a large margin in complex scenarios, demonstrating LaneLM’s powerful interactive capability. Modules specifically designed to handle spatial information, such as FPN [40], can also significantly enhance the model’s performance.

Additionally, to explore the instruction-following ability of LaneLM, we conduct extra ablation study on command token that sets the predictions to null, in which a cmd token is given at the beginning of the sequence when it comes to no lanes in the image. This instruction can reduce the post-processing of autonomous driving, enabling the visual model to receive semantic feedback from human being and other models (e.g. large language models).

Analysis on hallucination. Current large language models are still struggling with hallucination. Figure 6(a) shows hallucination in LaneLM. Eq. 10 endows the model with the capability of VQA but it makes it easier for the model to predict cyclic sequences. Figure 6(a) illustrates that the model has learned the abrupt change points that connecting $L_q$ and $L_{gt}$ on the side. LaneLM has learned the contextual representation of abrupt change points and consequently results in hallucination.

Study on hallucination removal. As shown in Figure 6(c), we collect the statistic results of lanes with hallucination from TuSimple, in which the red line demonstrates that a significant number of x-offsets with abnormal values have been filtered out by HR. As shown in Figure 6(d), to find the best percentile, we conducted a search for the performance of $\theta$ on the F1 and Acc metrics on TuSimple using LaneLM-128 (0-kp). We report the detailed metric values in Table 6.

Study on $n_{bins}$ per pixel. To investigate the impact of the resolution of bins (i.e., $n_{bins}$ per pixel) on performance, we vary the $n_{bins}$ and conduct our training on TuSimple using LaneLM-512 (0-kp). We report our result in Table 7. Most LD methods resize images into 800 pixel width in training and inference. $n_{bins}$=800 aligns with our intuition. Using large $n_{bins}$, which increases the vocabulary size, has no significant improvement and it will slow down training. Choosing a larger $n_{bins}$ does not yield benefits because the model’s hidden size is too small relative to $n_{bins}$, making classification difficult.

Qualitative results. We display the qualitative results in Figure 6(a-b). After keypoint prompting (2-kp), LaneLM can recall more lanes at night. The results show that LaneLM can effectively detect lanes in the multi-lane scenario (see Figure 6(a) right). The results demonstrate that LaneLM can be prompted from the semantic feedback. Semantic information directly determines the spatial attention of the model.

Figure 6. (a) The left image shows the hallucination on the side lane, while the right image is post-processed with the HR algorithm. (b) The left image shows the CLRNet prediction and the right is LaneLM prediction. (c) shows the frequency distribution of horizontal distances between adjacent keypoints, with the x-axis as the ln(frequency) and the horizontal axis as the distance values. (d) shows the impact of different $\theta$-percentile thresholds on F1-score and accuracy.

Figure 6. (a) The left image shows the hallucination on the side lane, while the right image is post-processed with the HR algorithm. (b) The left image shows the CLRNet prediction and the right is LaneLM prediction. (c) shows the frequency distribution of horizontal distances between adjacent keypoints, with the x-axis as the ln(frequency) and the horizontal axis as the distance values. (d) shows the impact of different $\theta$-percentile thresholds on F1-score and accuracy.

5. Discussion

Current autonomous driving technologies mainly focus on the fusion of visual sensors like LiDAR [52,53,54], radar, and cameras. Visual modality often fail in corner cases. Many recent works [55,56,57] has concentrated on Large Language Models (LLMs) to address the long-tail distribution in corner cases. However, they cannot interact with the visual models. Future autonomous vehicles will incorporate multiple redundant sensors for visual perception. In harsh conditions where sensors fail, a parallel LLM will leverage semantic information to prompt and enhance visual tasks, making the system more robust.

6. Conclusions

In this paper, we present autoregressive LaneLM and treat the LD task as visual question answering. We use the language model to make LD task promptable and interactive. The lanes and images are transformed into multi-modal embedding sequences and interact with each other in LaneLM. Then we alleviate the hallucination by HR. Experiments show our proposed method outperforms previous unpromptable SOTA methods. The design of LaneLM is quite simple and doesn’t have overly complicated structures. We hope that LaneLM can serve as a baseline for language-modeling-based approaches. We believe that our work will pave the way towards unification for LLM and visual models.

Appendix A.1

Theorem A1.

cross-attention mechanism does not leak the information of the future query tokens.

Proof 

(Proof). Let’s revisit the cross-attention function:

$$\mathrm{CrossAtt}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(A1)

Obviously, the softmax function preserves the order information of the query embeddings and does not leak information about future embeddings. Let n,m,D and ${\overrightarrow{\mathbf{q}}}^{1\times D}$ be the sequence length of the query tokens, the value tokens, the hidden size and the embedding of the query token, respectively. According to the principle of matrix block calculation, the information interaction in cross-attention is equivalent to:

$$Q^{n\times D}{\left(K^{m\times D}\right)}^T{V}^{m\times D}=\left(\begin{array}{c}{\overrightarrow{\mathbf{q}}}_1^{1\times D}\\ {\overrightarrow{\mathbf{q}}}_2^{1\times D}\\ ...\\ {\overrightarrow{\mathbf{q}}}_n^{1\times D}\end{array}\right){\left(K^{T}V\right)}^{D\times D}=\left(\begin{array}{c}{\overrightarrow{\mathbf{q}}}_1\left(K^{T}V\right)\\ {\overrightarrow{\mathbf{q}}}_2\left(K^{T}V\right)\\ ...\\ {\overrightarrow{\mathbf{q}}}_n\left(K^{T}V\right)\end{array}\right)$$

(A2)

Therefore, cross-attention preserves the order information of the query tokens, preventing any information leakage and every query embedding is conditioned on the whole visual information $K^{T}V$. □

Figure A1. F1@50 in 9 categories on CULane for keypoint prompting. LaneLM reaches the state of the art on CULane [45] with keypoint prompting, significantly outperforming unpromptable SOTA method [9] with as few as 4 keypoint prompts at the beginning of each lane. We found that in corner cases, the marginal benefit of prompting is quiet large.

Figure A1. F1@50 in 9 categories on CULane for keypoint prompting. LaneLM reaches the state of the art on CULane [45] with keypoint prompting, significantly outperforming unpromptable SOTA method [9] with as few as 4 keypoint prompts at the beginning of each lane. We found that in corner cases, the marginal benefit of prompting is quiet large.