NPLSD: Accelerating Line-Segment Detection on NPU Microcontrollers

1. Introduction

Line-segment detection (LSD) is a cornerstone of visual perception for robotics, autonomous navigation, and industrial metrology. While classical methods such as LSD [1] and ELSED [2] have long served as practical solutions, their accuracy is limited in challenging scenarios. Deep learning has reshaped LSD in two waves. The first introduced convolutional approaches including L-CNN [3], HAWP [4,5], and F-Clip [6], demonstrating the superiority of end-to-end learning. DeepLSD [7] later combined learned gradients with classical refinement. The second wave brought transformer-based detectors such as LETR [8], DT-LSD [9], and LINEA [10], which set new state-of-the-art results. These models, however, are designed for GPUs. Even efficient variants like M-LSD [11] and EM-LSD [12] target mobile SoCs with substantial memory.

The embedded landscape has shifted with the integration of on-chip NPUs into microcontrollers. The STM32N6 pairs a Cortex-M55 with the Neural-ART NPU [13,14], a convolutional accelerator that raises throughput by an order of magnitude. The ST Edge AI toolchain [15] maps convolutional networks onto this accelerator.

Despite this promise, line-segment detection remains absent from NPU microcontrollers [16]. The primary impediment is architectural. Transformer detectors rely on attention, grid-sampling, and normalization, operators unsupported by convolution-oriented NPUs. These execute on the host CPU, and attention mechanisms can exceed the on-chip activation budget entirely. A transformer detector thus derives negligible benefit from the NPU.

This mismatch is concrete: LINEA-N’s backbone is expressible on the NPU, but its transformer encoder-decoder is not. The self-attention score tensor alone exceeds the available SRAM, and grid-sampling has no accelerator primitive. This gap motivates NPLSD, a pair of line-segment detectors designed for the NPU. Both variants are built from supported operators only. NPLSD-H retains the convolutional HGNetv2-B0 backbone and replaces the transformer head with a fully-convolutional design. NPLSD-M adapts the M-LSD-tiny trunk to the accelerator’s operator set. Every operator maps onto the NPU.

Both detectors are warm-started from ImageNet and trained on ShanghaiTech Wireframe with an identical head and recipe. NPLSD-H reaches ${\mathrm{sAP}}^{10}=37.9$ (from-scratch: 33.3), so initialization accounts for 4.6 points. NPLSD-M reaches 41.9 with 4.2× fewer parameters. The trunk is the only variable, forming a controlled ablation. Since the NPU is an integer engine, both models are quantized to int8. Quantization costs 2.0 points on the dense trunk and 0.8 on the depthwise trunk.

The contributions of this paper are as follows:

• The transformer-NPU gap is characterized at the operator level. Profiling LINEA-N against the Neural-ART instruction set reveals that only its backbone is expressible on the accelerator; the decoder cannot fit on-chip at all.
• Two NPU-native detectors, NPLSD-H and NPLSD-M, are introduced. Both networks are constructed entirely from the supported operator set and share an identical head and training recipe, enabling a controlled ablation of the trunk.
• ImageNet warm-starting improves ${\mathrm{sAP}}^{10}$ by 4.6 points at no architectural cost. The depthwise trunk loses only 0.8 points to int8 quantization, versus 2.0 for the dense trunk, while achieving higher accuracy with 4.2× fewer parameters.

The remainder of this paper is organized as follows. The transformer-NPU gap is characterized in Section 2. The NPLSD architectures and training are presented in Section 3, together with the int8 quantization procedure. Experimental results are reported in Section 4. Discussion and concluding remarks are provided in Section 5 and Section 6.

2. The NPU Gap: An Architectural Analysis

The characterization proceeds by establishing why transformer-based detectors cannot benefit from the STM32N6 NPU. Table 1 presents the operator mapping for LINEA-N, derived from its exported graph and the accelerator’s documented operator support.

The convolutional components of the HGNetv2-B0 backbone and the CNN cross-scale fusion in the hybrid encoder are mappable to the NPU. In contrast, the transformer-specific operators are not supported. GridSample, the bilinear sampling core of the line-attention deformable decoder, has no Neural-ART primitive and can only execute on the Cortex-M55. The same applies to LayerNorm, the attention Softmax, and the TopK and Gather operations for query selection.

Critically, the decoder’s self-attention operates over 1,100 queries and materializes an $[8,1100,1100]$ score tensor. This tensor is approximately 39 MB in floating-point, which exceeds the on-chip activation budget. The full decoder is therefore unplaceable. Only the backbone is expressible on the accelerator, and it represents a minority of the network’s computation. The transformer-specific operations are consequently relegated to the host CPU. The accelerator is thus architecturally wasted on a transformer detector. This is the gap that NPLSD is designed to close.

3. NPLSDH: Architecture and Training

This section presents the design and training of the two NPLSD variants. The architectures comprise a trunk for feature extraction, a lightweight fusion stage, and a shared dense head for line parameter regression. The training objective and optimization procedure are then formulated.

3.1. Design Principle

The design of NPLSDH is governed by a single, non-negotiable constraint: every operator must execute natively on the Neural-ART NPU. This requirement is not merely an optimization consideration; it is a fundamental precondition for the network to benefit from the STM32N6’s accelerator at all. The operators supported by the NPU include convolution, batch normalization (folded into convolution during inference), ReLU, the bounded Clip activation (ReLU6), element-wise addition, concatenation, nearest-neighbor resize, and max-pool. Conversely, attention, grid-sampling, layer normalization, softmax, and top-k selection are explicitly excluded from the NPU’s instruction set. Both NPLSD variants are therefore constructed exclusively from the supported operator set.

3.2. NPLSD-H Architecture

The NPLSDH architecture comprises three main components: backbone, neck, and head.

• Backbone: HGNetv2-B0 [17,18], the convolutional backbone used by LINEA, is adopted. It produces feature maps at strides 8, 16, and 32 with 256, 512, and 1024 channels, respectively. Unlike the detection-transformer usage, ordinary batch normalization is kept rather than frozen affine layers. This ensures that during quantization, each batch normalization layer folds cleanly into its preceding convolution. The backbone is warm-started from the publicly available ImageNet-pretrained HGNetv2-B0 weights, with the optional affine layers dropped; every remaining backbone tensor is initialized from the checkpoint. A from-scratch run of the identical architecture is kept as an ablation.
• Neck: A lightweight convolutional feature pyramid fuses the three backbone scales top-down. It uses 1x1 lateral convolutions, nearest-neighbor 2x upsampling (a supported NPU resize operation), and 3x3 smoothing convolutions. An additional upsampling stage produces a single stride-4 map. For a 320x320 input, this yields an 80x80 grid for the dense head.
• Head: An F-Clip-style dense line head [6] predicts a six-channel output per grid cell. This includes a line-center likelihood (one channel), a sub-pixel center offset (two channels), a half-length (one channel), and a double-angle orientation (two channels). The double-angle encoding, defined as $(\cos 2\theta ,\sin 2\theta )$, removes the 180-degree direction ambiguity inherent in line segments. The head emits raw feature maps. The center map is decoded off the NPU through a lightweight arithmetic pass comprising sigmoid activation, 3x3 max-pool non-maximum suppression, and per-peak endpoint reconstruction.

The complete network has 2.63M parameters, smaller than LINEA-N’s 3.9M. Its exported graph contains only Conv, ReLU, Add, Concat, Resize, and MaxPool nodes, all of which map onto the accelerator.

3.3. NPLSD-M: Adapting the M-LSD Trunk

On mobile SoCs, M-LSD-tiny remains the reference lightweight detector, reporting ${\mathrm{sAP}}^{10}=58.0$ at 512-pixel input with 0.6M parameters [11]. Direct reuse of this trunk on the N6 is not feasible. The released model is float-only TFLite, whereas the integer accelerator does not execute floating-point convolutions. Its bilinear upsampling with aligned corners lacks a Neural-ART primitive, and the TFLite-style asymmetric padding triggers a code-generation fault. Furthermore, at 512-pixel resolution, a single early tensor of $96\times 256\times 256$ occupies 6.3 MB in int8, exceeding the N6’s 4.2 MB internal activation SRAM.

NPLSD-M is the accelerator-native redesign of this trunk, produced by the same methodology as NPLSD-H. The MobileNetV2 inverted-residual stack (blocks 0 through 10, taps at strides 4, 8, and 16) is warm-started from ImageNet. The A/B fusion blocks of M-LSD are retained, with their bilinear interpolation replaced by nearest-neighbor resize. Asymmetric pads become symmetric convolution padding, the four-channel input becomes standard RGB, and the M-LSD prediction head is replaced by the same F-Clip dense head used in NPLSD-H, at 320-pixel input. ReLU6 activations are retained as Clip nodes, which belong to the accelerator’s supported operator set. The exported graph holds 44 Conv, 20 Clip, 10 ReLU, 8 Add, 3 Concat, and 2 Resize nodes, with no Pad, for 0.62M parameters and 1.74 GMACC.

Since trunk choice is the only difference between the two variants, their comparison constitutes a controlled ablation of trunk families under a fixed head, loss, dataset, resolution, and schedule.

3.4. Training

Both variants are trained on the ShanghaiTech Wireframe benchmark. Ground-truth segments are rasterized into dense targets. A CenterNet-style Gaussian is placed on the center map at each ground-truth line center. Regression targets for offset, length, and angle are assigned to each positive center cell.

The loss function combines a penalty-reduced focal loss on the center map with L1 or smooth-L1 regression losses on the offset, length, and unit-normalized angle at positive cells. The total loss is defined as:

$$\mathcal{L}={\mathcal{L}}_{center}+{\lambda }_{reg}\left({\mathcal{L}}_{offset}+{\mathcal{L}}_{length}+{\mathcal{L}}_{angle}\right)$$

(1)

where ${\lambda }_{reg}$ balances the classification and regression terms.

Optimization is performed with AdamW using a cosine annealing learning rate schedule. Each model is trained for 120 epochs with a batch size of 16, selecting the checkpoint with the best validation ${\mathrm{sAP}}^{10}$. Input images are resized to 320x320. Evaluation is conducted using structural average precision (sAP) on the 128x128 reference space following the standard Wireframe benchmark protocol.

Warm-started from ImageNet, NPLSD-H achieves ${\mathrm{sAP}}^{10}=37.9$, while NPLSD-M reaches 41.9. Training NPLSD-H from scratch under the same recipe yields 33.3, indicating that ImageNet initialization contributes 4.6 points without any architectural modification. Notably, the accelerator operator set does not constitute the accuracy bottleneck; holding the graph fixed and varying only the initialization produces a larger accuracy shift than the gap between the two trunk families. Both results remain below transformer-based detectors, which is consistent with compact convolutional models evaluated at 320-pixel input.

Figure 1. NPLSD architecture comprises two accelerator-native trunks and a shared dense head. The dense HGNetv2-B0 of NPLSD-H and the depthwise MobileNetV2 stack of NPLSD-M feed a six-channel F-Clip head at stride 4, followed by an arithmetic decode. Every operator (Conv, ReLU/Clip, Add, Concat, Resize, MaxPool) maps onto the Neural-ART NPU.

Figure 1. NPLSD architecture comprises two accelerator-native trunks and a shared dense head. The dense HGNetv2-B0 of NPLSD-H and the depthwise MobileNetV2 stack of NPLSD-M feed a six-channel F-Clip head at stride 4, followed by an arithmetic decode. Every operator (Conv, ReLU/Clip, Add, Concat, Resize, MaxPool) maps onto the Neural-ART NPU.

3.5. Export and int8 Quantization

Since the Neural-ART is an integer engine, the configuration of record is the quantized one, and it is prepared entirely off-device. Each trained model is exported to ONNX at 320x320 input resolution and simplified to static shapes, yielding a clean graph over the supported operator set (the NPLSD-M graph is exported with symmetric padding from the start and contains no Pad node). Static int8 quantization [19] is then applied in QDQ (Quantize-Dequantize) format with per-channel weights, calibrated on 200 real Wireframe images; for NPLSD-M the bounded Clip activations are included in the quantized set. Quantization compresses the weights roughly fourfold, to 2.55 MB for NPLSD-H and 0.65 MB for NPLSD-M. All int8 results in Section 4 are produced by executing these quantized graphs in ONNX Runtime over the full validation split. The training notebooks and the evaluation code reproducing every accuracy figure are publicly available on Kaggle [20,21,22], together with the dataset and the ImageNet-pretrained trunk checkpoints [23].

4. Results

4.1. Qualitative Detections

Figure 2 shows NPLSD predictions on held-out ShanghaiTech Wireframe validation images spanning diverse indoor and outdoor man-made scenes. Across bedrooms, living rooms, facades, and terraces, the models recover the dominant scene structure, including wall, ceiling, window, door, and furniture edges together with the perspective convergence of facade and floor boundaries. These predictions are generated directly by the fully convolutional, accelerator-mappable networks. Fine, low-contrast, and heavily occluded segments are more frequently missed, with NPLSD-M recovering visibly more of the long structural edges, in line with its higher per-length recall (Table 5). The results show the complete set of raw decoded outputs rather than a curated subset, providing qualitative evidence that native line-segment detection on a microcontroller NPU can produce coherent wireframe reconstructions.

4.2. Accuracy on the Wireframe Benchmark

Table 2 lists structural average precision over the complete 462-image validation split, evaluated at the three standard matching thresholds. Six configurations are compared: each variant in fp32 and int8, together with the from-scratch NPLSD-H ablation pair. Warm-started NPLSD-H reaches ${\mathrm{sAP}}^{10}=37.9$; after static quantization with 200 real calibration images (QDQ format, per-channel weights), the int8 network intended for the accelerator retains 35.9. NPLSD-M reaches 41.9 and maintains 41.1 in int8, a loss of 0.8 points compared with 2.0 for the dense trunk. Degradations of this order are typical when compact convolutional networks are quantized to int8 [24]. Precision and recall are affected almost uniformly, as the curves in Figure 3 indicate, so accuracy statements made for the fp32 model carry over to the quantized one. At a confidence threshold of 0.15, the warm NPLSD-H proposes 129 segments per image on average against 74.2 annotated (NPLSD-M: 165.5), a proposal surplus that MiLSD [25] also exhibits and that the ranked sAP protocol absorbs.

For context, Table 3 situates these results among published line detectors across hardware classes. GPU-class detectors such as HAWP [4] at 66.5 and L-CNN [3] at 62.8 assume workstation memory budgets. M-LSD-tiny [11] brought learned detection to mobile SoCs at 58.0, yet still relies on gigabytes of RAM. Below the mobile tier, the literature is, to our knowledge, empty. The only microcontroller results available are the 25k F-Clip baseline (10.6), MiLSD (24.1) [25], and the present work, all three originating from the same research effort, reflecting the novelty of this design space. Within this comparison, NPLSD-H improves ${\mathrm{sAP}}^{10}$ by 11.8 points over MiLSD, and NPLSD-M by 17.0 points, with both networks constructed entirely from the accelerator’s supported operator set. NPLSD-M also closes 41 percent of the gap separating MiLSD from the mobile-SoC M-LSD-tiny, while remaining int8 and microcontroller-scale at 320-pixel input. For reference, the classical LSD algorithm of von Gioi et al. [1], evaluated on a desktop CPU, attains an ${\mathrm{sAP}}^{10}$ of 8.8. The int8 NPLSD-M model achieves approximately 4.7 times this score while targeting substantially more resource-constrained hardware.

4.3. Cross-Dataset Generalization: YorkUrban

Following the standard protocol established by L-CNN, HAWP, F-Clip, LETR, and M-LSD [3,4,6,8,11], which train on Wireframe and report YorkUrban as a zero-shot test, both NPLSD variants are evaluated on the 102 YorkUrban images without any fine-tuning (Table 4). All published detectors exhibit a sharp drop on this transfer; GPU-class models retain 42 to 45 percent of their Wireframe ${\mathrm{sAP}}^{10}$, while M-LSD-tiny retains 42 percent. NPLSD-M retains 41 percent (17.2 of 41.9), indicating that the accelerator-constrained design does not compromise cross-dataset generalization. The observed drop is expected: YorkUrban contains outdoor scenes and annotates only lines along the dominant Manhattan directions, with 118.8 ground-truth lines per image. Out of domain, quantization costs NPLSD-M only 0.3 points.

Recall stratified by ground-truth length appears in Table 5. For the warm NPLSD-H, short and medium segments are recovered at 54.0 and 62.2 percent, and segments longer than 25 units of the ${128}^2$ label space at 44.0 percent. NPLSD-M leads across all buckets at 61.0, 65.5, and 48.0 percent, respectively. Compared with the from-scratch model, pretraining improves long-segment recall by 5.9 points, the largest per-bucket gain. A long-line deficit nevertheless persists. This pattern is consistent with expectations. MiLSD, built on the same center, length, and angle encoding, exhibits similar behavior [25]; thus, the effect appears to arise from the representation family rather than from the NPU constraint introduced here. Multi-scale heads would be a natural extension and remain compatible with the accelerator’s operator set.

The behavior of the dense head can be characterized through direct examination of its output maps. Figure 4 renders the raw $80\times 80$ output channels for one validation scene: center likelihood peaks at segment midpoints, the half-length map grows along extended structural edges, and the decoded orientation stays piecewise constant across surfaces that share a direction. Precisely these three quantities are assembled into segments by the arithmetic decode.

4.4. Model Footprint

NPLSD-H has 2.63M parameters and 2.69 GMACC at 320x320; NPLSD-M has 0.62M parameters and 1.74 GMACC. Quantized to int8, the NPLSD-H weights occupy 2.55 MB, compared with 10.05 MB in floating-point; NPLSD-M compresses to 0.65 MB (Figure 5). Both quantized weight files fit comfortably within microcontroller-class flash budgets, roughly four times smaller than their float counterparts. Notably, the 0.65 MB NPLSD-M is smaller than MiLSD-class detectors while achieving 17 points higher in ${\mathrm{sAP}}^{10}$.

5. Discussion and Limitations

NPLSD demonstrates that line-segment detection can be made compatible with a microcontroller NPU through co-design with the accelerator’s operator set. The controlled trunk ablation carries its own lesson. With head, loss, data, and schedule held fixed, the depthwise MobileNetV2 trunk outscores the dense HGNetv2 trunk by 5.2 int8 points at 4.2 times fewer parameters, and it is markedly more robust to quantization. Both variants are nevertheless retained. Depthwise and dense convolutions place very different demands on an accelerator’s MAC array and memory system; accuracy per parameter need not translate into accuracy per millisecond. Which trunk wins on silicon is a question this paper deliberately leaves to hardware evaluation.

Two limitations frame the contribution. First, accuracy: at ${\mathrm{sAP}}^{10}=41.1$ in int8, NPLSD-M still trails transformer-based detectors and the 512-pixel mobile M-LSD-tiny. Knowledge distillation [27] and multi-scale heads are natural next steps, and warm-starting, previously listed as future work, is now measured at +4.6 points. Second, evaluation covers two datasets at a fixed resolution, and all results are simulation-level (ONNX Runtime). On-silicon latency, memory placement, and energy are outside the scope of this paper. Despite these limitations, NPLSD provides a baseline and methodology for NPU-ready geometric vision on microcontrollers.

6. Conclusions

This paper presented NPLSD, a pair of line-segment detectors designed for an NPU-equipped microcontroller. Motivated by an architectural gap, in which transformer detectors are largely inexpressible in a convolution accelerator’s operator set, both variants replace non-convolutional structure with a fully-convolutional design. By construction, every operator belongs to the Neural-ART supported set.

Warm-started on ShanghaiTech Wireframe, the 2.63M-parameter NPLSD-H reaches ${\mathrm{sAP}}^{10}=37.9$ (35.9 int8) and the 0.62M-parameter NPLSD-M reaches 41.9 (41.1 int8), 17.0 points above the best prior microcontroller result. Static int8 quantization, the format the integer accelerator requires, costs under one point on the depthwise trunk. The controlled ablation shows that initialization (+4.6) matters as much as trunk choice (+5.2), and that neither is limited by the accelerator operator set. NPLSD establishes that accelerator-compatible line-segment detection on microcontrollers is feasible at useful accuracy. Closing the remaining accuracy gap through knowledge distillation, and characterizing both variants on silicon, remain future work.