An Improved Deep Learning Framework for In-Situ Detection of Geometric Keypoints of Heliostats in Concentrated Solar Power Plants

1. Introduction

Due to their economical strength and higher safety on mass storage of energy, concentrated solar power (CSP) plants are playing an increasingly important role in modern renewable power systems [1,2]. The working efficiency of a CSP station is largely dependent on the tracking accuracy of the thousands of heliostats that act as the sun-tracking devices in a CSP plant. If this accuracy is not well controlled, the station may fail to reach its designated output power because of substantial solar-flux spillage and non-uniform thermal loading on the central receiver, which can even shorten the receiver’s service life [3,4]. Calibrating heliostat tracking accuracy is therefore a routine requirement for the stable operation of a commercial CSP station [3,4,5].

At present, most calibration methods work offline or intrusively. For example, Beam Characterization Systems (BCS) require maneuvering individual heliostats away from the central receiver to direct their reflected beams onto a Lambertian-like target for measurement of tracking errors [6,7,8,9]. Using BCS to calibrate all heliostats of a 50 MW CSP plant may take weeks, during which dynamic operational errors accumulate without compensation. Consequently, modern solar engineering is progressively transitioning from static offline characterization toward dynamic online servo control. This paradigm shift necessitates the deployment of high-frequency, non-intrusive measuring gauges [10,11,12,13]. Recent studies have begun to apply learning-based vision directly to heliostat metrology, including a hybrid YOLOv5-plus-Hough-Transform pipeline for corner retrieval and downstream PnP pose estimation [14], differentiable ray-tracing surface reconstruction [15,16], and the broader landscape of AI-driven heliostat control surveyed in [17].

Embedding machine vision feedback into the control loop of a heliostat tracking system faces several challenges. First, the visual cues of heliostats are often corrupted by over-exposure and mirror occlusions of the heliostat field [18,19,20,21,22]. Traditional keypoint detectors and pose estimation networks [23,24,25,26,27,28] often suffer from structural instability and high-frequency coordinate jitter in such situations. Secondly, the highly reflective nature of the heliostat surface makes visual detection of objects difficult. A highly specular surface has little intrinsic texture and instead displays the texture of reflected objects. The signal that a generic keypoint detector relies on, namely a stable local appearance pattern at a fixed object-relative position, therefore does not exist for the mirror facet. The most reliable visual cues are the corner points of the mirror facets, which appear as a high-contrast quadrilateral against an inhomogeneous specular background. This observation directly motivates a corner-based, geometry-aware formulation of the measurement task rather than a holistic pose-regression formulation.

To bridge the gap of vision-based feedback control, this study presents a deep-learning-based keypoint detection framework explicitly optimized for metrological evaluation of heliostats. Two innovations are introduced in the framework: (1) a high-resolution P2 feature branch integrated into the network’s neck to recover fine-grained boundary cues; and (2) a physically-driven geometric consistency loss ($L_{\mathrm{geo}}$) based on perspective-compatible geometric consistency that promotes topological stability against extreme optical noise [13,29]. The proposed system achieves a processing throughput of 25.14 FPS, a rate compatible with the latency and frequency budget typically required for closed-loop control of a servo system.

The contributions of this paper are summarized as follows. First, we adapt a YOLOv8-Pose backbone [30] into a heliostat-corner detection network by adding a high-resolution P2 feature branch that preserves the boundary cues otherwise destroyed by deep down-sampling, and we visualize the per-level feature energy to substantiate this design. Second, we introduce a geometry-consistency loss term $L_{\mathrm{geo}}$ that combines a log edge-ratio penalty (an image-plane structural consistency term) with a diagonal-intersection penalty (a projective-invariant term), both compatible with the perspective imaging geometry of an off-axis heliostat. Third, ablation studies and application-oriented performance discussion of the framework are provided in the paper, especially the quantitative metrics related to field deployment tasks.

2. Related Work

2.1. Heliostat Calibration and Vision-Based Measurement

Heliostat calibration has long been dominated by Beam Characterization Systems (BCS) and their camera-augmented variants, which are accurate but intrinsically offline and intrusive (Section 1) [4,5,6,7]. Vision-based online alternatives have therefore been explored, including camera observations of the receiver flux pattern, fiducial-marker placement on the heliostat support structure, and direct image-based detection of the mirror facet. The latter family is closest in spirit to the present work: it does not require physical fiducials on the mirror, does not interrupt power generation, and can be sampled at video rate [10,11,12,13]. Recent learning-based instances of this family include a hybrid YOLOv5 detector combined with a Hough-Transform corner extractor for EPnP-based pose recovery [14] and differentiable-ray-tracing surface inverse-rendering for in-situ metrology [15,16]; compared with these detection-plus-classical-CV cascades, the present detector uses a single end-to-end, geometry-regularised keypoint regressor, removing the post-processing latency and the failure modes that the Hough stage introduces when the facet boundary is partially saturated. Open challenges in this family include robustness to specular glare, the lack of intra-facet texture [20,21], and the need to deliver geometrically consistent observations to a downstream controller.

2.2. Deep Learning-Based Keypoint Detection

Modern keypoint detection has been driven by two parallel directions: holistic top-down regressors and high-resolution heatmap predictors [23,24,25,26]. Single-stage detectors such as the YOLO-Pose family [27,30] attach a keypoint head to an object detector, allowing both localization and per-instance keypoint regression in one network pass; they are attractive for industrial deployment because they offer a favourable speed/accuracy trade-off on a single GPU. Heatmap-based pipelines instead predict per-keypoint spatial response maps and decode coordinates by spatial argmax, often with multi-resolution feature fusion [31,32]. Coordinate-classification methods such as SimCC reformulate localization as one-dimensional classification along each image axis [33], while RTMPose pushes the inference latency to the millisecond range on commodity hardware [28]. Both families are typically benchmarked on natural-scene human-pose datasets where targets exhibit rich, repeatable textures and where small misalignments between predicted keypoints are tolerable. Heliostat corner localization deviates from these assumptions: there is essentially no intra-facet texture, the four corners must form a topologically valid quadrilateral at every frame, and isolated coordinate jitter is not acceptable as it propagates directly into servo commands. Consequently, off-the-shelf keypoint detectors transfer imperfectly, motivating the addition of high-resolution branches and structural constraints described in Section 3.

2.3. Geometry-Constrained Keypoint Detection

A recurring strategy in industrial vision is to inject geometric priors into a learning-based pipeline so that predictions remain physically plausible under noise. Examples include pairwise distance and angle penalties, ratio-based shape losses, projective and homography-consistency objectives, and explicit reprojection terms tied to a known camera model [13,29]. The pertinent observation for the heliostat setting is that, under an off-axis pinhole camera viewing a tilted rectangular mirror, parallel-edge or right-angle constraints are violated by the perspective projection itself. Consequently, the only meaningful regularizers are quantities that are invariant under the perspective projection: cross-ratios, line-incidence relations, and intersections of corresponding lines. Our $L_{\mathrm{geo}}$ combines a log edge-length-ratio term with an offset term on the diagonal-intersection point relative to its ground-truth location. The diagonal-intersection term is a projective invariant (collinearity and the cross-ratio along each diagonal are preserved by perspective projection), whereas the edge-length-ratio term is best understood as image-plane structural consistency: it is not strictly invariant under perspective, but it is perspective-compatible in the sense that its target value is read directly from the ground-truth quadrilateral and therefore does not contradict the imaging geometry. Related ideas appear in scene-text quadrilateral detection, document-corner regression, and document dewarping, where a four-corner topology must be preserved against noisy cues [13,29].

3. Materials and Methods

3.1. Overall Pipeline

The proposed online tracking-error measurement system is a single-camera pipeline (Figure 1) that converts a continuous image stream from a fixed observation viewpoint into a time-resolved geometric error signal suitable for closed-loop servo feedback. At each time step, the camera captures an RGB frame containing one or several heliostat facets visible from the observation viewpoint. The frame is forwarded to the modified YOLOv8-Pose detector [30], which predicts, per visible facet, four ordered corner keypoints (top-left, top-right, bottom-left, bottom-right). These four 2D keypoints are then compared with the geometric configuration that the facet is commanded to have at the same instant, and the residual between observed and commanded geometry serves as the tracking-error signal that is forwarded to the heliostat controller [34].

This pipeline frames the deep network not as a stand-alone classifier but as a measurement instrument inserted into a control loop. Two design consequences follow. First, the output must be available at a frequency compatible with the servo control loop; for the prototype reported here the end-to-end throughput is 25.14 FPS (Section 4, Table 2), which is sufficient for the slow mechanical dynamics of a heliostat drive. Second, the per-frame output must be geometrically consistent: the four predicted corners must form a valid (non-self-intersecting, non-collapsed) quadrilateral that approximates the projected facet shape, otherwise the downstream controller would receive structurally invalid measurements and could execute large, abrupt commands.

3.2. Problem Formulation and Optical Challenges

The objective of this study is to perform high-speed and accurate localization of the four image-plane corners of a heliostat mirror from a monocular RGB image. This 2D measurement constitutes the foundational geometric observation required for the downstream closed-loop servo control of CSP systems. Treating the four corners as the primary observable is deliberate: a rectangular facet is fully determined, up to a rigid-body pose with respect to the camera, by the projected positions of its four corners, and four 2D–3D correspondences are sufficient for a downstream PnP-style pose estimator [13,35]. Compared with detecting the full facet contour, four-corner detection is also markedly more robust under partial occlusion because losing one corner still leaves enough geometric information to flag the frame as degenerate and skip it.

Under a standard pinhole camera model, a 3D point ${\mathbf{P}}_w={[X_w,Y_w,Z_w,1]}^{\top }$ residing on the heliostat surface is mapped to the 2D image coordinate $\mathbf{p}={[x,y,1]}^{\top }$ via the perspective projection matrix:

$$z_c\mathbf{p}=\mathbf{K}[\mathbf{R}\mid \mathbf{t}]{\mathbf{P}}_w$$

(1)

where $\mathbf{K}$ is the camera intrinsic matrix, $[\mathbf{R}\mid \mathbf{t}]$ denotes the extrinsic pose parameters relative to the mirror, and $z_c$ is the depth of the 3D point in the camera frame (the homogeneous-coordinate scale factor). The symbol $z_c$ is reserved for this geometric depth and is distinct from the scalar loss weights ${\lambda }_{\mathrm{geo}}$ and ${\lambda }_{\mathrm{inter}}$ introduced later. Because $z_c$ varies continuously across the tilted mirror plane, affine properties—such as the parallelism between opposite edges of the rectangular mirror—are violated in the perspective image plane. Any structural regularization applied to the detection network therefore cannot rely on simplistic parallel-line assumptions, and must instead invoke projective invariants such as cross-ratios and line incidences [13].

Given an input image tensor $\mathbf{X}\in {\mathbb{R}}^{H\times W\times 3}$, the detection framework predicts, per visible facet, a set of four corner keypoints $\widehat{\mathbf{Q}}={\left\{{\widehat{\mathbf{q}}}_k\right\}}_{k=0}^3$, where ${\widehat{\mathbf{q}}}_k=({\widehat{x}}_k,{\widehat{y}}_k)$ denotes the predicted image coordinate of the k-th corner. The corresponding ground-truth set annotated by domain experts is denoted by $\mathbf{Q}={\left\{{\mathbf{q}}_k\right\}}_{k=0}^3$. We adopt a fixed corner ordering—top-left (TL), top-right (TR), bottom-left (BL) and bottom-right (BR)—during both annotation and prediction, so that diagonal correspondences (TL↔BR and TR↔BL) are well defined and can be used inside the geometric loss without combinatorial ambiguity.

The YOLOv8-Pose architecture is adopted as the baseline detector due to its highly efficient Cross Stage Partial (CSP) backbone, which is well-suited for industrial edge-computing deployments [30,36]. However, its default feature pyramid (relying heavily on P3 to P6 layers) entails excessive spatial downsampling, which over-smooths the high-frequency geometric discontinuities essential for precision metrology. Pixel-accurate corner localization in this regime is fundamentally limited by the receptive-field stride at the level from which the keypoint head reads its features; a pose head attached to P3–P6 only sees an effective stride of at least 8 pixels relative to the input resolution, which is insufficient when the facet edge spans only a few tens of pixels in the captured image.

3.3. Resolution-Preserving Feature Fusion and Energy Analysis

To address the critical attenuation of micro-structures at mirror boundaries, a high-resolution P2 feature level is integrated into the network’s neck (Figure 2). In the standard YOLOv8-Pose-P6 stack, the keypoint head receives features from levels P3 through P6, whose spatial strides relative to the input range from 8 to 64. We extend the multi-scale fusion path so that the keypoint head additionally reads a P2 feature with an effective stride of 4, which approximately doubles the spatial precision available at the head input without altering the backbone’s semantic depth or the pretrained weight initialization for the deeper levels [31,32]. To empirically validate this architectural intervention, we extract and quantify the feature response intensity across all pyramid levels. For a given feature tensor ${\mathbf{F}}^{\left(l\right)}\in {\mathbb{R}}^{C_l\times H_l\times W_l}$ at pyramid level l, the channel-wise activation is aggregated using root-mean-square (RMS) energy:

$$E^{\left(l\right)}(h,w)=\sqrt{{\displaystyle \frac{1}{C_l}}\sum _{c=1}^{C_l}{\left({\mathbf{F}}_c^{\left(l\right)}(h,w)\right)}^2}$$

(2)

Following a robust quantile normalization process to suppress background sensor noise, the resulting pseudo-color energy maps (Figure 2a–e) reveal a critical phenomenon: the P2 feature exhibits profound spatial convergence, tightly anchoring to the mirror’s physical boundaries. In contrast, deeper semantic levels (P5, P6) respond broadly to the global scene context, losing the localized precision needed for accurate few-pixel-level corner measurement. From the perspective of measurement, this is the expected behaviour: deeper levels integrate large receptive fields that are useful for object presence but have already discarded the boundary location information by spatial pooling, whereas P2 retains short-range gradient cues that are co-located with the actual facet corners. Adding P2 into the head input therefore restores the high-frequency information that subsequent loss minimization can exploit; the deeper levels continue to contribute by gating the per-anchor visibility classification and confidence estimation.

3.4. Geometry Consistency Regularization Under Perspective Projection

The standard keypoint regression head in generic pose estimation architectures optimizes spatial coordinates by minimizing a visibility-masked distance metric:

$$L_{\mathrm{kpt}}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{k=0}^3}{v}_k·\mathcal{D}({\widehat{\mathbf{q}}}_k,{\mathbf{q}}_k),$$

(3)

where $v_k\in \{0,1\}$ indicates keypoint visibility, and $\mathcal{D}$ typically represents the Object Keypoint Similarity (OKS) metric [37]. However, this point-wise independent optimization ignores the holistic topological constraints of the quadrilateral. Under severe optical noise or partial glare, independent regression frequently yields topologically implausible concave shapes or severe twists. From a measurement-engineering point of view this is a critical defect: even if the average pixel error is small, a single mispositioned corner that crosses one of its neighbours converts the predicted facet into a self-intersecting quadrilateral, and the geometric error estimator downstream cannot interpret such a frame.

A naive remedy would be to penalize departures from a rigid rectangular shape, e.g. by enforcing parallel opposite edges or right angles. As discussed in Section 3, a rectangular facet projects to a non-rectangular quadrilateral under an off-axis pinhole projection, so such a penalty would push the network toward physically wrong predictions. The remedy must therefore be expressed in quantities that are invariant under the perspective projection. We choose two such invariants and combine them.

To explicitly enforce topological stability without violating perspective geometry, a novel geometry consistency loss ($L_{\mathrm{geo}}$) is formulated using perspective-compatible geometric quantities, i.e. quantities whose target values are read from the ground-truth projection rather than from a planar-rectangle prior. The first component evaluates the scale-invariant geometric edge-ratio ($L_{\mathrm{ratio}}$):

$$L_{\mathrm{ratio}}=\left|log{\displaystyle \frac{d({\widehat{\mathbf{q}}}_0,{\widehat{\mathbf{q}}}_1)}{d({\widehat{\mathbf{q}}}_2,{\widehat{\mathbf{q}}}_3)}}-log{\displaystyle \frac{d({\mathbf{q}}_0,{\mathbf{q}}_1)}{d({\mathbf{q}}_2,{\mathbf{q}}_3)}}\right|+\left|log{\displaystyle \frac{d({\widehat{\mathbf{q}}}_0,{\widehat{\mathbf{q}}}_2)}{d({\widehat{\mathbf{q}}}_1,{\widehat{\mathbf{q}}}_3)}}-log{\displaystyle \frac{d({\mathbf{q}}_0,{\mathbf{q}}_2)}{d({\mathbf{q}}_1,{\mathbf{q}}_3)}}\right|$$

(4)

where $d(\mathbf{a},\mathbf{b})={\parallel \mathbf{b}-\mathbf{a}\parallel }_2$ denotes the Euclidean distance. The use of the logarithm has two pragmatic effects. First, it converts a multiplicative ratio mismatch into an additive scale-symmetric penalty, so that a 10% error on a small edge contributes the same loss magnitude as a 10% error on a large edge. Second, it removes the numerical asymmetry that a raw ratio would introduce when an edge is short. To prevent log singularities for very small edges, the implementation clamps both numerator and denominator to a small positive lower bound before taking the logarithm.

The second component enforces diagonal-intersection consistency ($L_{\mathrm{inter}}$). Let $L_1=\parallel {\mathbf{q}}_3-{\mathbf{q}}_0\parallel$ and $L_2=\parallel {\mathbf{q}}_2-{\mathbf{q}}_1\parallel$ denote the two diagonal lengths, and let

$${\widehat{\mathbf{d}}}_1={\displaystyle \frac{{\mathbf{q}}_3-{\mathbf{q}}_0}{L_1}},{\widehat{\mathbf{d}}}_2={\displaystyle \frac{{\mathbf{q}}_2-{\mathbf{q}}_1}{L_2}}$$

(5)

denote the corresponding unit-norm direction vectors ($\parallel {\widehat{\mathbf{d}}}_1\parallel =\parallel {\widehat{\mathbf{d}}}_2\parallel =1$). The two ground-truth diagonals are then parameterised by arc length as ${\ell }_1\left(s\right)={\mathbf{q}}_0+s{\widehat{\mathbf{d}}}_1$ and ${\ell }_2\left(t\right)={\mathbf{q}}_1+t{\widehat{\mathbf{d}}}_2$, where $s\in [0,L_1]$ and $t\in [0,L_2]$ are measured directly in image-plane pixels. Their intersection $\mathbf{I}$ must satisfy

$${\mathbf{q}}_0+s{\widehat{\mathbf{d}}}_1={\mathbf{q}}_1+t{\widehat{\mathbf{d}}}_2.$$

(6)

Taking the 2D cross product of both sides with ${\widehat{\mathbf{d}}}_2$ eliminates t and yields the closed-form solution

$$s={\displaystyle \frac{det\left({\mathbf{q}}_1-{\mathbf{q}}_0,{\widehat{\mathbf{d}}}_2\right)}{det\left({\widehat{\mathbf{d}}}_1,{\widehat{\mathbf{d}}}_2\right)+\epsilon }}={\displaystyle \frac{det\left({\mathbf{q}}_1-{\mathbf{q}}_0,{\widehat{\mathbf{d}}}_2\right)}{sin\theta +\epsilon }},$$

(7)

where $\theta$ is the angle between ${\widehat{\mathbf{d}}}_1$ and ${\widehat{\mathbf{d}}}_2$, and $\epsilon$ is a small positive constant that prevents division by zero when the two diagonals become nearly collinear ($\theta \to 0$). Because ${\widehat{\mathbf{d}}}_1$ and ${\widehat{\mathbf{d}}}_2$ are unit-norm, $det({\widehat{\mathbf{d}}}_1,{\widehat{\mathbf{d}}}_2)$ is exactly $sin\theta$, so $\epsilon$ acquires the clear physical meaning of a lower bound on the admissible angular separation of the two diagonals rather than the role of a purely numerical safeguard.

The ground-truth intersection follows from Eq. (7) in one line as $\mathbf{I}={\mathbf{q}}_0+s{\widehat{\mathbf{d}}}_1$. The predicted intersection $\widehat{\mathbf{I}}$ is computed by the same closed form applied to the predicted corner set $\widehat{\mathbf{Q}}$ – i.e. by substituting ${\widehat{\mathbf{q}}}_k$ for ${\mathbf{q}}_k$ throughout Eqs. (5)–(7) – and the Euclidean distance between the two intersection points serves as the structural penalty

$$L_{\mathrm{inter}}=\parallel \widehat{\mathbf{I}}-\mathbf{I}{\parallel }_2.$$

(8)

Both intersection points are expressed in normalised image coordinates (pixel coordinates divided by the input image size) before this distance is evaluated, so $L_{\mathrm{inter}}$ is dimensionless and is directly comparable in magnitude with $L_{\mathrm{ratio}}$.

Geometrically, $L_{\mathrm{inter}}$ penalises configurations in which the two predicted diagonals fail to meet at the same projected point as the ground-truth diagonals. Because the cross-ratio along each diagonal is preserved by the perspective projection [13], requiring the two predicted diagonals to intersect at the correct projected point is equivalent to requiring the projected facet to have the correct internal cross-ratio structure – without imposing parallelism, right angles, or rectangularity in the image plane. $L_{\mathrm{inter}}$ is therefore a perspective-invariant topological regulariser, complementary to the image-plane edge-length-ratio regulariser $L_{\mathrm{ratio}}$ of Eq. (4), which constrains scale consistency along the facet sides rather than the projective incidence of the diagonals.

The complete training objective combines this topological regulariser with the standard YOLO bounding-box, classification and keypoint terms:

$$L_{\mathrm{total}}=L_{\mathrm{YOLO}}+{\lambda }_{\mathrm{geo}}\left(L_{\mathrm{ratio}}+{\lambda }_{\mathrm{inter}}{L}_{\mathrm{inter}}\right).$$

(9)

In our reference implementation the geometric weight ${\lambda }_{\mathrm{geo}}$ is held at a small value so that $L_{\mathrm{geo}}=L_{\mathrm{ratio}}+{\lambda }_{\mathrm{inter}}{L}_{\mathrm{inter}}$ behaves as a stabiliser rather than as a primary objective: it shapes the search space to exclude topologically invalid quadrilaterals while leaving the bulk of the gradient signal to the standard keypoint and bounding-box terms. The loss is clipped to a finite range so that the gradient remains bounded when degenerate, near-parallel-diagonal configurations briefly appear early in training – that is, when $sin\theta \to 0$ in Eq. (7) and the closed-form s would otherwise dominate the optimisation step.

4. Results

4.1. Dataset and Implementation Protocols

The proposed framework was evaluated on an industrial heliostat image dataset captured under outdoor conditions. The dataset comprises 300 high-resolution images of operating heliostats captured at various times of day. The four corner points of each facet of the heliostat in the image are labeled in a fixed TL/TR/BL/BR order, together with a visibility flag, so that the geometric loss in Section 3 can be computed only over fully visible facets. The dataset is treated as a self-contained engineering benchmark rather than a public competition task. It serves as a testbed for evaluating deep-learning methods for the detection and control of heliostats in a CSP plant [11].

Each image contains a single heliostat composed of an $8\times 8$ grid of sub-mirror facets, yielding $64\times 4=256$ corner annotations per fully visible image. Corners that fall inside saturated (over-exposed) or shadowed (under-exposed) regions are intentionally not annotated and are excluded from the loss by the per-keypoint visibility flag $v_k$ (see Eq. (3)); this convention propagates the optical-degradation statistics of the deployment scene into the supervision signal rather than into the labelling protocol. The 300 images are split at the image level into train, validation and test partitions in an $8:1:1$ ratio (240 / 30 / 30 images), corresponding to up to $61,440/7,680/7,680$ annotated corners under the fully visible upper bound.

The model was trained on an NVIDIA RTX 4090D GPU using the AdamW optimizer [38] for 300 epochs, with input tensors resized to $1280\times 1280$ to fully leverage the high-resolution P2 capabilities. Mixed-precision (AMP) computation was enabled to reduce GPU memory pressure, and the keypoint-loss gain was raised relative to the default value to align the optimization budget with the precision-driven objective of this work. Geometric data augmentation was deliberately conservative (small rotation, small translation, small scale, no synthetic perspective warp) so that the model is exposed to operationally realistic facet shapes rather than to aggressively warped quadrilaterals; mosaic and a small mixup factor were retained so that anchor-based pose detection still benefits from multi-instance composition during training. The optimizer used an initial learning rate of $5\times {10}^{-4}$ with a cosine schedule, and the geometry weight was set to ${\lambda }_{\mathrm{geo}}=0.05$; all other settings followed the YOLOv8-Pose defaults.

For evaluation we use four metrics. mAP@0.5 reports the average precision under a relatively loose IoU threshold and quantifies whether the facet is found at all. mAP@0.5:0.95 averages precision over a sweep of stricter IoU thresholds and is therefore the most sensitive to localization accuracy of the bounding box that surrounds the four keypoints. AR@0.5 reports the average recall under the loose IoU threshold and complements precision by quantifying how often a present facet is missed. End-point error (EPE) is the mean Euclidean distance between predicted and ground-truth keypoint coordinates over matched detections. End-to-end frames per second (FPS) is reported as the wall-clock throughput including pre-processing, inference and post-processing.

4.2. Quantitative Analysis

To strictly isolate the contributions of the proposed architectural and mathematical modifications, a comprehensive ablation study was conducted (Table 1).

Table 1 shows that the baseline model can already detect the facets at a high rate. Its limitations lie on the localization precision of corner points. Adding $L_{\mathrm{geo}}$ on top of an unmodified backbone (Only_Geo) does not by itself reduce EPE, because the underlying coordinates are still produced from low-resolution features that fundamentally cannot resolve the boundary at sub-pixel level. Adding the P2 branch (Only_P2) alone restores the high-frequency information and reduces EPE from 2.58 to 1.89 px, which is consistent with a stride-driven precision floor: halving the head’s effective stride approximately halves the unavoidable rounding error contribution, and the remaining 1.89 px is dominated by genuine optical ambiguity (glare-saturated edges, motion blur). Combining both interventions retains the EPE level of Only_P2 and additionally tightens the strict mAP@0.5:0.95 score, which is the metric most sensitive to localization quality. We interpret this as evidence that $L_{\mathrm{geo}}$ removes residual outlier configurations that would otherwise lower the strict-IoU precision but would barely affect the average pixel error—in other words, $L_{\mathrm{geo}}$ matters most where it is hardest to detect statistically, namely in the high-IoU regime that controls servo-loop usability.

4.3. Comparison with a Generic Pose Estimator

To benchmark the proposed framework against a generic baseline, the widely used open-source framework MMPose is evaluated on the same heliostat dataset under its default top-down configuration. Under identical conditions it reaches only mAP@0.5 $=0.3267$ and AR@0.5 $=0.3207$, far below the proposed framework (mAP@0.5 $=0.9927$ in Table 1); its EPE of $2.0139$ px is computed only over the few facets it manages to detect and is therefore not directly comparable with the dense matches of our detector. This large detection gap shows that generic top-down detectors, designed for human-scale articulated poses with rich intrinsic texture [25,26,39], degrade severely on the texture-poor, highly specular heliostat facets, confirming that the task-specific design adopted here is necessary rather than incremental.

4.4. Qualitative Analysis

Qualitative comparisons across different ablation configurations are presented in Figure 3.

As shown in Figure 3, the Baseline model produces structurally distorted or collapsed quadrilaterals when the corner boundaries are blurred for various reasons. The proposed framework greatly improved the geometric resilience of the model by acquiring a topologically valid quadrilateral structure with the majority of captured samples. Two qualitative regularities are worth noting. First, failure modes of the Baseline model cluster on samples where one corner sits in a saturated glare patch; the loss of local gradient information propagates almost exclusively to that corner, which then collapses toward an internal point and turns the predicted quadrilateral into a degenerate, near-triangular shape. Second, the residual failure modes of Only_P2 cluster on samples where two adjacent corners are simultaneously affected, in which case the unconstrained pairwise regression produces visually small but topologically incorrect twists; once $L_{\mathrm{geo}}$ is added, the diagonal-intersection penalty rules these twists out because they would force the two predicted diagonals to meet at the wrong projected point. The results in Figure 3 are consistent with the design intent of each component: P2 supplies localization sharpness, $L_{\mathrm{geo}}$ supplies topological stability.

5. Discussion

5.1. Error Justification for Vision-Based Feedback Control

Although the proposed framework reduces the mean EPE to 1.89 pixels, this absolute static error may appear numerically modest when juxtaposed with tightly controlled indoor metrology. However, this residual error is a deliberate and theoretically expected consequence of the operational environment and our architectural trade-offs. Firstly, the intense specular glare inherent to functioning heliostats fundamentally saturates local pixel gradients, rendering mathematically perfect sub-pixel corner localization highly improbable from a single monocular snapshot. Secondly, to strictly adhere to the high-frequency latency constraints required for active control loops, computationally expensive iterative sub-pixel refinement modules—commonly utilized in static photogrammetry—were intentionally excluded from our pipeline.

Rather than viewing this 1.89-pixel deviation as a metrological limitation, it must be contextualized within its primary prospective application: dynamic closed-loop servo control. As depicted in Figure 4a (error curves) and Figure 4b (box plots), our proposed geometric regularization actively suppresses catastrophic outliers (the extreme upper tail of the error distribution). Suppressing these large structural deviations is expected to reduce the abrupt coordinate jumps that a downstream controller would otherwise have to reject; whether this translates into smoother actuator behaviour can only be confirmed once the loop is closed, which is beyond the scope of the present open-loop evaluation.

A sun-tracking heliostat moves at only about ${15}^{\circ }$ per hour, so the sampling-rate requirement is modest, and the measured throughput of 25.14 FPS (Table 2) leaves ample margin for closed-loop control. Moreover, the bounded error tail without self-intersecting quadrilaterals that $L_{\mathrm{geo}}$ is designed to produce is well suited to short-window temporal filtering, so closed-loop control of the integrated detector + filter + actuator system becomes feasible in future deployment [34,40].

Table 2. Runtime profile and latency breakdown of the proposed measurement framework.

Performance Metric	Measured Value
End-to-end FPS (wall-clock) ↑	25.14
Inference-only FPS ↑	27.51
Preprocess time (ms) ↓	1.996
Inference time (ms) ↓	36.349
Postprocess time (ms) ↓	0.784

Table 2. Runtime profile and latency breakdown of the proposed measurement framework.

Performance Metric	Measured Value
End-to-end FPS (wall-clock) ↑	25.14
Inference-only FPS ↑	27.51
Preprocess time (ms) ↓	1.996
Inference time (ms) ↓	36.349
Postprocess time (ms) ↓	0.784

5.2. Deployment Considerations

Translating the proposed vision-based measuring system into a deployable field system raises a number of issues that may affect its performance. First, camera installation stability is essential because the absolute measurement budget of the system is the sum of the detector’s pixel error and any residual misalignment between the camera’s optical axis and the heliostat field reference frame. A rigid mount, periodic auto-recalibration against fixed scene references, and a thermal shield on the camera enclosure are practical countermeasures; these are standard in industrial machine-vision deployments and can be reused without modification [12,13].

Second, although the training set used in this model covers strong-glare conditions, the cameras of the vision-based measuring system are expected to have automatic exposure control functions as well as high-resolution imaging sensors. Sensor-level remediation tends to be more effective than pure post-processing because once a corner falls inside a saturated region, no amount of downstream filtering can recover the lost gradient [18,19].

Third, given the output rate of 25.14 FPS, the pose estimation accuracy can be further improved by integrating filters into the control loop [34,40]. Temporal filters can suppress the per-frame jitter over a short time window and maintain reasonable estimates when momentary detection failures occur.

Fourth, edge-computing deployment is preferable to a centralized inference server. An embedded GPU or NPU at the edge is sufficient to maintain the reported FPS and avoids network-induced jitter on the control loop [36]. The predicted four-corner coordinates can be combined with a known 3D facet model and a calibrated camera intrinsic matrix to produce a 6-DoF pose via PnP, or combined with a calibrated reference homography to generate a 2D tracking-error vector directly for the angle control of heliostats [13,35].

6. Conclusions

This study proposes a high-speed, geometry-aware visual measurement framework for in-situ detection and calibration of heliostats in concentrated solar power plants. By adapting the YOLOv8-Pose architecture with a high-resolution P2 branch and a perspective-invariant geometry consistency loss, the framework can recover fine-grained boundaries while ensuring rigid topological stability under extreme conditions. The proposed framework achieved an end-to-end processing speed of 25.14 FPS and a mean end-point error of 1.89 px with the real-world dataset. Although the experiments were carried out at a small-scale CSP station, the methodology is applicable to large-scale commercial CSP plants, provided that a suitable dataset is constructed.

The current framework focuses on localization of 2D image keypoints instead of estimating the heliostat pose. The data-driven model outputs the image coordinates of the four corners of each facet, while the estimation of heliostat tracking pose is left to a downstream module that uses the predicted keypoints with EPnP-type algorithms [14]. This separation allows the keypoint detector to be easily integrated with different pose estimation algorithms [13,35].

The reported throughput (25.14 FPS) is based on the specified hardware. A production deployment will most likely run on an embedded edge accelerator next to the camera, where memory and power constraints differ; the network may need quantization, channel pruning, or knowledge-distillation-driven compression to retain real-time behaviour on the target platform [36]. Such compression is feasible because the geometric constraints are imposed at training time only and do not enlarge the inference graph.

Overall, the proposed detector provides fast and geometrically reliable 2D observations that can serve as a practical front-end for closed-loop heliostat calibration in CSP plants; closing the detector–filter–actuator loop on a physical drive remains the primary direction of future work.