Robust Low-Overlap Point Cloud Registration via Edge-Guided Features and Quaternion Averaging

1. Introduction

The convenient acquisition of point cloud data has led to its widespread application in image processing and object reconstruction [1]. Point cloud registration (PCR) is a fundamental problem in this domain, aiming to estimate the six-degree-of-freedom relative pose between two point clouds [2] by determining the optimal rigid transformation matrix. Accurate estimation of this transformation is critical for downstream tasks such as 3D reconstruction and localization.

Classical registration methods, such as the Iterative Closest Point(ICP) algorithm proposed by Besl and McKay [3], and the Normal Distributions Transform (NDT) introduced by Magnusson and Martin [4], have demonstrated strong performance under high-overlap conditions. However, these methods often converge to local minima when dealing with low-overlap data. To address this, Rusu et al. incorporated the Random Sample Consensus (RANSAC) algorithm [5] to provide a coarse alignment, which is subsequently refined using ICP [6]. While this pipeline helps avoid local traps, it remains sensitive to outliers and performs poorly in sparse correspondence scenarios [7].

Recent learning-based methods, such as PRNet by Y. Wang et al. [8] and the improved HOUV algorithm by Yan Y. et al. [9], have shown promising performance by leveraging large-scale feature matching in overlapping regions. However, their reliance on sufficient overlap makes them less effective in low-overlap cases, where feature sparsity severely hampers registration accuracy [10].

Several studies have explored alternative strategies to address this challenge. Zhang et al. introduced a method combining shape and texture features [11], while Sun et al. proposed a registration framework incorporating Intrinsic Shape Signature (ISS) keypoints as edge features [12]. These approaches improve robustness under low-overlap conditions. Nonetheless, traditional edge detectors such as ISS and Harris-3D depend on global geometric statistics [13], making them computationally expensive and less scalable for dense point clouds.

Normal Vector Extrema Change (NVEC) has been widely adopted in differential geometry to describe sharp variations in surface curvature, and edge extraction methods based on this principle have been widely applied in 3D printing [14] and boundary detection in medical imaging [15]. Meanwhile, quaternions, as compact mathematical tools for representing rigid body transformations in three-dimensional space, have been extensively used in point cloud registration [16], especially in multi-view alignment [17] and spacecraft attitude estimation [18].

Motivated by these insights, this study presents an edge-guided and quaternion-averaged fusion algorithm to enhance registration robustness under low-overlap conditions, by leveraging the complementary strengths of NVEC-based edge detection and quaternion averaging.

2. Research Methods

In this study, two input point clouds are defined as the source and target point clouds. First, edge features are extracted from the raw point cloud using the Normal Vector Extrema Change method. Subsequently, Fast Point Feature Histogram are computed for the extracted edge points, and feature correspondences are established through KDTree-based nearest neighbor search. An initial pose estimation is then performed using these correspondences, with an emphasis on preserving local geometric consistency. Finally, a point-to-plane ICP algorithm is applied for fine alignment, completing the precise registration process. The complete workflow is illustrated in Figure 1.

2.1. Edge Extraction Based on Normal Vector Extrema Change

Based on the definition of geometric variation, this study introduces a method for edge feature extraction grounded in Normal Vector Extrema Change (NVEC). By computing the maximum angular deviation between normal vectors in the local neighborhood of each point in the cloud, the method determines whether the point lies on a geometric edge. This approach effectively reduces computational complexity while improving the efficiency of edge detection.

Let the input point cloud be denoted as ${\left\{p_i\right\}}_{i=1}^N$. A voxel downsampling process is applied to normalize point density and reduce computational redundancy. The 3D space is partitioned into cubic voxels, and each voxel is represented by the geometric centroid of the points it contains, resulting in a sparse point cloud $P_d$. The voxel size is estimated as follows:

$$L=\sqrt[3]{\frac{3}{\pi \delta N}}·∆l$$

(1)

where $\delta$denotes the sampling ratio, $N$is the number of original points, and $\Delta l$is the maximum edge length of the point cloud’s bounding box. This adaptive strategy ensures the preservation of geometric features across varying scales.

For each point $p_i\in P_d$, a local neighborhood is constructed, and its normal vector is estimated using Principal Component Analysis (PCA). A covariance matrix is computed within a spherical region, and the normal vector $\overrightarrow{{\mathrm{n}}_{\mathrm{i}}}$is defined as the eigenvector corresponding to the smallest eigenvalue. To avoid directional ambiguity, all normals are consistently oriented toward the centroid of the point cloud.

For each point $p_i$, its k-nearest neighbors in Euclidean space form the neighborhood set ${\left\{p_j\right\}}_{j=1}^k$, denoted as $N_k\left(i\right)$. A radius constraint is applied:

$$\Vert p_i-p_j\Vert <r$$

(2)

where $k$is typically set between 10 and 30 to ensure local stability, and $r=3L$is chosen to constrain outlier inclusion and preserve local geometric consistency.

Let $\overrightarrow{n_i}$and $\overrightarrow{n_j}$denote the normal vectors of point $p_i$and its neighbor $p_j$, respectively. The angle between them is computed as:

$${\theta }_{ij}=\arccos \left({\displaystyle \frac{\overrightarrow{n_i}\cdot \overrightarrow{n_j}}{\Vert \overrightarrow{n_i}\Vert \Vert \overrightarrow{n_j}\Vert }}\right)$$

(3)

Since all normals are normalized, this simplifies to:

$${\theta }_{ij}=\arccos \left(\overrightarrow{n_i}\cdot \overrightarrow{n_j}\right)$$

(4)

The maximum angular deviation of the normal vector is defined as the edge response of point $p_i$:

$$E_i=\underset{j\in N_k\left(i\right)}{\max }{\theta }_{ij}$$

(5)

If $E_i$approaches zero, the point lies on a smooth surface; conversely, a large value indicates a geometric edge.

To improve robustness, the variance of angular deviations is also used as a secondary criterion:

$${\sigma }_i^2={\displaystyle \frac{1}{k}}{{\displaystyle \sum _{j=1}^k\left({\theta }_{ij}-\overline{{\theta }_l}\right)}}^2$$

(6)

According to empirical engineering experience, the angular threshold ${\theta }_{thresh}$ is set to 12°, and the variance threshold ${\sigma }_{thresh}$ is set to 8°. A point is classified as an edge point if:

$$E_i>{\theta }_{thresh}\mathrm{and}{\sigma }_i>{\sigma }_{thresh}$$

(7)

To further suppress noise and outliers, an edge response range constraint is imposed:

$${\theta }_{high}>E_i>{\theta }_{low}$$

(7)

where ${\theta }_{\mathrm{high}}$ is set to 45°, and ${\theta }_{\mathrm{low}}$ is set to 8°. These thresholds help eliminate spurious boundaries caused by measurement noise.

2.2. FPFH Descriptors and Matched Point Pairs

To construct local descriptors for edge points, the Fast Point Feature Histogram (FPFH) method is employed due to its robustness in capturing neighborhood geometry while maintaining invariance to rigid transformations. For each edge point $p_i$, a spherical neighborhood with radius $r$is defined to balance local geometric stability and descriptive capability.

Let the set of edge descriptors for the source point cloud $P_s$be denoted as $\left\{p_i\right\}$, and those for the target point cloud $Q_t$as $\left\{q_j\right\}$. For each descriptor in the source set, its two nearest neighbors in the target descriptor space—denoted as ${p_i}^{\prime }$and ${q_i}^{\prime }$—are identified, with corresponding distances $d_1$and $d_2$, respectively.

To suppress ambiguous matches, a ratio test is applied. A match is accepted only if:

$${\displaystyle \frac{d_1}{d_2}}<\rho$$

(8)

where $\rho$is the ratio threshold, set to 0.8 in this study. This filtering strategy helps eliminate incorrect correspondences arising from repetitive structures or similar local patterns.

To further enhance the robustness of the matching process, a mutual consistency check is performed, inspired by the concept of inverse consistency. Specifically, if a point $p_i$in the source point cloud is matched to a point $q_j$in the target, the reverse correspondence—from $q_j$back to $p_i$—must also hold:

$$p_i\to q_j,q_j\to p_i$$

(9)

This constraint ensures one-to-one matching and eliminates inconsistent or one-to-many correspondences, which are common in densely sampled point clouds. As a result, the geometric consistency of the final correspondences is significantly improved.

The final set of reliable matched point pairs, filtered by both the ratio test and mutual consistency check, is denoted as $\left\{\left(p_i,q_i\right)\right\}$, represented as the set $M$.

2.3. Local Rigid Transformation Estimation and Quaternion Averaging

For the constructed set of matched point pairs, density-based spatial clustering is performed on the source point cloud $P$ using the DBSCAN algorithm. This method identifies core points whose local neighborhoods contain a sufficient number of nearby points and groups them into clusters $C_k$. Points not assigned to any cluster are discarded.

This strategy naturally filters out outliers introduced during descriptor matching. For each point in the resulting valid clusters $C_k$, its corresponding point in the matches pair set $M$is retrieved to form a new set of valid correspondences $\left\{\left(p_{ik},q_{ik}\right)\right\}$, collectively denoted as $M_k$. These pairs form the basis for subsequent rigid transformation estimation.

For each matched point pair set $M_k$, the rigid transformation is estimated using Singular Value Decomposition (SVD). The centroids of the source and target point sets are first computed:

$$\overline{p}={\displaystyle \frac{1}{N_k}}{\displaystyle \sum _{i=1}^{N_k}{p}_i},\overline{q}={\displaystyle \frac{1}{N_k}}{\displaystyle \sum _{i=1}^{N_k}{q}_i}$$

(10)

The centered covariance matrix is then constructed:

$$H_k={\displaystyle \sum _{i=1}^{N_k}\left(q_i-\overline{q}\right)}{\left(p_i-\overline{p}\right)}^T$$

(11)

Applying SVD to

, we obtain:

$$H_k=U_k{\sum }_k{V}_k^T$$

(12)

The resulting rotation matrix $R_k$ and translation vector $t_k$ are given by:

$$R_k=V_k{U}_k^T,t_k=\overline{q}-R_k\overline{p}$$

(13)

This transformation represents the estimated rigid alignment for the selected cluster.

Since rigid transformations lie in the Special Euclidean group $SE\left(3\right)$, direct averaging in Euclidean space is not valid. To obtain a globally consistent estimate, the quaternion representation of each rotation matrix $R_k\in SO\left(3\right)$is computed and denoted as $q_k\in {ℝ}^4$. These quaternions are then averaged to obtain a representative rotation:

$$\overline{q}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{i=1}^K{q}_i}$$

(14)

The averaged quaternion $\overline{q}$is converted back to a rotation matrix, denoted as $R_{avg}$. The corresponding translation vector is computed as:

$$t_{avg}=\overline{q}-R_{avg}\cdot \overline{p}$$

(15)

Here, $\overline{q}$and $\overline{p}$represent the centroids of the source and target sets, respectively. This estimated transformation pair $\left(R_{avg},t_{avg}\right)$serves as the initial global pose for subsequent fine registration.

2.4. Point-to-Plane ICP Fine Registration

The traditional ICP algorithm estimates the optimal rigid transformation that aligns the source point cloud to the target by minimizing the Euclidean distance between corresponding points. In contrast, the point-to-plane ICP variant minimizes the sum of squared distances between transformed source points and the tangent planes of their corresponding points in the target point cloud. Given an initial pose estimate, the objective function is formulated as:

$$\underset{R,t}{\min }{\displaystyle \sum _{i=1}^N{\left\{n_i^T\cdot \left[q_i-\left(R{p}_i+t\right)\right]\right\}}^2}$$

(16)

where $p_i$ denotes the i-th point in the source point cloud, $q_i$ is its corresponding nearest neighbor in the target $p_i$, $n_i$ is the unit normal vector at $q_i$, $R$and $t$are the rotation matrix and translation vector to be estimated.

During each iteration, the orthogonal projection residual is computed as the point-to-plane distance:

$$r_i=n_i^T\cdot \left[q_i-\left(R{p}_i+t\right)\right]$$

(17)

This residual measures how far the transformed point $p_i$deviates from the tangent plane at $q_i$.

The surface normals $n_i$are pre-computed on the target point cloud $Q$using local neighborhood PCA. For each source point $p_i$its nearest neighbor $q_i$in the target is identified, and the residuals are computed using the corresponding surface normals. Based on the least-squares principle, the incremental rigid transformation is estimated. The transformation is iteratively refined until convergence is achieved or the maximum number of iterations is reached. The final rigid transformation is obtained by composing the converged incremental transformation with the initial pose.

3. Experiment

3.1. Data Set

In this study, the Stanford 3D Scanning Repository was utilized as the experimental dataset [19]. As one of the earliest and most widely used publicly available 3D scanned datasets in the fields of computer graphics and computational geometry, it was collected and released by the Stanford Computer Graphics Laboratory. This dataset has been widely adopted for evaluating tasks such as point cloud registration, 3D reconstruction, surface resampling, and normal estimation. The point clouds were acquired using a Cyberware 3030 MS 3D laser scanner through segmented 360-degree scanning. Each scan frame contains approximately 35,000 to 50,000 points.

3.2. Implementation Detail

The computer configuration used in the experiment is an Intel(R) I9-13900K CPU (Intel, Santa Clara, CA, USA), an NVIDIA GeForce GTX4060 graphics card (NVIDIA, Santa Clara, CA, USA), with 8 GB of video memory. In this experiment, three datasets—Bunny, Dragon, and Happy Buddha—were selected, with multiple viewpoint point clouds from each dataset used for testing. Parameters not explicitly mentioned in the code were set to their default values.

3.3. Accuracy Evaluation Metrics

For edge extraction evaluation, precision and recall are adopted as error metrics. They are computed as:

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(18)

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(19)

where $TP$ denotes the number of true positives, i.e., correctly identified edge points; $FP$ represents false positives, i.e., non-edge points incorrectly classified as edge points; and $FN$ refers to false negatives, i.e., actual edge points that were not correctly identified. Precision reflects the discriminative capability of the edge detector, while recall reflects the completeness of edge detection.

For registration accuracy, the Root Mean Square Error (RMSE) is used as a quantitative metric. Given a set of point correspondences $\left\{\left(p_i,q_i\right)\right\}$, where

and $q_i$ are corresponding points from the transformed source and target point clouds respectively, the RMSE is calculated as follows:

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{\mathrm{N}}}{\displaystyle \sum _{i=1}^N{\Vert R{p}_i+t-q_i\Vert }^2}}$$

(20)

where $N$ is the number of matched point pairs, $R$ and $t$ are the estimated rotation matrix and translation vector. RMSE measures the average geometric alignment error of the registered point clouds, serving as an indicator of overall registration accuracy.

4. Discussion

4.1. Edge Extraction Validation

Edge extraction experiments were conducted on the bunny000 model from the Bunny dataset. The result of ISS-based edge extraction is shown in Figure 2(a), while the result obtained using the proposed NVEC method is presented in Figure 2(b).

As illustrated in Figure 2, the NVEC method more effectively captures structurally meaningful edge points, especially in regions with complex geometric variation. It achieves this without discarding essential boundary details. In contrast, the ISS method tends to miss critical geometric features and often produces sparse and incomplete edge representations due to its reliance on global saliency measures.

Figure 3 further compares the performance of ISS, Harris-3D, and NVEC in terms of precision and recall. The results show that as recall increases, the precision of the NVEC method remains consistently high, outperforming the other two methods. This indicates that NVEC achieves a better balance between accuracy and coverage in edge extraction. Overall, the NVEC method provides a more reliable and complete edge representation, making it particularly advantageous for downstream tasks such as point cloud registration.

4.2. Point Cloud Registration Validation

To evaluate the effectiveness of the proposed Edge-Preserving Iterative Closest Point (EPICP) algorithm in point cloud registration, we conducted comparative experiments using three Stanford datasets: Bunny, Dragon, and Happy Buddha. Three methods were evaluated: traditional ICP, RANSAC-enhanced ICP, and the proposed EPICP, where RANSAC was applied as an initial pose optimizer for all methods.

As shown in Table 1, EPICP achieves lower RMSE than the other two methods, indicating improved registration accuracy for rigid object alignment. Although EPICP incurs a slightly longer runtime, this can be mitigated through Python-based multithreaded parallelism, enabling efficient execution.

TTo further validate the robustness of EPICP under low-overlap conditions (less than 50%), two additional experiments were conducted. In the first experiment, the Happy Buddha model was tested with a 47% overlap. In the second, the Bunny model was used with only 33% overlap. All three methods were evaluated under identical conditions. As shown in Figure 4, EPICP consistently outperformed the other methods across both overlap settings.

These findings demonstrate that EPICP provides superior accuracy in low-overlap point cloud registration scenarios. It consistently outperforms traditional methods and exhibits strong robustness even when the overlap is significantly limited.

Table 2 presents RMSE values and statistical results for the two overlap settings. Under 47% and 33% overlap, the registration accuracy of EPICP improved by 61.9% and 75.7%, respectively, compared to baseline methods. The corresponding standard deviations are within acceptable bounds, confirming result stability.

Figure 5 illustrates the RMSE variation under different overlap ratios. EPICP maintains stable registration accuracy under moderate-to-low overlap. When the overlap becomes extremely low, RMSE increases significantly and the registration becomes less stable. Nevertheless, compared with traditional algorithms, EPICP still exhibits notable performance advantages under challenging conditions.

4.3. Ablation Study

To validate the contribution of each module within the EPICP algorithm, a series of ablation experiments were conducted on the Stanford Bunny dataset. As shown in Figure 3, different module combinations were tested to assess their individual and joint impact on the final registration accuracy. In the first configuration, only point-to-plane ICP was applied without any initialization. The RMSE reached 1.91 cm, indicating that without a good initial pose, the registration tends to converge to a suboptimal solution. In the second configuration, pose initialization was enabled but without refinement, yielding an RMSE of 2.83 cm. This shows that initial alignment alone is insufficient and leads to large residual errors. In the third configuration, only outlier rejection was activated (with initialization and fine registration disabled), achieving an RMSE of 0.93 cm. Although the exclusion of mismatched correspondences improves accuracy, the absence of initialization still results in notable alignment errors. The fourth configuration, using only fine registration, reduced the RMSE to 1.02 cm, suggesting that this module contributes to accuracy, but its effect is limited without supporting components. Configurations five to seven, which combined two modules at a time, showed moderately improved performance, with RMSEs remaining relatively stable. These results confirm that the modules are complementary and can cooperate effectively. In the eighth configuration, when all three modules (initialization, outlier rejection, and fine registration) were activated, the RMSE was minimized, and the highest accuracy was achieved. This demonstrates the capability of the proposed EPICP framework to maintain high registration precision under low-overlap conditions by leveraging robust initialization, effective correspondence filtering, and accurate refinement.

Table 3. Ablation study on Bunny Dataset.

	EdgeExtraction	FPFH	Local Registration	Global Registration	RMSE(cm)
1)				✓	1.91
2)	✓			✓	2.83
3)		✓		✓	0.93
4)			✓	✓	1.02
5)	✓	✓		✓	1.26
6)		✓	✓	✓	1.26
7)	✓		✓	✓	1.42
8)	✓	✓	✓	✓	0.16

Table 3. Ablation study on Bunny Dataset.

	EdgeExtraction	FPFH	Local Registration	Global Registration	RMSE(cm)
1)				✓	1.91
2)	✓			✓	2.83
3)		✓		✓	0.93
4)			✓	✓	1.02
5)	✓	✓		✓	1.26
6)		✓	✓	✓	1.26
7)	✓		✓	✓	1.42
8)	✓	✓	✓	✓	0.16

5. Conclusions

To address the challenge of point cloud registration under low-overlap conditions, this study proposes a novel registration framework that integrates edge enhancement with quaternion-based fusion. The proposed method overcomes the limitations of traditional approaches that often fail to extract reliable local features in sparse-overlap scenarios, thereby improving registration accuracy. Specifically, the method first calculates the maximum angular deviation between neighboring normals to detect geometric edges. These edge points are then used for robust feature matching. A local rigid transformation is subsequently estimated, followed by quaternion-based averaging to obtain a reliable initial pose. Finally, a point-to-plane ICP refinement step is performed to achieve precise alignment. Experimental results on the Stanford 3D Scanning Repository demonstrate that the proposed method significantly outperforms traditional techniques in terms of accuracy and robustness, particularly under low-overlap conditions.

However, some limitations remain. The method relies heavily on accurate edge detection; noisy or incomplete edge features may affect registration outcomes. In extremely low-overlap scenarios, over-filtering may reduce the number of matching pairs, potentially leading to registration failure. Nevertheless, the proposed approach shows strong potential for achieving accurate and reliable registration under challenging conditions. Future work will focus on improving edge detection robustness, optimizing module integration, and enhancing adaptability to diverse and complex registration environments.