Large-Scale Point Cloud Semantic Segmentation with Density-Based Grid Decimation

1. Introduction

Point clouds, serving as a fundamental geospatial data structure [1], play an import role in 3D perception and understanding. Accurate semantic segmentation of point clouds into distinct entities like roads, buildings, and trees is vital for applications such as 3D reconstruction or autonomous driving. Traditional techniques for the semantic segmentation of point clouds include region growth-based segmentation [2], model fitting-based segmentation [3], graph optimization-based segmentation [4], and edge information-based segmentation [5]. While these methods are reliable and widely used in commercial applications, they often encounter challenges when processing extensive point cloud data.

The past several years have witnessed significant advancements in semantic segmentation algorithms rooted in deep learning, offering promising solutions for handling extensive earth observation data. Notable developments in point cloud semantic segmentation domain include Point-Net [1], Point-Net++ [6], Point-CNN [7], DGCNN [8], Point-RNN [9], Point Transformer [10]. However, the scope of many such methods is confined to smaller point cloud datasets, a limitation partly attributed to their dependence on time-consuming or inefficient point sampling methods. For example, using the farthest point sampling (FPS) algorithm for sampling 10% of points from a 1 million point cloud takes over 200 seconds [6]. In contrast, RandLA-Net introduced by [11], incorporates the Random Sampling (RS) algorithm, which exhibited remarkable efficiency, completing the same sampling task in just 0.004 seconds. Subsequently, many works [13,14,15] have adopted RS algorithm during the down-sampling processes in their networks. This substantial decrease in sampling time highlights the potential of the RS algorithm in optimizing semantic segmentation networks for large-scale outdoor point cloud data.

Large-scale outdoor point cloud datasets often contain millions or even billions of individual points, making it computationally infeasible to process them in their entirety. Therefore, preprocessing before network training is crucial to reduce the computational burden by selecting representative subsets of points for analysis. While grid-based decimation is a commonly employed preprocessing technique [13,14,15,16,17], it fails to address the uneven distribution of object categories within large-scale point clouds, primarily due to its use of a fixed grid size. This limitation often leads to unsatisfactory segmentation results, especially for small-sized objects. To address variations in point cloud density across different outdoor scenes, we propose a density-based grid decimation algorithm that dynamically adjusts grid size based on the density of input point clouds.

Accurate semantic segmentation of point clouds also depends on robust feature extraction from their complex structures. This involves developing innovative architectures that can effectively down-sample large-scale point clouds while preserving important spatial structures and semantic information. Recent studies, such as those by [10,18,19], have shown that Transformer-based models excel in point cloud semantic segmentation, highlighting their remarkable feature recognition and extraction capabilities. However, these Transformer-based methods can result in significant computational overhead and require considerable memory resources, especially when processing large point batches. Thus, developing a network architecture that balances feature extraction capabilities with GPU constraints remains a key focus.

To address the challenges of uneven point density, imbalanced sampling, and limited feature extraction capabilities in large-scale outdoor point clouds, we propose a novel framework, RT-Net, that leverages a density-based grid decimation algorithm as its foundation. This decimation algorithm dynamically adjusts grid sizes based on point cloud density, ensuring balanced representation of object categories and improving computational efficiency. Further, RT-Net incorporates advanced attention-based modules to enhance feature extraction from both local and global structures. These innovations enable RT-Net to achieve superior segmentation performance, particularly for small-sized object categories, while maintaining computational feasibility on modern GPUs. The main contributions of this paper are as follows:

• Density-Based Grid Decimation Algorithm: A novel preprocessing method that dynamically adjusts grid sizes based on point density, addressing imbalanced sampling and improving computational efficiency compared to traditional grid-based approaches.
• Attention-Based Modules: Two new modules—Local Attention Aggregation (LAA) and Attention Residual (AR)—are designed to efficiently capture both local and global features, reducing memory consumption and computational overhead.
• RT-Net Architecture: A new segmentation network that integrates the density-based grid decimation algorithm with the proposed attention-based modules. RT-Net outperforms existing benchmarks, especially in segmenting small-sized object categories, as demonstrated on large-scale datasets.

2. Related Works

2.1. Deep Learning Methods for Extracting Point Cloud Features

Deep learning-based methods for extracting features from point clouds are mainly categorized into projection-based, voxel-based, and point-based strategies. The projection-based method converts 3D point clouds into multiple 2D views to leverage established 2D convolutional neural networks [15,20,21,22,23], but they face challenges like occlusion and overlap. Voxel-based approaches involves converting point clouds into 3D voxel grids, allowing for the application of 3D convolutions for feature extraction [24,25,26,27,28], yet they often lead to data redundancy and increased computational demands, making them less fitting for processing large-scale outdoor point cloud datasets. In contrast, point-based approaches have become increasingly popular for their adaptability to point cloud structures and flexibility for adjustment, as evidenced by the works of [1,6,7,10,12,13,14,16,17,18,19,29,30,31].

The success of Transformer models in both natural language processing and computer vision has spurred their increasing adoption in the domain of point cloud semantic segmentation. Point Transformer, introduced by Zhao [10] is a pioneering study which explores the implementation of Transformer architecture within point cloud semantic segmentation. However, as the model goes deeper, it tends to cause overfitting in the network’s feature extraction ability on the majority of point categories. Recognizing these limitations, [18,19] improved the network by optimizing feature extraction modules using various techniques. PReFormer [17] further enhanced self-attention calculations in Transformer, thereby improving memory efficiency and accuracy in segmenting point clouds. These innovations underscore the potential of attention to balance feature extraction capabilities with computational feasibility, making them promising solutions for the semantic segmentation of extensive point cloud datasets.

2.2. Preprocessing of Point Clouds Before Training

Point cloud preprocessing involves a range of processing steps aimed at preparing point clouds for integration into training networks. Given current hardware constraints, direct training on all points within large-scale point clouds is impractical. Consequently, thinning processing is typically employed as a necessary step. Many preprocessing techniques for vast point cloud datasets avoid heuristic or complex mathematical procedures due to their high memory and time consumption. Instead, spatial region filtering [32] and grid-based decimation [11,15,33,34] are preferred alternatives for preprocessing point clouds. Since spatial region filtering relies heavily on empirical knowledge, grid-based decimation is gaining popularity. However, when utilized in large-scale complex point cloud scenes, grid-based decimation often results in uneven point cloud distribution. KPFCNN [35] leverages density-based kernels for convolution processing in point cloud segmentation, but it still faces the challenge of introducing redundant data. To address this, we propose using density-based grid decimation to mitigate the uneven distribution of object categories.

3. Methodology

3.1. Network Architecture

Our network’s design follows an encoder-decoder pattern (see Figure 1.). Initially, a fully connected (FC) layer is used to process the input point clouds, allowing the extraction of features for each point. During the encoding phase, every encoding layer incorporates a Local Attention Aggregation (LAA) module along with a RS module. The network uses four encoding layers to progressive down-sample the point size in a sequential manner (from N to N/4, then N/16, N/64, and finally N/256), with N denoting the initial point count. Concurrently, the feature dimensions for each point are expanded following the sequence. The architecture bridges the encoder and decoder with a shared MLP (Multi-Layer Perceptron) featuring a 1x1 convolutional kernel, an activation function, and a normalization layer to encapsulate the contextual information of the point clouds. In the decoding phrase, each decoder layer consists of an Attention Residual (AR) module and an interpolation Up-sampling (US) module. The shape transformations of the point clouds are reversed compared to the encoder. Finally, a pair of FC layers is employed to produce the semantic label predictions.

3.2. Local Attention Aggregation

Figure 2. details the LAA module. It consists of two units: Local Spatial Encoding and Attention Aggregation. The former unit integrates each point’s positional context in relation to others, which enables the latter unit to consider spatial relationships among points during self-attention operations, rather than solely relying on feature similarity to calculate attention coefficients. The latter unit employs a self-attention mechanism to extract internal features from each block of the K-nearest neighbor points. Unlike the LFA module introduced by Hu [30], we have replaced its Attentive Pooling unit with our Attention Aggregation unit, as shown on the right side of Figure 2,

$${\mathrm{p}}_{\mathrm{r}\mathrm{e}\mathrm{l},\mathrm{k}}={\mathrm{p}}_{\mathrm{c}\mathrm{o}\mathrm{o}\mathrm{r},\mathrm{k}}-{\mathrm{p}}_{\mathrm{c}\mathrm{o}\mathrm{o}\mathrm{r},\mathrm{i}},$$

(1)

$$\mathrm{d}\mathrm{i}\mathrm{s}{\mathrm{t}}_{\mathrm{k}}=\left|\right|{\mathrm{p}}_{\mathrm{c}\mathrm{o}\mathrm{o}\mathrm{r},\mathrm{k}}-{\mathrm{p}}_{\mathrm{c}\mathrm{o}\mathrm{o}\mathrm{r},\mathrm{i}}\left|\right|,$$

(2)

$${\mathrm{r}}_{\mathrm{k}}={\mathrm{p}}_{\mathrm{c}\mathrm{o}\mathrm{o}\mathrm{r},\mathrm{k}}\oplus {\mathrm{p}}_{\mathrm{c}\mathrm{o}\mathrm{o}\mathrm{r},\mathrm{i}}\oplus {\mathrm{p}}_{\mathrm{r}\mathrm{e}\mathrm{l},\mathrm{k}}\oplus \mathrm{d}\mathrm{i}\mathrm{s}{\mathrm{t}}_{\mathrm{k}},$$

(3)

$${\mathrm{f}}_{\mathrm{p}\mathrm{o}\mathrm{s},\mathrm{k}}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}\left({\mathrm{r}}_{\mathrm{k}}\right).$$

(4)

Here, computes the Euclidean distance, represents the operation of concatenation and signifies the convolution operation. In the above equations, is the relative position bias computed from the center point coordinates and the feature point coordinates, and is the corresponding absolute distance. The set combines these values and is passed through a convolutional layer to obtain the local spatial encoding.

The Attention Aggregation unit merges the collection of neighboring features, to derive aggregated features for each center point. It utilizes a self-attention block to expand the receptive field of each point, capturing long-range contexts for better generalization. First, the collection of neighboring features undergo standard self-attention operations, producing attention features of. These attention features are subsequently merged with the neighboring point features of. This operation facilitates a comprehensive capture of local features within point clouds by supplementing them with self-attention features, thus enhancing the overall representation. Finally, the concatenated features are summed and passed through a MLP layer with shared parameters to derive the aggregated features of. Through Local Spatial Encoding and Attention Aggregation units, the input point clouds transform into their corresponding aggregated features that effectively encode local contextual information.

3.3. Attention Residual

As depicted in Figure 1 and Figure 3., the AR module receives inputs from both the preceding layer’s output features and the aggregated features generated by the LAA module. First, it conducts up-sampling on the previous layer’s output. The up-sampled features are then fed into the Attention block as the Query input. The Key and Value inputs for the Attention block are extracted from the LAA module’s aggregated features using a 1×1 convolutional layer. Subsequently, attention features are computed in the Attention block, and their feature dimensions are restored through concatenation with the up-sampled point cloud features. This procedure is devised to accelerate computations and enhance the precision of point cloud feature extraction. Further elaboration on this topic are provided in the Ablation of AR in Section Ⅳ. Finally, the concatenated result is transformed by a shared MLP into attention residual features, which serves as the output of the AR module.

The equations used in this module are represented as follows:

$$\mathrm{A}\mathrm{R}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{a}\mathrm{t}\mathrm{e}\left(\mathrm{r}\mathrm{e}\mathrm{s}.,\mathrm{ }\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\max \left({\displaystyle \frac{\mathrm{Q}{\mathrm{K}}^{\mathrm{T}}}{\sqrt{\mathrm{d}}}}\right)\mathrm{V}\right)$$

(1)

$$\mathrm{Q},\mathrm{K},\mathrm{V}=\mathrm{X}{{\mathrm{W}}_{\mathrm{q}}}^{{\displaystyle \frac{\mathrm{d}}{2}}},\mathrm{X}{{\mathrm{W}}_{\mathrm{k}}}^{{\displaystyle \frac{\mathrm{d}}{2}}},\mathrm{X}{{\mathrm{W}}_{\mathrm{v}}}^{{\displaystyle \frac{\mathrm{d}}{2}}}$$

(1)

Here, is the feature of the residual layer. Q stands for the query matrix, K for the key matrix, and V for the value matrix. X represents the point set matrix, while,, are the linear transformation matrices.

3.4. Density-Based Grid Decimation

Given the extensive volume of points within raw point clouds, direct utilization of the original data for model training is considered impractical. Data preprocessing becomes a necessary step before model training. In the original point cloud set, each element is delineated by its three-dimensional coordinates. The sequence of steps involved in density-based grid decimation for point clouds is as follows:

• Compute the upper and lower bounds of the 3D coordinates in the original point cloud set.
• Specify the grid size and calculate the grid count per dimension.
• Determine the three-dimensional grid indices for every point within the point set by their grid index.
• Classify points in based on the indices calculated in the previous step. Points sharing the same index are grouped into a grid. Unlike traditional grid-based decimation, where the grid size is fixed, our approach considers the density of points and dynamically adjusts the grid size accordingly. If the point count in a grid exceeds a preset threshold, the grid is subdivided into four equal-sized sub-grids. This process repeats until the number of points in all grids falls below the preset threshold. Subsequently, each grid randomly retains one point while discarding the rest, completing the density-based grid decimation process, and obtaining

For the sparse point clouds (m is the number of sub-point clouds, automatically generated according to grid-based decimation) resulting from density-based grid decimation processing, they serve as the basis for model training. Utilizing a sampling method based on probability ensures that the data used for training in each epoch is not fixed. This variability arises because the data sampled in each epoch differs, aiding the model in comprehending more intricate scenes. In the experiments section, the data preprocessing process involves simultaneous density-based grid decimation and probability sampling operations on both the training and testing datasets. Additionally, for every point in the original point clouds, the model identifies its closest neighbor in the thinned point clouds. This process allows mapping predictions from the thinned point clouds back to the original for precise performance assessment. Figure 5 shows that, under identical sampling settings, the points sampled using our method are more concentrated and primarily distributed in high-density regions, such as those near roads and buildings. In contrast, conventional grid-based decimation results in more dispersed sampling points.

Figure 4. Grid-based decimation and density-based grid decimation on the L001 region of Toronto3D.

Figure 4. Grid-based decimation and density-based grid decimation on the L001 region of Toronto3D.

Figure 5. Semantic segmentation results on Toronto3D datasets, compared with RandLA-Net.

Figure 5. Semantic segmentation results on Toronto3D datasets, compared with RandLA-Net.

4. Experiments and Results

4.1. Density-Based Grid Decimation

In the following experiments, we used three benchmark datasets: Toronto3D [36], Semantic3D [37], and SemanticKITTI [38], applying the same network architecture to them. The initial grid size in the preprocessing phrase was configured at 0.1. An Adam optimizer was employed for training with its default parameter settings. To reduce the influence of class imbalance, we opted for a weighted cross-entropy loss, with class weights calculated as the inverse of their frequency in the training samples. The model was trained for 100 epochs. We initiated the learning rate at 0.01, with each subsequent epoch’s rate set to 95% of its predecessor. For the KNN algorithm, we utilized the Kd-Tree module from the Scikit-learn package, setting it to find 16 closest neighbors. All algorithms in this section were implemented using PyTorch 1.13.1 and CUDA 12.4 on Ubuntu 22.04. Our experiments were performed on a NVIDIA GeForce RTX 3090 24G GPU.

4.2. Density-Based Grid Decimation

In this section, we assess the performance of our RT-Net architecture on large-scale point clouds and compare its results with those of state-of-the-art semantic segmentation algorithms. For the Toronto3D dataset, following the practice described by Tan [36], we designated the L002 region as the test subset, with the remaining regions form the training subset. Performance assessment was conducted using eight categories of mean Intersection over Union (mIoU) along with their respective IoU metrics. For the Semantic3D dataset, following the guidelines set by Thomas [35], we assigned two regions as the test subset, with the rest of the regions allocated for training and validation purposes, and applied the same evaluation metrics. For the SemanticKITTI dataset, sequences 00 to 10 are used as the training set, sequence 08 as the validation set, and sequences 11 to 21 as the test set. During the preprocessing stage, infrequent categories are discarded, resulting in 19 categories being used for training and evaluation.

Table 1. summarizes the quantitative results for the Toronto3D dataset, presenting mIoU and IoU values for each category. The data from the table indicates that RT-Net, integrated with a density-based grid decimation approach for semantic segmentation on the Toronto3D dataset, surpasses current state-of-the-art models in mIoU. Observations also reveal that density-based grid decimation significantly enhances the segmentation performance of RT-Net across various small-sized object categories, as shown in Table 1. Among these categories, fences demonstrate the most significant improvement, with a 20.87% increase from 49.42% to 70.29%. Road markings, poles, and utility lines have also seen substantial improvements, with increases of 17.5%, 10.11%, and 5.67%, respectively. However, there are slight reductions in the IoU scores for certain categories: roads see a decrease from 94.69% to 92.28%, natural areas from 96.62% to 95.03%, utility lines from 88.06% to 86.97%, and cars from 93.37% to 88.06%, respectively.

It should be noted that our rigorous replication of RandLA-Net’s experiment yielded lower mIoU value (76.64%) than the authors’ reported figures (81.77%). Despite ensuring consistency in parameter settings according to the original study, discrepancies likely arose due to variations in experimental conditions. In our replication of RandLA-Net, employing the density-based grid decimation method, we realized enhanced outcomes as well, with an increase in performance from 76.64% to 81.55%. As shown in Fig.6, L001 demonstrates the strong segmentation performance of our method on edge points, while L002 highlights its effectiveness in segmenting pole point clouds. Additionally, L003 and L004 illustrate the method’s ability to accurately identify ground marks.

Figure 6. Semantic segmentation results on SemanticKITTI.

Figure 6. Semantic segmentation results on SemanticKITTI.

Under identical experimental conditions, we conducted semantic segmentation experiments on the Semantic3D dataset. The proposed RT-Net outperforms other semantic segmentation models, as shown in Table 2., reaching a state-of-the-art mIoU score of 79.88%. Specifically, our model achieves remarkable results in the segmentation of high vegetation, hardscape, and cars categories, with IoU scores of 90.35%, 60.48%, and 67.55%, respectively. These results highlight our model’s robust segmentation capabilities, particularly in challenging terrains and diverse landscape categories, and indicate its strong generalization potential.

Our comparative analysis across different datasets indicates that the results can be attributed to two main factors: the robust feature extraction capabilities of the attention modules and the density-based grid decimation method’s proficiency in evenly distributing point cloud categories and preserving fine details.

4.3. Efficiency of Density-Based Grid Decimation

In this section, the efficiency of our density-based grid decimation is assessed against the traditional grid-based method. For density-based grid decimation, the initial grid can choose as large a value as possible according to the span of the actual space coordinates to ensure that there is enough space for calculation, but at the same time, too large a value will lead to a waste of computational efficiency. We conducted a series of four experiments to evaluate their impact on semantic segmentation performance. All the preprocessing approaches were followed by the full RT-Net semantic segmentation network.

Table 4. demonstrates our density-based grid decimation method’s advantage, with higher mIoU scores over the grid-based method. It is noteworthy that initializing the grid size at 1.0 yielded the best outcomes, with a mIoU score of 86.91%. However, we ultimately opted for density-based grid decimation starting at a grid size of 10. This choice was influenced by the substantial time expenditure of the former solution, which is almost 120 times that of the latter, for only a marginal enhancement in mIoU.

It is worth mentioning that our preprocessing strategy is not limited to RT-Net; it is versatile and can be applied to other networks. This adaptability is evident from successful experiments conducted with RandLA-Net, as indicated in Table 1. The integration of density-based grid decimation into the RandLA-Net methodology also resulted in notable enhancements, particularly for small-sized object categories such as road markings, utility lines, poles, cars, and fences. These enhancements are attributed to the ability of density-based grid decimation to effectively recognize and preserve intricate details within the scene. Consequently, it emerges as an effective preprocessing technique for large-scale point clouds, offering efficiency gains across different semantic segmentation networks.

To validate the efficacy of the modules in RT-Net, we carried out a set of ablation studies concentrating on the LAA and AR modules. These studies were all conducted on the Toronto3D dataset applying density-based grid decimation, with evaluations focused on the L002 region.

4.4. Ablation of RT-Net Framework

The core components of RT-Net framework are the LAA and AR modules, while the remainder of the framework adheres to the design put forward by Hu [11]. To showcase the effectiveness of each component, we designed two key ablation experiments:

• Removal of Self-Attention Pooling: This structure facilitates the aggregation of features from neighboring points in the point clouds. Upon its removal, we replaced it with standard max/mean/sum pooling for the local feature encoding.
• Removal of Attention Residual: This structure enhances the effectiveness of residual adversarial networks by emphasizing feature values through attention mechanisms. Upon its removal, we utilized an original residual connection.

Table 5. presents the mIoU results derived from our network ablation studies. The results underscore the crucial function of the self-attention pooling module in enhancing the network’s overall performance. Its absence leads to a notable 21.13% drop in mIoU, stemming from the substantial loss of detailed features during random sampling phrase. On the other hand, the Attention Residual (AR) module is also crucial for the network’s overall performance. Its removal is associated with a significant drop in mIoU, amounting to an 11.98% decrease. This indicates that while its impact may be less dramatic than the self-attention pooling module, the AR module is nonetheless a key contributor to the network’s effectiveness.

4.5. Ablation of AR

As described in Section Ⅲ, we formulated attention residuals and subsequently calculated attention residual features. To assess the impact of different residual block structures, we conducted additional ablation experiments, summarized in Table 6.:

• RT-Net with Residual: Utilizes a standard residual connections module.
• RT-Net with Attention Residual: Uses an attention residual connections module with addition.
• RT-Net with Attention Residual (Concatenation): Employs our complete attention residual module with concatenation.

Table 6. displays the impact of different residual configurations on RT-Net’s performance. Implementing the attention residual module in RT-Net led to a 5.15% improvement in mIoU scores over the use of the standard residual structure. Furthermore, utilizing the attention residual module with concatenation in RT-Net yielded an additional 3.04% increase in mIoU.

5. Conclusions

This paper presents RT-Net, a novel semantic segmentation framework for large-scale point clouds, harnessing the power of random sampling, attention mechanisms, and density-based grid decimation. RT-Net features innovative local attention aggregation and attention residual modules designed to capture a comprehensive range of features within point clouds, including both local and global characteristics. The integration of a density-based grid decimation algorithm for preprocessing large-scale point clouds addresses the issue of imbalanced sampling categories encountered with traditional preprocessing methods. These innovations enable RT-Net to outperform existing methods, particularly in the segmentation of small-sized object categories such as road markings, utility lines, poles, and fences. Our approach has been rigorously evaluated on the Toronto3D, Semantic3D, and SemanticKITTI datasets, achieving state-of-the-art mIoU scores of 86.79% on Toronto3D and 79.88% on Semantic3D, while demonstrating significant improvements in small object categories across all three benchmark datasets. These results highlight RT-Net’s robustness and adaptability to various datasets and scene types.

While RT-Net has demonstrated outstanding performance, there are several promising areas for future work. One such area is the implementation of domain-adaptive point cloud transfer learning. Considering the labor-intensive process of annotating point cloud data, exploring domain-adaptive transfer learning is a crucial next step. This approach, by leveraging existing labeled datasets, could significantly enhance segmentation performance and streamline the learning process across various scenes. Additionally, achieving real-time capabilities in semantic segmentation of point clouds is vital for applications that require rapid analytical and decision-making processes. Further research in this area will not only reinforce RT-Net’s practicality in real-world applications but also ensure the network's ongoing adaptability and responsiveness to new data.