Towards Unified Point Cloud 3D Place Recognition

1. Introduction

Place recognition is one of the key tasks in computer vision and robotics. It plays an important role in simultaneous localization and mapping (SLAM) frameworks, which identify previously visited locations using visual, structural, and/or semantic cues. There are different solutions for place recognition, and most of them are based on image information – Visual Place Recognition (VPR) methods [1]. Despite the success of VPR methods, their performance is affected by various environmental factors such as limited field of view, viewpoint variations, etc.

Alternatively, 3D Point Cloud-based Place Recognition (3D-PCPR) is a promising approach motivated by the relative invariance of the point cloud structure to changes in illumination, weather, seasons, etc. 3D-PCPR is also of interest due to the widespread use of light detection and ranging (LiDAR) sensors in robots and autonomous vehicles [2]. Due to the limited number of data for 3D-PCPR and the difficulties in collecting them, existing approaches rely on specific datasets. For this reason, there is no guarantee that the methods will work with the same accuracy outside of the training set. In order to deal with unseen scenes, our work has been dedicated to increasing the generalizability of 3D-PCPR.

Place recognition has traditionally been viewed as an instance retrieval task, where the location of the query is estimated using the locations of the most similar instances obtained by querying a large geotagged database. Given a point-cloud query, the 3D-PCPR method will predict an embedding such that the query will be close to the most structurally similar point clouds in the embedding space [3,4].

In contrast to images, where pixels are regularly distributed on an image plane, point clouds have irregular density, and in real-time scenarios they are sparse. For this reason, many methods rely on sparse convolution [5,6,7,8]. Another group of methods [9,10] replaces the cubical neighborhood in convolution with the k-nearest neighbor and performs a reduction using max pooling instead of summation.

In real-world scenarios, the quality of point cloud data depends on specific LiDAR, and point cloud density varies over space. In our work, we use the domain shift to adjust our model for the 3D-PCPR task (see Figure 1). This brings high generalizability to our approach. We have achieved comparable accuracy for a small number of training-tuning epochs. We also tested several techniques that take into account the density variation in different regions of the point cloud. For example, by measuring the local distance variation or the mean distance, we can estimate the local density in the area. This idea is motivated by graph theory [11,12] and other recent works [13,14,15].

We contribute by adapting a 3D backbone and aggregation features from the foundation model to the 3D-PCPR task in order to increase generalizability across different datasets.

2. Related Works

2.1. Learning-Based Methods for 3D-PCPR

The latest 3D-PCPR methods are based on the use of specific datasets such as HeLiPR [16], Oxford RobotCar [17], CS-Campus3D [18], etc. One of the first methods [19] for solving 3D-PCPR is based on PointNet, where the local feature descriptors obtained from the network are fed into the NetVLAD layer. NetVLAD interprets the output of a neural network $f(.)$ as local descriptors for VLAD (Vector of Locally Aggregated Descriptors) [20], which maps an input point cloud P to a compact discriminative global descriptor vector $f\left(P\right)$. MinkLoc3D [5] extracts local features with a 3D Feature Pyramid Network (FPN) architecture and uses generalized mean pooling (GEM) [21] to produce a global point cloud descriptor. CrossLoc3D [18] solves the representation gap in cross-source 3D place recognition using multi-grained features, adaptive kernels, and an iterative refinement process that gradually shifts embedding spaces from different sources to a single embedding space. HeLiOS [22] is a recent work that emphasizes the Lidar Place Recognition problem. HeLiOS utilizes overlap mining and guided-triplet loss to overcome incorrect pairing. Its architecture is based on the Spherical Transformer proposed by [23].

2.2. Foundation Models for 2D and 3D Understanding

Due to the diversity of the open world, the collected annotated 3D data contain a large number of “unseen” objects, and many existing place recognition solutions perform poorly outside the training distribution. The issue of low generalizability has been resolved (trained on an enormous amount of data from the Internet) in 2D vision by Contrastive Vision-Language Pre-training (CLIP) [24], which proposes to learn transferable visual features with natural language supervision. The authors of [25] introduce the AnyLoc model for image-based 2D place recognition without specific training. AnyLoc explores the properties of dense foundation model features and combines them with unsupervised feature aggregation techniques. Foundation models encode rich visual features that serve as the right substrate upon which a universal VPR solution may be built. A universal solution must be applicable anywhere (in any environment), anytime (day-night or seasonal variations), anyview (robust to perspective viewpoint variations). A universal approach to 3D-PCPR should satisfy the same requirements.

There are many unannotated object datasets consisting of point cloud and image pairs. Inspired by CLIP-based adaption methods, many papers [26,27,28,29,30,31] rely on transferring the 2D knowledge from the CLIP model to the 3D space. For example, the Uni3D foundation model [31] is trained on large-scale 3D data [29], which consists of four object datasets: Objaverse [32], ShapeNet [33], 3D-FUTURE [34], and ABO [35]. The point clouds in such datasets are uniformly dense. The point cloud is partitioned into several point patches with farthest point sampling (FPS) [36] and k nearest neighbors (kNN). Then, from the point patches mini-PointNet obtains a sequence of point embeddings, which can be treated as input to standard transformers [37]. Due to the analogy of point patches to image patches, 2D VIT can be used as a 3D backbone to encode the point cloud [29,30,31]. Instead of training 2D ViT from scratch for point clouds, the authors of [31] use structurally equivalent ViT from the foundation model.

2.3. Point Cloud Sampling

Point clouds are very memory-intensive, especially with a large batch size, and point sampling is a necessity to make deep learning methods possible. According to topological data analysis [11,38], a subset $Q\subseteq P$ is called a $\delta$-sample of a metric space ($P,d$), if $\forall p\in P$, there exists a $q\in Q$, so that $\mathrm{d}(p,q)\le \delta$. Also, Q is called $\delta$-sparse if $\forall (q,r)\in Q\times Q$ with $q\ne r,\mathrm{d}(q,r)\ge \delta$. Approximation Q of point cloud P can be expressed by covering and packing conditions: $\forall p\in P,\mathrm{d}(p,Q)\le \gamma$ and $\forall q\ne q^{\prime }\in Q,\mathrm{d}\left(q,q^{\prime }\right)\ge {\gamma }^{\prime }$. If $\gamma ={\gamma }^{\prime }$, Q is called as $\gamma -\mathrm{net}$ of P at resolution $\gamma$.

Many voxel-based approaches rely on sampling, assuming that each voxel has a homogeneous point density [15]. It is common practice to partition the point cloud data into several voxels and perform random sampling in each voxel to build a training set with the maximum number of points allowed by the architecture. In object-wise datasets, points are compact, and the density distribution slightly changes over the different regions. Since point cloud datasets typically have a very high number of points, deep learning models are unable to include every data point simultaneously. The two most popular methods for point cloud downsampling are farthest point sampling (FPS) and random sampling (RS). FPS is an iterative method that maximizes point cloud coverage by selecting the farthest points from those already chosen.

Recent methods [13,14,39,40] exploit advancements in deep learning for downsampling purposes. Despite their relatively good performance, these methods are highly dependent on the dataset. The works [13] and [14] underscore the importance of differentiating between the various points to sample. They distinguish edge and non-edge points by calculating the correlation of distances to k-neighborhoods.

2.4. Heterogeneous Point Cloud

Geomapping with point clouds collected by airborne or close-range scanning LiDARs is complicated because of heterogeneity. For instance, LiDAR sensors (e.g., Velodyne units) employ rotating beams mounted on the autonomous vehicle, creating spatial heterogeneity (the further from LiDAR origin the distance is, the less dense the points become). The standard sampling techniques with the voxelization method [15] can lead to accuracy degradation because LiDARs can produce non-informative points and regions. In [15] uniform grid-based sampling is used, followed by calculation of the average point density per grid. It creates more balanced training sets to learn [41]. In order to estimate density for patch-based methodology to form input, codensities and correlation maps can be used [12,13,14]. They measure compactness and irregularity in point cloud, respectively. Due to a fixed hyperparameter in $k-$nearest neighbors ($kNN$), receptive field decreases in denser regions [42]. High value of k allows to capture the global information about a scene (not only local structures).

3. Problem Definition

The goal of 3D-PCPR is to retrieve the highest-probability place from a pre-built database of places. It is based on the calculation of similarities between global descriptors. Global descriptors should have a certain discriminativeness: discriminative for different places but keeping similar for places close to each other.

Problem Statement:Considering a query point cloud $\mathbf{q}$ and a database $DB=\left\{{\mathbf{p}}_1,\dots ,{\mathbf{p}}_s\right\}$, where s is the number of point clouds in the database, the goal of 3D-PCPR is to retrieve the point cloud ${\mathbf{p}}_i\in DB$ that is structurally most similar to $\mathbf{q}$ and the one that covers the same location.

We aim to generate a global descriptor for each point cloud $\mathbf{p}\in {\mathbb{R}}^{N\times 3}$. A point cloud can be defined as an unordered set of points ${\left\{{\mathbf{x}}_j\right\}}_{j=1}^N$, where ${\mathbf{x}}_j$ are the Cartesian coordinates for the $j^{\mathrm{th}}$ point in the 3-dimensional space. The point cloud is encoded with some mapping function $\Omega =h\left(f\right(·\left)\right)$, where the feature extractor $f(·):{\mathbb{R}}^{N\times 3}\to {\mathbb{R}}^{n\times d}$ derives local embeddings from N points, and the aggregation function $h(·):{\mathbb{R}}^{n\times d}\to {\mathbb{R}}^e$ compresses these embeddings into a global descriptor. The task of point cloud-based retrieval is to find a global descriptor in the database that gives the minimum difference with the global descriptor of the query.

4. Method

The idea of our pipeline is shown in Figure 2. It can be separated into three main steps: grouping – to organize the point cloud into several patches and encoding (mini-PointNet plus ViT) – to create representative tokens; and optimal transport – to aggregate output tokens in order to form the final global feature vector. The foundation model increases generalizability across different datasets.

Grouping

This step consists of sampling and k-nearest neighbors (kNN). The farthest point sampling selects the data points that are the farthest from each other. These points serve as centers for the patches, maximizing the coverage and diversity. First, we sample the G farthest points in the cloud using the farthest point sampling (FPS) method, where G is the number of points for the downsampled representation. These downsampled points play the role of centers for forming groups (patches). Then, for each center, we find k neighbors with the kNN algorithm. In this stage, we tested other different sampling strategies to select “the best” centers. The points in the $i-$th patch are centered as ${\mathbf{p}}_{\mathbf{ji}}-{\mathbf{c}}_{\mathbf{i}}$, where ${\mathbf{p}}_{\mathbf{ji}}$, ${\mathbf{c}}_{\mathbf{i}}$ are neighbor and center coordinates, $i=\overline{1,G}$ and $j=\overline{1,k}$. We artificially add a zero constant channel to the point cloud, treating all regions equally in terms of color.

Encoding

We use a standard approach [31,43] to calculate neighbor features:

$${\mathbf{f}}_{ji}=h_{\theta }({\mathbf{p}}_{\mathbf{ji}}-{\mathbf{c}}_{\mathbf{i}},\mathbf{0}),$$

(1)

where $h_{\theta }$ is a shared MLP as described in PointNet [19,44]. A neural network must yield consistent results regardless of the sequence order of the input points (permutation invariance). For this reason, we use max pooling as a symmetric function to encode patches:

$${\mathbf{f}}_i^{\prime }=\underset{j=1,\dots ,k}{max}{\mathbf{f}}_{ji},$$

(2)

Finally, we send patch embeddings to a foundation model to obtain representative tokens. The foundation models [30,31,37,45] were trained only on uniformly dense and colorized point clouds, which is not the case with the scene point clouds produced by the LiDAR sensor.

Optimal Transport

The output from the foundation model: $\mathbf{X}\in {\mathbb{R}}^{n\times d}$ – local feature tokens, $\mathbf{t}\in {\mathbb{R}}^d$ – class token (global scene representation). We aggregate the output (feature and cls tokens), assigning a set of features to a set of clusters. Our minimalist approach is based on one linear layer to reduce the dimensionality of local features: $\mathbf{F}={\mathbf{W}}_f\left(\mathbf{X}\right)+{\mathbf{b}}_f,$ where ${\mathbf{W}}_f\in {\mathbb{R}}^{l\times d}$ and ${\mathbf{b}}_f\in {\mathbb{R}}^l$ – the corresponding bias vector. We learn a score matrix and add a dustbin column to neglect non-informative features:

$$\overline{\mathbf{S}}=\left[\begin{array}{c}\mathbf{S},z{\mathbf{1}}_N^{\top }\end{array}\right]\in {\mathbb{R}}^{n\times (m+1)}$$

(3)

Then, the Sinkhorn algorithm [46] is applied to optimize the assignment of features to clusters. Finally, we concatenate all features and form a global feature vector similar to VLAD methodology [47].

4.1. Combined Rank-Triplet Loss

As in [22], we find the balance between ranking and the distance of relevant scans to make the training model more robust. The goal of the Truncated Smooth-AP (TSAP) loss [22,48] is to maximize the average precision for the top$-k$ positives. Although it works well, it does not directly control the distance between the positives and negatives, resulting in unstable training. Triplet margin loss [5,18,19,49] is defined as follows:

$$L_{Trip}\left(a,p,n\right)=max\left\{0,d_g\left(a,p\right)-d_g\left(a,n\right)+m\right\},$$

(4)

where $d_g(x,y)$ is a distance in the embedding space; $a,p,n$ are the anchor, positive, and negative elements of the training triplet; and m is a hyperparameter (margin). Since global feature vectors are normalized, we can easily select an appropriate value for the margin.

Finally, our combined rank-triplet loss can be expressed as follows:

$$L=L_{TSAP}+w·L_{Trip}\left(a,p,n\right),$$

(5)

where w is the relative weight of a triplet margin loss.

5. Experiments

5.1. Datasets and Evaluation Metric

We use a standard Oxford RobotCar dataset [19] for 3D-PCPR in the outdoor environment and three in-house datasets of a university sector (U.S.), a residential area (R.A.), and a business district (B.D.). Originally, Oxford RobotCar [17] was collected in a vehicle with LiDAR sensors while driving on urban and suburban roads. Each 3D scan consists of triplets of $(x,y,z)$, which are the 3D Cartesian coordinates of the LiDAR return relative to the sensor (in meters). Oxford RobotCar has a wide coverage of the city as well as repetitive routes in different weather conditions at different times. The total number of maps is 24741.

The CS-Campus3D dataset [18] is a cross-source dataset consisting of point cloud data from both aerial and ground LiDAR maps, which are tagged with UTM coordinates. The aerial data are collected by an airborne LiDAR sensor mounted on an airplane. The ground maps are collected on a Boston Dynamic Spot and a Clearpath Husky equipped with a Velodyne VLP 16 LiDAR. Similarly to a modified version of the Oxford RobotCar dataset [19], the map coordinates are shifted and scaled to $[-1,1]$. The total number of maps is 27520 and 7705 for aerial and ground data, respectively. We refer to Table 1 in [18] for a more detailed comparison of the datasets.

HeLiPR [16] is a large dataset that captures diverse environments: a narrow residential area, an urban cityscape, and environments with high dynamic change. The dataset was acquired over 37 days, with data collection occurring 3 to 4 times. It includes heterogeneous LiDARs, with OS2-128, VLP-16, Livox Avia, and Aeries II, while other existing datasets involve only spinning LiDARs. HeLiPR has twice as many points as other considered datasets, providing more object-type points to consider. It is also normalized to the range $[-1,1]$ [22].

A robot is successfully localized if the retrieved place is close to the ground truth position ($<d$). The threshold d is a user-defined parameter and is related to the resolution of the submap. We retrieve the top N matched places in the database for each query for further geometric verification [19,50]. $Recall@N$ is defined as the percentage of true neighbors for the query that are correctly retrieved based on the top N neighbors predicted from the network. $Recall@N$ equals to $Recall@1\%$ if N equals one percent of the total number of places in the database. As in [5,18,19,22,48], we use $AverageRecall@N$ and $AverageRecall@1\%$ for evaluation, where $AverageRecall@N$ is defined as the averaged $Recall@N$ on all localized queries.

5.2. Knowledge Transferring

Our approach inherits knowledge from the object point cloud domain. We compare our approach based on the foundation model with different pipelines that are specifically trained for the 3D-PCPR task. According to the Table 1 and Table 2, it is demonstrated the successful transfer of knowledge. Figure 3 shows the approximate dependency of the average recall of the top-1 candidate on the number of patches and their size. We use a high value of k to achieve a larger receptive field.

5.3. Density Variation

The FPS algorithm intends to cover the entire scene to provide more geometrical understanding and treats all samples equally. We discovered that patches of the scene can have different impacts on global geometric awareness. When we sample points in the regions with low codensity, we start to lose geometric awareness of the scene. In contrast, sampling in the regions with high codensity can lead to a loss of local details.

$$\mathrm{select}G\subseteq G^{★},G^{★}=(1+\alpha )·G,$$

(6)

where $G^{★}$ are proposals (centers) generated with the FPS algorithm. We select the centers with highest / lowest codensity to increase global geometric awareness / local details. Figure 4 shows that losing geometric awareness can be critical for 3D point cloud place recognition.

5.4. Uni-PCPR

Our approach, using standard sampling methods and a foundational model as a 3D backbone with knowledge transferring from uniformly dense 3D object point clouds, showed impressive results in terms of generalizability across different datasets (see Table 1 and Table 2). We solve the optimal transport on Oxford RobotCar dataset with minimal convolution layers to prove that not all output features should be equally treated to form the final descriptor. Uni-PCPR is based on the domain shift and optimal transport aggregation technique. We achieved comparable accuracy with the state-of-the-art approach without specific training on 3D-PCPR domain.

We do not train the entire pipeline (Uni-PCPR backbone is frozen), but only the final part with optimal transport, aggregating the feature and class tokens to form the final output global feature vector. We focused our attention on high generalizability, knowledge transfer and restricted ourselves to using more layers to aggregate features. As a remark, we want to note that these results are received without long tuning and with a substantial dimensionality reduction to improve efficiency.

6. Discussion

We investigate different strategies for intra-modal representational maximization in order to further improve the representational capability of our approach. Although we noticed slight improvement by more careful selection of the sampling set (centers) as initial input to the neural network, our experiments showed that finding the balance between geometric awareness and noise reduction is not a trivial task. Selection of “diverse” encoded point patches in feature space does not guarantee good spatial coverage. However, many works rely on the idea of diversity of tokens. We refer to several works, which can be promising for future study [10,13,51,52,53].

Also, as it was mentioned previously, we did not use the full potential of training to adapt to specific data for point cloud place recognition. Although potential techniques for a more adaptive approaches can be developed and investigated. Our results showed the high potential for future development of 3D-PCPR methods with high generalizability across different environments.

7. Conclusions

Place recognition is a crucial SLAM task that should be solvable in any environment. However, there is a shortage of geotagged point cloud data collected under different conditions. Due to the diversity of point cloud data from other domains, we propose a knowledge-transferring method. Although point clouds obtained from LiDAR in real time are sparse and nonuniformly distributed, we demonstrate in this work that more careful feature aggregation can optimize the foundational backbone for point cloud recognition tasks. This paper presents a new approach to solving the 3D place recognition task by transferring knowledge from other domains. Transporting features more carefully from the foundation model yields one of the best results in terms of generalizability across different datasets. This work is a first step toward a unified solution for place recognition in any environment.