LGSBA: Local Gaussian Splatting Bundle Adjustment for Optimizing 3DGS Rendering Quality

1. Introduction

Recently, 3DGS [1] has emerged as a cutting-edge method for generating dense and efficient maps from sparse 3D points. Unlike traditional 3D reconstruction techniques which center around accuracy and system integrity, 3DGS is dedicated to refine and densify sparse maps, offering a more efficient and flexible approach to creating high-quality scene representations, which makes it particularly valuable for real-time applications.

A critical challenge in 3DGS lies in its initialization process, which heavily depends on the quality of initial camera poses and point clouds. These inputs are typically provided by SfM methods such as COLMAP [2], which rely on extensive Bundle Adjustment optimization to refine camera poses. However, traditional BA focuses solely on maximizing pose accuracy without explicitly accounting for the quality of the reconstructed map. Consequently, while BA-refined poses are geometrically precise, they do not necessarily yield optimal Gaussian maps when integrated into a 3DGS system. This concern is not only pertinent to 3DGS-based SLAM but also to the broader realm of 3DGS optimization. Therefore, achieving a balance between pose precision and map reconstruction quality represents a critical and valuable research direction for enhancing both the fidelity of the reconstruction and the visual quality of rendered images.

In this article, we introduce an extension to the conventional BA optimization process, termed LGSBA, which is designed to enhance the quality of the Gaussian map produced by 3DGS. Specifically, our approach first employs traditional BA to obtain relatively accurate camera poses. Subsequently, LGSBA refines these poses using a sliding window that associates adjacent keyframes, and incorporates a loss function that evaluates the rendering quality of the Gaussian map. This additional optimization stage effectively balances the need for pose accuracy with the objective of high-quality map reconstruction.

Furthermore, we propose a tightly coupled online reconstruction framework based on ORB-SLAM, integrated with the Robot Operating System for real-time communication. In our system, ORB-SLAM is employed to pre-select keyframes and provide initial camera poses and point clouds in real time, thereby eliminating the initialization bottleneck in the 3DGS reconstruction process. The combination of our LGSBA optimization and the online ORB-SLAM framework consistently produces Gaussian maps that surpass those obtained via both traditional 3DGS optimization and the improved Scaffold-GS method, as evidenced by superior rendering quality metrics. This integrated approach not only accelerates the overall process but also yields high-quality 3DGS maps, demonstrating its effectiveness in both offline and real-time applications.

The key contributions of this article are summarized as follows:

• Worse Pose but Better 3DGS: Our findings reveal that the quality of initial camera poses and point cloud information significantly influences the subsequent 3DGS optimization process. Remarkably, our experiments demonstrate for the first time that higher precision in these initial inputs does not necessarily lead to better Gaussian maps. In some cases, particularly in outdoor scenes, overly accurate poses that strictly conform to the physical world may paradoxically degrade the quality of the Gaussian map.
• ORB-based Initialization for 3DGS: We propose an ORB-based approach for initializing 3DGS that utilizes camera poses from keyframes and aggregates point clouds from all keyframes. This method not only enables real-time processing and supports large-scale image sequences, but also ensures that the Gaussian map is built upon high-quality initial inputs.
• Sliding Window for Local Gaussian Splatting Bundle Adjustment: We introduce a sliding window technique during the 3DGS training process to optimize the poses of viewpoints within a localized window centered on the current viewpoint. In this approach, the Gaussian map is first refined to enhance its local structure, followed by an optimization of all poses in the window by minimizing the discrepancy between rendered and captured images. This dynamic process substantially improves map quality and rendering fidelity.
• Tightly Coupled Online 3DGS Optimization: We further propose a tightly coupled 3DGS optimization system that integrates with ORB-SLAM’s various threads to acquire real-time updates of point clouds and camera poses. Leveraging ROS for seamless communication, this framework continuously refines the 3DGS map online. Our practical experiments demonstrate that this integrated approach not only accelerates the overall mapping process but also significantly enhances the quality of the final Gaussian maps.
• Open-Source Codebase: To support further research and development in this field, we release our code and experimental results at https://github.com/wla-98/worse-pose-but-better-3DGS.

2. Related Work

2.1. 3D Gaussian Splatting

Benefiting from its explicit representation, short training time, and real-time rendering speed, 3DGS [1] has quickly outpaced Neural Radiance Fields [3] to become a prominent research focus in the field of 3D reconstruction. Numerous studies aim to enhance the pixel-level scene quality generated by 3DGS, which remains a key objective of 3DGS optimization.

Some studies [4,5] focus on optimizing the original parameters of 3DGS. However, parameter adjustments often require extensive experimentation and are closely tied to the specific characteristics of a given scene, limiting their generalizability. Other studies [6,7,8] improve rendering quality by incorporating depth estimation information, but acquiring depth data involves additional resource costs.

Despite these advancements, limited research has specifically explored how the initialization of camera poses and sparse point clouds impacts subsequent 3DGS results. Furthermore, most 3DGS reconstruction rely on COLMAP [2] pipeline, which consumes substantial computational resources when processing large image sets, significantly reducing the efficiency of scene reconstruction. Addressing this limitation is a key objective of this article.

2.2. Visual SLAM

Direct SLAM methods [9,10,11,12] excel in handling low-texture and dynamic environments. However, due to their assumption of consistent grayscale intensity, these methods are highly sensitive to illumination changes and noise, which makes them less suitable for real-world applications. Feature-based SLAM [13,14,15,16,17,18] is highly robust, delivering excellent real-time performance and supporting multi-task parallel processing to construct high-precision maps. These systems integrate seamlessly with data from other sensors, providing accurate and reliable solutions for localization and mapping. However, they produce sparse point cloud maps, which hinder downstream tasks and demand considerable time for parameter tuning.

NeRF-based SLAM [3,19,20,21,22,23,24] excels in 3D dense modeling, rendering novel viewpoints, and predicting unknown regions, significantly enhancing 3D model generation and processing under optimal observational inputs. Nevertheless, these methods depend on large amounts of data and computational resources for neural network training, making real-time performance difficult to achieve. Furthermore, their implicit representations are less suited for downstream tasks.

Over the past two years, numerous studies have proposed SLAM systems based on 3DGS with explicit volumetric representations. Most of these systems [25,26,27,28,29,30,31,32] rely on 3D reconstruction frameworks, such as ORBSLAM [14,15], DROID-SLAM [33] or DUSt3R [34]to provide camera poses and initial point clouds for initializing the Gaussian map. Alternatively, they require depth information corresponding to RGB images as a prior to constrain the map reconstruction process. MonoGS [35] stands out by eliminating dependency on other SLAM systems for initialization. It unifies 3D Gaussian representations as the sole three-dimensional model while simultaneously optimizing the Gaussian map, rendering novel viewpoints, and refining camera poses.

However, all such systems face a critical limitation: achieving both higher-precision camera pose optimization and dense map reconstruction using only a monocular camera is challenging. This article clarifies the underlying reasons behind these limitations and proposes a novel approach that combines ORBSLAM initialization with LGSBA for pose optimization. This integration enhances the 3DGS framework, enabling it to achieve better results in terms of Gaussian map and rendered images across various scenarios.

3. Method

3.1. ORBSLAM Initialization

The ORBSLAM initialization process selects a series of keyframes $\{K_i\mid i=1,\dots ,N_p\}$ from a given image sequence $\{I_i\mid i\in {\mathbb{N}}^{+}\}$. The selection criteria for each keyframe $K_i$ are based on the system’s current tracking quality, the degree of scene change, and the distribution and update of map points. Camera poses and map points are optimized through extensive local bundle adjustment and global bundle adjustment operations. This process can be mathematically expressed and simplified as follows:

$$\{T_{K_i}^{*},P_j^{*}\}=\underset{T_{K_i},P_j}{\arg \min }\sum _{j=1}^{N_i}\sum _{i=1}^{N_k}\rho \left(\right||\pi \left(T_{K_i}{P}_j\right)-f_{i,j}{\left|\right|}^2),$$

(1)

where $N_i$ denotes the number of feature points used for optimization within the keyframe $K_i$, $N_k$ represents the number of keyframes in the current window (may including all images), $\rho$ is the robust Huber cost function, employed to mitigate the influence of outliers, $\pi \left(·\right)$ refers to the projection operation that maps points from 3D space to the 2D image plane, $f_{i,j}$ is the observed pixel coordinate of map point $P_j=(x_j,y_j,z_j)$ in keyframe $K_i$ and the camera poses $T_{K_i}$ is represented in the following matrix form:

$$T_{k_i}=\left[\begin{array}{cc}{R}_{k_i}& t_{k_i}\\ \mathbf{0}& 1\end{array}\right],$$

(2)

where $R_{k_i}\in SO\left(3\right).$ is the rotation matrix, and $t_{k_i}\in {\mathbb{R}}^3$ is the translation vector.

We further extend the 3D map points $P_j$ to $P_j^{exp}=(x_j,y_j,z_j,r_j,g_j,b_j)\in {\mathbb{R}}^6$ by incorporating the RGB information from the image, aligning the 3D map points with their corresponding 2D image points. Along with all optimized keyframe pose information $\{T_i^{*}\mid i=1,\dots ,N_p\}$ and the RGB images associated with the keyframes, this information is jointly output.

3.2. Local Gaussian Splatting Bundle Adjustment

3DGS provides an explicit and compact representation of 3D structures. Each Gaussian retains the spatial coordinates $\mu =(x,y,z)$ derived from the initialization point cloud and a covariance matrix $\Sigma$, defined as:

$$G\left(\mathbf{x}\right)=e^{-{\displaystyle \frac{1}{2}}{(\mathbf{x}-\mu )}^T{\Sigma }^{-1}(\mathbf{x}-\mu )},$$

(3)

while encoding color information through spherical harmonics (SH) coefficients $\mathbf{c}$ obtained from the image’s RGB data. In addition, opacity $\alpha$ is also encoded. To ensure the positive definiteness of $\Sigma$, it is parameterized using a rotation matrix R and a scaling matrix S, i.e.,

$$\Sigma =RS{S}^T{R}^T.$$

(4)

Thus, a single 3D Gaussian is defined as $G(\mu ,\mathbf{c},\alpha ,R,S)$, where $\mu$ represents the 3D mean position, $\mathbf{c}$ the SH color coefficients, $\alpha$ the opacity, R the rotation matrix, and S the scaling matrix.

After system initialization, given the set of all 3D points $\{P_{K_i}\mid i=1,\dots ,N_p\}$ observable by a keyframe $K_i$ with pose $T_{K_i}^{*}$, the set of 3D Gaussians initialized from keyframe $K_i$ is defined as:

$${\mathcal{G}}_{K_i}=\{G(P_j,{\mathbf{c}}_j,{\alpha }_j,R_j,S_j)\mid j=1,\dots ,N_i\}.$$

(5)

During the 3DGS training process, a subset of these Gaussians is randomly selected for optimization. Using the keyframe poses obtained from the ORB-SLAM initialization, all 3D Gaussians observable from the selected keyframe $K_i$ are projected onto the camera’s 2D plane. The projected 2D position $P_j^{2D}$ of a 3D Gaussian $P_j$ is computed using a pose transformation $T_{K_i}$. However, calculating the projected covariance ${\Sigma }_j^{2D}$ requires an affine approximation of the projective transformation [36], expressed as:

$${\Sigma }_j^{2D}=J_{K_i}{R}_{K_i}\Sigma R_{K_i}^T{J}_{K_i}^T,$$

(6)

where $R_{K_i}$ denotes the rotational component of the keyframe pose $T_{K_i}^{*}$ and $J_{K_i}$ is the Jacobian corresponding to the affine approximation. This substitution, however, introduces errors—errors that are especially pronounced and uneven in outdoor scenes. Moreover, highly accurate physical-world poses can sometimes result in a poorer Gaussian map, thereby degrading the optimization quality.

The projected position $P_j^{2D}$ and covariance ${\Sigma }_j^{2D}$ of each 3D Gaussian determine its contribution to the rendered image. For any pixel ${\mathbf{C}}_{K_i}^k$ on the keyframe image plane, all 3D Gaussians that are projected onto that pixel are sorted based on their distance from the camera and indexed as $1,\dots ,M$. These Gaussians form a set

$$\{{\mathcal{G}}_{K_i}^m(P_j,{\mathbf{c}}_j,{\alpha }_j,R_j,S_j)\mid 1\le m\le M\}.$$

Using the color and opacity information of these Gaussians, the final pixel color ${\mathbf{C}}_{K_i}^k$ is computed via $\alpha$-blending:

$${\mathbf{C}}_{K_i}^k=\sum _{m=1}^M{\mathbf{c}}_{K_i}^m{\alpha }_{K_i}^m\prod _{n=1}^{m-1}\left(1-{\alpha }_{K_i}^n\right),$$

(7)

which is performed for every pixel to render the image $I_{K_i}^{pred}$ from the keyframe’s perspective. Subsequently, a loss function is used to measure the difference between the rendered image $I_{K_i}^{pred}$ and the ground truth image $I_{K_i}^{true}$.

3.2.1. Improved Loss Function

In the original formulation, the loss for a single keyframe was defined as:

$$L=\lambda L_1+(1-\lambda )SSIM,$$

(8)

where $L_1$ is the L1 loss, $SSIM$ denotes the Structural Similarity Index Measure, and $\lambda$ is a weighting coefficient.

With the introduction of LGSBA, the loss function is modified to account for all keyframes within a sliding window. For example, if the sliding window contains the current keyframe and 3 frames before and after (a total of $N=7$ frames), the overall loss is:

$$L_{\mathrm{total}}=\sum _{i=1}^N{L}_i,$$

(9)

and for each keyframe i within the window, the loss is given by:

$$L_{\mathrm{window}}=\sum _{i\in W}\left(\lambda L_1^{\left(i\right)}+(1-\lambda )SSI{M}^{\left(i\right)}\right),$$

(10)

where W represents the index set of keyframes in the sliding window.

3.2.2. Improved Gradient Optimization

During incremental training, both the camera pose parameters and the parameters of all 3D Gaussians need to be optimized. In the original gradient derivation, the partial derivative of the 2D projection with respect to a keyframe pose $T_{K_l}$ was given by:

$${\displaystyle \frac{\partial P_j^{2D}}{\partial T_{K_l}}}={\displaystyle \frac{\partial P_j^{2D}}{\partial P_j}}{\displaystyle \frac{\mathcal{D}{P}_j}{\mathcal{D}{T}_{K_l}}},$$

(11)

and similarly, for the covariance:

$${\displaystyle \frac{\partial {\Sigma }_j^{2D}}{\partial T_{K_l}}}={\displaystyle \frac{\partial {\Sigma }_j^{2D}}{\partial J_{K_l}}}{\displaystyle \frac{\partial J_{K_l}}{\partial P_j}}{\displaystyle \frac{\mathcal{D}{P}_j}{\mathcal{D}{T}_{K_l}}}.$$

(12)

In the improved formulation with a sliding window, the gradient of the total loss with respect to the pose $T_{K_l}$ becomes:

$${\displaystyle \frac{\partial L_{\mathrm{total}}}{\partial T_{K_l}}}=\sum _{i\in W}\left(\lambda {\displaystyle \frac{\partial L_1^{\left(i\right)}}{\partial T_{K_l}}}+(1-\lambda ){\displaystyle \frac{\partial SSI{M}^{\left(i\right)}}{\partial T_{K_l}}}\right).$$

(13)

For each keyframe i, the gradient components are computed as:

$${\displaystyle \frac{\partial L_1^{\left(i\right)}}{\partial T_{K_l}}}={\displaystyle \frac{\partial P_j^{2D\left(i\right)}}{\partial T_{K_l}}},$$

(14)

$${\displaystyle \frac{\partial SSI{M}^{\left(i\right)}}{\partial T_{K_l}}}={\displaystyle \frac{\partial {\Sigma }_j^{2D\left(i\right)}}{\partial T_{K_l}}}.$$

(15)

This extended gradient formulation ensures that the optimization process within the sliding window jointly refines the camera poses by considering both pixel-level differences and structural similarities across all relevant keyframes.

3.2.3. Overall Optimization Framework

At the commencement of 3DGS training, keyframe poses are first updated through traditional BA to obtain relatively accurate estimates. Following this, the LGSBA module takes over, applying the sliding window-based loss and corresponding gradient optimization to further refine these poses with the objective of enhancing the Gaussian map quality. Specifically, the rendered images are compared against ground truth images for each keyframe in the sliding window, and the resulting losses are aggregated and minimized.

The refined poses, in turn, contribute to a more accurate projection of the Gaussians onto the 2D image plane, thereby improving both the local structure of the Gaussian map and the overall rendering fidelity. The gradients are computed following the chain rule on the Lie group $SE\left(3\right)$, where for a keyframe pose $T\in SE\left(3\right)$ and its corresponding Lie algebra element $\tau \in se\left(3\right)$, the left Jacobian is defined as:

$${\displaystyle \frac{Df\left(T\right)}{DT}}\triangleq \underset{\tau \to 0}{lim}{\displaystyle \frac{log\left(f\left(exp\left(\tau \right)\circ T\right)\circ f{\left(T\right)}^{-1}\right)}{\tau }}.$$

(16)

Based on this, the following derivatives are obtained:

$${\displaystyle \frac{\mathcal{D}{P}_j}{\mathcal{D}{T}_{K_i}}}=\left[\begin{array}{cc}I& -P_j^{\times }\end{array}\right],{\displaystyle \frac{\mathcal{D}{R}_{K_i}}{\mathcal{D}{T}_{K_i}}}=\left[\begin{array}{cc}\mathbf{0}& -{R_{K_i}}_{:,1}^{\times }\\ \mathbf{0}& -{R_{K_i}}_{:,2}^{\times }\\ \mathbf{0}& -{R_{K_i}}_{:,3}^{\times }\end{array}\right],$$

(17)

where × denotes the skew-symmetric matrix of a 3D vector and ${R_{K_i}}_{:,k}$ represents the k-th column of $R_{K_i}$.

In summary, the proposed LGSBA enhances the conventional BA process by introducing a sliding window that considers multiple keyframes simultaneously, and by incorporating a rendering quality-based loss function. This dual-stage optimization effectively balances the refinement of camera poses with the improvement of the Gaussian map, leading to higher-quality 3DGS reconstructions. Practical experiments demonstrate that the proposed approach not only improves the rendered image quality but also achieves superior performance compared to traditional 3DGS optimization pipelines.

4. Experiments

All experiments were conducted on a Windows 11 WSL2 system, equipped with an Intel Core i7-11700K CPU and an NVIDIA RTX 3090 GPU.

4.1. Dataset Description

We evaluated our approach on three publicly available datasets: TUM-RGBD, Tanks and Temples, and KITTI. These datasets offer complementary advantages in terms of scene types, motion patterns, and evaluation metrics, which enables a comprehensive assessment of the proposed method under varied conditions.

4.1.1. TUM-RGBD Dataset (TUM)

The TUM-RGBD dataset is widely used for indoor SLAM research. It contains multiple sequences of both dynamic and static indoor scenes captured using a handheld RGB-D camera. Ground-truth trajectories with high precision are provided, and common evaluation metrics include Absolute Trajectory Error (ATE) and Relative Pose Error (RPE). This dataset is ideal for assessing the stability of feature tracking in the presence of dynamic disturbances and the precision of backend optimization.

4.1.2. Tanks and Temples Dataset (TANKS)

The Tanks and Temples dataset is designed for large-scale outdoor 3D reconstruction tasks. It features high-resolution scenes with complex geometry and detailed textures. This dataset is frequently used to evaluate multi-view 3D reconstruction and dense scene modeling methods, allowing us to assess the capability of our 3DGS approach in modeling radiance fields at different scales.

4.1.3. KITTI Dataset (KITTI)

The KITTI dataset is collected from a vehicle-mounted platform and is primarily targeted at SLAM and visual odometry research in real-world traffic scenarios. It provides continuous image sequences along with high-precision GPS/IMU data, making it suitable for evaluating the system’s ability to suppress accumulated errors over long-term operations and its robustness in challenging road environments.

The complementary nature of these datasets—in terms of indoor/outdoor settings, different motion modalities (handheld, vehicle, aerial), and diverse evaluation metrics (ATE/RPE)—enables a systematic evaluation of:

• The robustness of the SLAM front-end under dynamic disturbances.
• The radiance field modeling capabilities of 3DGS in scenes of varying scales.
• The suppression of cumulative errors during long-term operations.

4.2. Evaluation Metrics

To quantitatively analyze the reconstruction results, we adopt several evaluation metrics that measure the differences between the reconstructed images and ground truth, the accuracy of camera poses, and the geometric precision. The metrics used are as follows:

• L1 Loss: This metric evaluates image quality by computing the pixel-wise absolute difference between the reconstructed image and the ground truth image. A lower L1 loss indicates a smaller discrepancy and, consequently, better reconstruction quality:

  $$L_1={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left|I_{\mathrm{reconstructed}}\left(i\right)-I_{\mathrm{gt}}\left(i\right)\right|,$$

  (18)

  where N is the total number of pixels.
• PSNR: PSNR measures the signal-to-noise ratio between the reconstructed and ground truth images, expressed in decibels (dB). A higher PSNR value indicates better image quality. It is computed as:

  $$\mathrm{PSNR}=10{log}_{10}\left({\displaystyle \frac{R^2}{MSE}}\right),$$

  (19)

  with R being the maximum possible pixel value (typically 255 for 8-bit images) and $MSE$ defined as:

  $$MSE={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(I_{\mathrm{reconstructed}}\left(i\right)-I_{\mathrm{gt}}\left(i\right)\right)}^2.$$

  (20)
• SSIM: SSIM measures the structural similarity between two images by considering luminance, contrast, and texture. Its value ranges from -1 to 1, with values closer to 1 indicating higher similarity:

  $$SSIM(x,y)={\displaystyle \frac{(2{\mu }_x{\mu }_y+C_1)(2{\sigma }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_1)({\sigma }_x^2+{\sigma }_y^2+C_2)}},$$

  (21)

  where ${\mu }_x,{\mu }_y$ are the means, ${\sigma }_x^2,{\sigma }_y^2$ the variances, and ${\sigma }_{xy}$ the covariance of images x and y, with $C_1$ and $C_2$ being small constants.
• LPIPS: LPIPS assesses perceptual similarity between images using deep network features. It is computed as:

  $$\mathrm{LPIPS}(I_{\mathrm{reconstructed}},I_{\mathrm{gt}})={\displaystyle \frac{1}{K}}\sum _{k=1}^K{∥{\varphi }_k\left(I_{\mathrm{reconstructed}}\right)-{\varphi }_k\left(I_{\mathrm{gt}}\right)∥}_2,$$

  (22)

  where ${\varphi }_k\left(·\right)$ represents the feature map extracted from the k-th layer of a deep convolutional network, and K is the total number of layers. A lower LPIPS indicates a smaller perceptual difference.
• APE: APE quantifies the geometric precision of the reconstructed camera poses by computing the Euclidean distance between the translational components of the estimated and ground truth poses. For N poses, it is defined as:

  $$APE={\displaystyle \frac{1}{N}}\sum _{i=1}^N∥{\mathbf{t}}_{\mathrm{reconstructed}}\left(i\right)-{\mathbf{t}}_{\mathrm{gt}}\left(i\right)∥.$$

  (23)
  If rotation errors are also considered, APE can be extended as:

  $$AP{E}_{\mathbf{SE}\left(3\right)}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left(\parallel {\mathbf{t}}_i^{\mathrm{err}}\parallel +\beta \parallel {\mathbf{r}}_i^{\mathrm{err}}\parallel \right),$$

  (24)

  where ${\mathbf{r}}_i^{\mathrm{err}}$ is the axis-angle error for rotation, and $\beta$ is a weighting factor (typically set to 1).
• RMSE: RMSE measures the average error between the reconstructed and ground truth camera poses, computed as:

  $$RMSE=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{∥{\mathbf{T}}_{\mathrm{reconstructed}}\left(i\right)-{\mathbf{T}}_{\mathrm{gt}}\left(i\right)∥}^2},$$

  (25)

  where ${\mathbf{T}}_{\mathrm{reconstructed}}\left(i\right)$ and ${\mathbf{T}}_{\mathrm{gt}}\left(i\right)$ denote the reconstructed and ground truth poses for frame i, respectively.

These metrics provide a comprehensive evaluation of image reconstruction quality, visual fidelity, and geometric accuracy. In our experiments, we use these metrics to quantitatively compare different methods and provide a solid basis for performance evaluation.

4.3. Local Gaussian Splatting Bundle Adjustment

Our experiments aim to validate two key improvements proposed in this subsection: (1) a real-time initialization method based on ORB-SLAM; and (2) the effectiveness of the Local Gaussian Splatting Bundle Adjustment during the 3DGS training process. To achieve this, we designed three groups of experiments to evaluate various aspects of the system using the aforementioned evaluation metrics.

Figure 1. System Overview: The system starts with a large number of RGB frames and integrates two key processes: ORB-SLAM initialization and LGSBA, showing outstanding innovation. In the ORB-SLAM initialization stage, by using the advanced ORB-SLAM algorithm and comprehensively referring to multi-dimensional indicators such as scene change, feature point distribution and tracking quality, a small number of representative keyframes are accurately selected from a large number of RGB frames. This effectively simplifies the input system and reduces data redundancy. Subsequently, based on these selected keyframes, a 3D point cloud is constructed through precise feature matching and triangulation, and the poses of keyframes are accurately calculated to provide precise initial parameters for the subsequent 3DGS processing, greatly saving the initialization time. After entering the 3DGS training, it seamlessly takes over the results of the ORB-SLAM initialization and uses the 3DGS technology to discretize the scene into a Gaussian distribution form to build an initial 3D model. Particularly prominent is the innovative introduction of LGSBA process. This process utilizes the sliding window mechanism, focuses on the viewpoint optimization within the local window with the current viewpoint as the core, comprehensively considers indicators such as pixel-level differences and structural similarity, and iteratively adjusts the Gaussian parameters and poses through the loss function, significantly enhancing the rendering quality. Eventually, a high-precision 3DGS map and intuitive rendered RGB images are generated, bringing new breakthroughs to the 3DGS field.

Figure 1. System Overview: The system starts with a large number of RGB frames and integrates two key processes: ORB-SLAM initialization and LGSBA, showing outstanding innovation. In the ORB-SLAM initialization stage, by using the advanced ORB-SLAM algorithm and comprehensively referring to multi-dimensional indicators such as scene change, feature point distribution and tracking quality, a small number of representative keyframes are accurately selected from a large number of RGB frames. This effectively simplifies the input system and reduces data redundancy. Subsequently, based on these selected keyframes, a 3D point cloud is constructed through precise feature matching and triangulation, and the poses of keyframes are accurately calculated to provide precise initial parameters for the subsequent 3DGS processing, greatly saving the initialization time. After entering the 3DGS training, it seamlessly takes over the results of the ORB-SLAM initialization and uses the 3DGS technology to discretize the scene into a Gaussian distribution form to build an initial 3D model. Particularly prominent is the innovative introduction of LGSBA process. This process utilizes the sliding window mechanism, focuses on the viewpoint optimization within the local window with the current viewpoint as the core, comprehensively considers indicators such as pixel-level differences and structural similarity, and iteratively adjusts the Gaussian parameters and poses through the loss function, significantly enhancing the rendering quality. Eventually, a high-precision 3DGS map and intuitive rendered RGB images are generated, bringing new breakthroughs to the 3DGS field.

4.3.1. Experiment 1: Comparison of ORB-SLAM and COLMAP Initialization

This group of experiments focuses on the initialization stage. We compare the initialization performance of COLMAP and ORB-SLAM using the TUM-RGBD, Tanks and Temples, and KITTI datasets. The results, summarized in Table 1, show that COLMAP requires a significantly longer time for sparse reconstruction and camera pose estimation (e.g., approximately 40 minutes for around 200 images) compared to ORB-SLAM, which reduces the initialization time to under 3 minutes.

Moreover, to assess the impact of the initialization method on 3DGS map optimization, we rendered the Gaussian maps generated under identical keyframe selection and pose conditions. In indoor scenarios (e.g., TUM-fg2 desk and TUM-fg3 long office), COLMAP yielded slightly higher PSNR values (by 0.91 dB and 0.73 dB, respectively) compared to ORB-SLAM. However, in outdoor or large-scale scenes (e.g., TANKS train, KITTI 00, TANKS m60, and TUM-fg1 floor), ORB-SLAM initialization produced notably better rendering results, with PSNR improvements of 3.04 dB, 6.52 dB, 2.11 dB, and 9.24 dB, respectively. On average, ORB-SLAM initialization achieved approximately 0.016 lower L1 loss, 1.6 dB higher PSNR, 0.051 higher SSIM, and 0.075 lower LPIPS than COLMAP, demonstrating its superiority in both speed and the quality of the generated Gaussian maps.

Figure 2. Different maps obtained from dataset TUM-fg2-desk with COLMAP initialization method and original 3DGS optimization. (a) Sparse point cloud from traditional SfM based system. (b) Gaussian map from 3DGS optimization. (c) Rendered image from Gaussian map with a specific viewpoint.

Figure 2. Different maps obtained from dataset TUM-fg2-desk with COLMAP initialization method and original 3DGS optimization. (a) Sparse point cloud from traditional SfM based system. (b) Gaussian map from 3DGS optimization. (c) Rendered image from Gaussian map with a specific viewpoint.

4.3.2. Experiment 2: Comparison of Different 3DGS Training Methods

This experiment evaluates the robustness of the ORB-SLAM initialization and the performance gains achieved by incorporating the LGSBA optimization into the 3DGS training process. Starting from the same set of initial point clouds and camera poses provided by ORB-SLAM, we compare three training strategies: the original 3DGS training method, an advanced Scaffold-GS system, and our 3DGS training approach enhanced with LGSBA. The experiments were conducted on TUM-RGBD, Tanks and Temples, and KITTI datasets.

In the 3DGS map optimization phase, under identical keyframe and pose conditions, we rendered the Gaussian maps and evaluated them using PSNR, SSIM, LPIPS, and L1 loss. As detailed in Table 2, under 30,000 iterations, our LGSBA-enhanced method improved the PSNR by 2.17 dB, 0.29 dB, 10.4 dB, 4.38 dB, 5.73 dB, 1.82 dB, 2.15 dB, and 0.50 dB in various scenes (e.g., TUM-fg2 desk, TUM-fg3 long office, TUM-fg1 floor, Tanks Caterpillar, Tanks M60, Tanks Panther, Tanks Horse, and KITTI 00, respectively). Overall, compared to the original 3DGS training, the LGSBA method reduced the average L1 loss by approximately 0.012, increased PSNR by about 1.9 dB, improved SSIM by 0.068, and lowered LPIPS by 0.081. Similar improvements were observed when compared against the Scaffold-GS method. These results confirm that the high-quality initial point clouds and camera poses obtained via ORB-SLAM provide a solid foundation, and the LGSBA optimization significantly enhances the map quality and rendering fidelity, especially in outdoor and large-scale scenes.

Figure 3 showcases representative rendering results across multiple scenes using different initialization and training strategies, juxtaposed against the ground truth images. The first and second columns illustrate the outcomes of 3DGS trained with COLMAP and ORB-SLAM initialization, respectively, while the third and fourth columns display Scaffold-GS under the same initialization settings. In contrast, the fifth column presents our LGSBA-enhanced approach, and the last column shows the ground truth. Notably, our method exhibits more faithful reproduction of intricate geometric structures (e.g., the edges of machinery, detailed textures on the ground, and refined object boundaries) and more consistent color transitions. This superior fidelity is especially evident in outdoor or large-scale scenes (e.g., the excavator and sculpture models), where highly accurate camera poses and refined Gaussian parameters lead to sharper features and fewer rendering artifacts. Overall, these visual results corroborate the quantitative improvements reported in Table 2, underscoring the effectiveness of our LGSBA optimization in achieving high-quality 3DGS reconstructions.

4.3.3. Experiment 3: Evaluation of Camera Pose Accuracy and Rendering Details

This experiment assesses the impact of LGSBA optimization on camera pose accuracy and the rendering quality of the 3DGS map. We quantitatively evaluated the reconstructed camera poses using Absolute Pose Error (APE) and Root Mean Square Error (RMSE), and compared the optimized trajectories with the ground truth. For the TUM-RGBD dataset, high-precision ground-truth poses were obtained from a motion capture system; for the Tanks and Temples dataset, COLMAP-reconstructed trajectories served as the reference; and for the KITTI dataset, real poses were provided by a vehicle-mounted GPS-RTK system.

The results (see Table 3 and Figure 4) indicate that, in some indoor scenes (e.g., TUM-fg2 desk and TUM-fg3 long office), LGSBA slightly increased APE and RMSE compared to the poses estimated directly by ORB-SLAM. However, from the rendering perspective, LGSBA significantly improved the final image quality. For instance, in the TUM-fg2 desk scene, the PSNR improved from 25.40 dB to 26.37 dB, and in the TUM-fg3 long office scene, from 27.83 dB to 28.62 dB, highlighting that LGSBA effectively enhances rendering quality. Notably, in large outdoor scenes (e.g., KITTI and Tanks and Temples), the improvements were even more pronounced—with PSNR gains of up to 5.59 dB in the Tanks Caterpillar scene and 0.50 dB in the KITTI 00 scene.

In summary, although LGSBA may lead to a slight increase in physical pose error in certain scenarios, the overall enhancement in the rendered 3DGS map—reflected in improved visual quality and finer detail reconstruction—demonstrates its effectiveness. Figure 4 visually compares the optimized camera trajectories with the ground truth, providing further evidence of the rendering improvements brought about by LGSBA.

4.4. Online Integration of ORB-SLAM with 3DGS Map Optimization

4.4.1. System Overview:

In this experiment, we evaluate an online mapping system that tightly integrates ORB-SLAM with 3DGS optimization via our Local Gaussian Splatting Bundle Adjustment. As shown in Figure 5, the system leverages ORB-SLAM to perform real-time keyframe-based pose estimation and sparse point cloud generation, while the GSMaster node performs incremental optimization of a 3D Gaussian map. This online framework is implemented using the ROS distributed computing architecture, where separate nodes concurrently handle image preprocessing, SLAM processing, and 3DGS optimization.

4.4.2. Experimental Procedure:

The experiment is structured around the following key components:

• Real-Time Initialization and Data Flow: The system ingests preprocessed image streams (e.g., from a smartphone) published on the /camera/rgb topic. ORB-SLAM processes these images in real time by extracting features, estimating camera poses, and selecting keyframes based on tracking quality, parallax, and entropy thresholds. The resulting keyframes, poses, and sparse point clouds are published on dedicated topics (/slam/pose, /slam/pointcloud, and /slam/rgb) for subsequent 3DGS training.
• Incremental 3DGS Map Optimization using LGSBA: The GSMaster node subscribes to the SLAM outputs and initializes the 3DGS module by converting sparse points into Gaussian distributions, parameterized by mean $\mu$, covariance $\Sigma$, color c, and opacity $\alpha$. Before loop closure is detected, an incremental training mode is activated. In this mode, LGSBA refines both camera poses and Gaussian parameters using a sliding window that jointly minimizes a loss function—comprising L1 loss and $(1-SSIM)$ error—across multiple keyframes. This dual-stage optimization balances pose accuracy with the rendered Gaussian map quality, enabling updated map renderings at 5–10 frames per second.
• Dynamic Training Mode Switching: When ORB-SLAM detects loop closure (indicated via the /slam/loop topic), the system automatically transitions from incremental to a global (random) training mode. In this phase, only the global 3D Gaussian map is optimized while camera poses remain fixed. This switch ensures global consistency and further refines rendering details by applying operations such as voxel downsampling and the removal of low-opacity Gaussians.
• Experimental Results: In a controlled laboratory environment, the online system was tested using an 80-second video captured at 1024×768 resolution and 30 FPS (downsampled to 15 FPS for processing). Out of approximately 1200 frames, ORB-SLAM selected 174 keyframes. Compared to an offline reconstruction pipeline, the online method dramatically reduced initialization time (from over 60 minutes of manual keyframe selection to under 3 minutes) while improving map quality. After 30,000 optimization iterations, the online method achieved average improvements of approximately +0.49 dB in PSNR, +0.013 in SSIM, and -0.042 in LPIPS over the offline baseline, all while maintaining real-time performance. Detailed experimental processes and visual results are shown in Figure 6, Figure 7, and Figure 8, with quantitative results summarized in Table 4.

4.4.3. Experimental Results Comparison:

To evaluate the performance of our proposed online 3DGS optimization method, we conduct a quantitative comparison with the offline reconstruction approach that utilizes ORB-SLAM for initialization, followed by 3DGS optimization. As shown in Table 4, our method achieves lower L1 error and LPIPS at 7K iterations, indicating improved perceptual quality. At 30K iterations, our approach maintains comparable L1 error while achieving slightly better PSNR and SSIM, demonstrating its effectiveness.

Unlike the quantitative analysis, we do not provide a direct qualitative comparison with the offline method. Instead, Figure 7 and Figure 8 present only the reconstruction results of our online approach. The Gaussian maps in Figure 7 illustrate the optimized 3D representation, while Figure 8 shows rendered images from multiple viewpoints, highlighting the structural and texture fidelity of our method.

However, since our system relies on ORB-SLAM for initialization, its robustness is affected in dynamic and low-texture environments. In such scenarios, inaccurate camera tracking degrades the reconstruction quality, which remains a limitation of our approach.

This experiment demonstrates that the online integration of ORB-SLAM with 3DGS optimization and LGSBA significantly enhances both the efficiency and quality of the mapping process. The system not only accelerates initialization and keyframe management but also produces high-quality 3D Gaussian maps suitable for real-time robotic applications.

5. Conclusion

In this article, we have made improvements to the original 3DGS system mainly in three aspects, including ORB-SLAM initialization, the introduction of LGSBA training, and the construction of a tightly coupled ORB-SLAM and 3DGS system. Eventually, we have managed to shorten the construction time by 10 times while improving the PSNR of the 3DGS map rendering. Based on our experiments, we have also discovered an interesting phenomenon: more accurate camera poses and point clouds in the physical world do not necessarily lead to the optimization of a 3DGS map of better quality, which is the core reason for our introduction of the LGSBA algorithm. We have also open-sourced the relevant source code of our work and some experimental results, which can be found at https://github.com/wla-98/worse-pose-but-better-3DGS.

However, our research lacks a more in-depth exploration of the better quality of camera poses and the optimization quality of 3DGS maps, which will be the core part of our future work.