1. Introduction
Robots have become ubiquitous in various sectors, including manufacturing, agriculture, service industries, and autonomous driving [Reference Cheein8]. Robots equipped with advanced sensors can detect motion and efficiently map unknown environments using simultaneous localization and mapping (SLAM) technology [Reference Lee, Lee, Choi and Lee23, Reference Li, Gu, Li, Li, Guo, Gao, Zhao, Yang, Li and Dong25, Reference Wang, Zhang, Song, Pang and Zhang42]. In this study, we utilize a visual SLAM (VSLAM) system based on images from monocular, stereo, and RGB-D cameras [Reference Yu, Zheng and Ou51]. The camera captures successive image frames, providing a rich tapestry of visual information. These data facilitate the extraction of feature points in the environment and enable the estimation of motion and position by computing geometric relationships between adjacent frames. This process relies on the extraction of feature points from images and the subsequent matching of these points to compute relative motion.
In exploring SLAM technology, researchers have employed a variety of strategies to overcome the inherent challenges. In essence, a SLAM system is a state estimation problem. In the construction of traditional SLAM systems, real-time collection of information such as distance, image, and motion is achieved by extracting and processing sensor data. In vision-based SLAM systems, feature point extraction is performed by analyzing successive images using techniques such as ORB [Reference Campos, Elvira, Rodríguez, Montiel and Tardós7, Reference Mur-Artal and Tardós28], SIFT [Reference Wang, Zhao, Gong, Chen and Zuo45], and the optical flow [Reference Qin, Li and Shen30, Reference Teed and Deng40, Reference Teed and Deng41] method. This enables motion estimation by comparing feature changes between successive images. Traditional SLAM relies on these hand-designed methods for feature extraction, but these methods perform poorly in environments with large lighting variations and feature-poor environments. Additionally, traditional SLAM systems typically presuppose a static environment, exhibiting constrained capabilities in managing dynamic scenes. With enhanced computational capacities, contemporary SLAM systems increasingly diverge from traditional probability-based estimation methods, favoring advanced techniques such as least squares methods and Bundle Adjustment (BA) to optimize the 3D structure and camera trajectories across the map while reducing reliance on filtering methods like Kalman and particle filtering for managing data uncertainty. Although technological advances have made SLAM systems more stable and reliable, they still face the challenge of lack of robustness in practical applications and may suffer from problems including but not limited to feature point trajectory loss, as well as cumulative positioning drift over time.
Deep learning, a pivotal subfield of machine learning, addresses complex challenges by emulating neural networks akin to those found in the human brain. Recently, the application of deep learning in SLAM systems, exemplified by technologies such as SfM-Learner [Reference Zhou, Brown, Snavely and Lowe54] and CNN-SVO [Reference Loo, Amiri, Mashohor, Tang and Zhang26], has become increasingly prevalent. VSLAM systems that leverage deep learning exhibit robust generalization capabilities, enabling effective operation in complex environmental scenarios [Reference Zhang, Dong, Guo, Ye, Gao, Xiang, Chen and Wu52]. Deep learning is typically employed across multiple sessions to address issues such as feature point trajectory loss, divergence in optimization algorithms, and the accumulation of drift. Although SLAM algorithms that integrate deep learning mitigate many of these issues, these systems, despite their enhanced robustness, often exhibit less precision than classical systems on standard benchmarks.
Developed in 2021 by Zachary Teed and Jia Deng at Princeton University, the deep learning-based visual SLAM system, DROID-SLAM [Reference Teed and Deng41], has garnered widespread acclaim since its unveiling. DROID-SLAM is closer to the DeepV2D [Reference Teed and Deng39] and BA-Net [Reference Tang and Tan37] architectures. DROID-SLAM is based on an advanced optical flow estimation method, RAFT [Reference Teed and Deng40], which mainly improves optical flow computation by using deep learning techniques. DROID-SLAM utilizes the advanced RAFT optical flow estimation method, significantly enhancing optical flow computations through the application of deep learning techniques. The RAFT network comprises a feature coding layer that extracts features pixel-by-pixel, a feature correlation layer producing 4D correlation data for all pixels, and a cyclic updating algorithm utilizing GRUs to iteratively refine the optical flow field by leveraging this 4D correlation information. Leveraging RAFT, DROID-SLAM incorporates an iterative update mechanism that persistently refines the camera pose and scene depth information. Another pivotal innovation in DROID-SLAM is the integration of the Dense Bundle Adjustment (DBA) layer. DROID-SLAM is highly accurate and more robust, with a much lower incidence of catastrophic failures. Although primarily trained on monocular video, DROID-SLAM can also utilize stereo or RGB-D video to enhance its performance during testing phases. While DROID-SLAM is an excellent system, in many deep learning-driven vision SLAM systems, including DROID-SLAM, the model’s ability to generalize is affected by its training data, and these systems can suffer from degraded performance when confronted with environments that are significantly different from the training set. The efficacy of the features learned by the model during training is intimately tied to the diversity and representativeness of the training data. If the scenarios or objects in the test environment are not adequately represented in the training set, the model may not be able to understand and interpret the features of these new scenarios properly.
In this research, by enhancing the deep learning-based SLAM system, DROID-SLAM, we present the following contributions aimed at augmenting the system’s generalization capabilities:
-
• We employ the Exponential Moving Average (EMA) technique, an averaging methodology that assigns greater weight to more recent data points. In the realm of deep learning, EMA is utilized to smooth model parameters; in our research, we have integrated EMA into the DROID-SLAM system to enhance both the model’s generalization capabilities and the stability of the training process.
-
• We implement the Bayes update online learning method to integrate it seamlessly into the DROID-SLAM system. Within the deep learning framework, the Bayes update online learning method is primarily employed to perpetually adjust and refine model parameters, thereby allowing the system to adapt dynamically to new information in the data stream.
2. Related work
Classical SLAM systems have undergone extensive exploration and enhancement since their establishment as a foundational concept; consequently, a comprehensive review is beyond the scope of this study. Instead, we focus on recent advancements in three primary categories: Deep Learning VSLAM, Exponential Moving Average (EMA), and Bayes online learning.
Deep Learning VSLAM—In 2007, Davison et al. pioneered MonoSLAM [Reference Davison, Reid, Molton and Stasse11], a groundbreaking method for simultaneous localization and map construction using a monocular camera. This seminal solution established a robust foundation for further research in the field. Concurrently, Klein and Murray introduced a VSLAM solution [Reference Klein and Murray22] that integrated a keyframe mechanism, marking another significant advancement. This method distinctly delineated the front-end and back-end of the system, significantly enhancing system accuracy and robustness. Despite the relative limitations in application scenarios and occasional tracking losses, the foundational concepts and key technologies of these early VSLAM solutions continue to influence contemporary SLAM systems significantly. These pioneering ideas have not only galvanized subsequent research but also propelled the expansion of VSLAM technology into a wider array of application domains.
Traditional SLAM systems are categorized into two principal schools of thought: the feature point method [Reference Campos, Elvira, Rodríguez, Montiel and Tardós7, Reference Mur-Artal and Tardós28, Reference Qin, Li and Shen30] and the direct method [Reference Di Giammarino, Brizi, Guadagnino, Stachniss and Grisetti13]. The feature point method concentrates on precisely extracting and tracking uniformly distributed corner points, along with their distinctive descriptors, from successive video frames. Owing to their invariant properties, these descriptors effectively facilitate the matching of key points across frames. Subsequently, the system can initially recover the camera’s motion orientation and accordingly construct the environmental map’s coordinate system. Furthermore, the system meticulously refines the camera pose and the map through an iterative optimization process designed to minimize reprojection errors, thereby enhancing the accuracy of both positioning and map construction. The direct method adopts an alternative method, directly estimating the camera’s position and orientation based on photometric errors, thus eliminating the need to extract corner points and their descriptors. This method offers computational efficiency and, as it does not depend on global feature descriptions, faces challenges in characterizing overall image properties and managing closed-loop detection. Particularly, the direct method has yet to discover a perfect solution for eliminating positional drift that accumulates over extended operations, limiting its application in high-precision or long-duration autonomous navigation tasks.
In recent years, the integration of computer vision with deep learning has significantly matured, leading to substantial enhancements in the practical performance of vision-related tasks such as instance classification [Reference Xiao, Li, Yan, Gao and Wang48], object detection [Reference Ahmad, Ma, Yahya, Ahmad, Nazir and u. Haq2], and behavior recognition [Reference Jaouedi, Boujnah and Bouhlel19]—improving accuracy, execution efficiency, and robustness. Consequently, scholars have investigated the incorporation of deep learning methods into SLAM systems, aiming to achieve enhanced accuracy and robustness in environmental perception and localization. Deep learning is commonly applied to various subcomponents within SLAM systems, including deep-learning-based visual odometry [Reference Deraz, Badawy, Elhosseini, Mostafa, Ali and El-Desouky12], closed-loop detection [Reference Zhong and Fang53], and semantic SLAM [Reference Xiao, Wang, Qiu, Rong and Zou47]. A significant portion of the research concentrates on end-to-end trained SLAM systems [Reference Johari, Carta and Fleuret20]; however, these methods do not constitute complete SLAM systems and primarily focus on small-scale reconstruction, processing only two to a dozen frames of data. These systems often lack essential features of contemporary SLAM systems, such as loop closure and global bundle adjustment, thereby limiting their capability for large-scale reconstruction. My research is grounded in the advanced deep learning SLAM system, DROID-SLAM [Reference Teed and Deng41], inspired by Dusmanu et al. [Reference Dusmanu, Schönberger and Pollefeys14], who developed a neural network within the SfM pipeline to enhance keypoint localization accuracy. The DROID-SLAM system executes feature extraction using RAFT [Reference Teed and Deng40], facilitated by a neural network that processes through the DBA layer. DROID-SLAM is highly accurate and represents a significant improvement over prior efforts, demonstrating robustness with a markedly reduced incidence of catastrophic failures.
Although initially trained on monocular video, DROID-SLAM can also utilize stereo or RGB-D video to enhance performance during testing. However, similar to many deep learning-based SLAM systems, DROID-SLAM faces challenges in generalization when encountering environments significantly different from its training set. Numerous studies have advanced modifications based on DROID-SLAM [Reference Rosinol, Leonard and Carlone31], and our research also builds upon this foundational algorithm.
Increase genenoralization capacity—The generalization capability of neural networks in deep learning refers to their adeptness at adapting to novel, previously unseen data. This capability serves as a measure of whether the knowledge acquired by the model during the learning and training phases can be effectively leveraged for new datasets. To enhance the generalization ability of models, researchers have employed techniques such as data augmentation [Reference Taylor and Nitschke38], regularization [Reference Yoshida and Miyato50], cross-validation [Reference Nie, De Santis, Carratù, O’Nils, Sommella and Lundgren29], early stopping [Reference Bai, Yang, Han, Yang, Li, Mao, Niu and Liu3], batch normalization [Reference Segu, Tonioni and Tombari33], and ensemble methods [Reference Ganaie, Hu, Malik, Tanveer and Suganthan17]. Our research draws inspiration from the seminal works of Zhaowei Cai et al. and Aodong Li et al.
Zhaowei Cai et al. [Reference Cai, Ravichandran, Maji, Fowlkes, Tu and Soatto6] proposed an innovative technique as an alternative to batch normalization (BN) [Reference Bjorck, Gomes, Selman and Weinberger4], termed Exponential Moving Average Normalization (EMAN). Unlike traditional batch normalization, where statistics are computed independently for each batch, EMAN leverages the BN statistics from the student model within the teacher model. This technique updates its statistics through an EMA, enhancing generalization by diminishing reliance on individual data batches, thereby rendering the model more stable and efficient in handling diverse datasets. Originally utilized in statistics and signal processing, primarily for smoothing time series data in economic and financial market analysis [Reference Gruevski and Gaber18], this concept is now gradually being integrated into deep learning methodologies. In our research, EMA performs smoothing updates within the intermediate layers of the network, thereby enhancing both the model’s performance and its generalization capability.
Aodong Li et al. [Reference Li, Boyd, Smyth and Mandt24] introduced an innovative Bayes online learning framework designed to address the challenges posed by non-static data distributions. This framework integrates concepts such as change-point detection, switching dynamical systems, and Bayes online learning, making it particularly adept at managing the frequent distributional shifts common in online learning environments. Inspired by this work, we considered incorporating Bayes online learning to enhance the online adaptability of the entire SLAM system, enabling it to better navigate new environments.
Bayes online learning is extensively employed across various domains, including financial market forecasting, medical care, autonomous vehicle technology, and other fields. In Bayes online learning, model parameters are treated as random variables, with their distributions encapsulating our uncertainty regarding these parameters. During the learning process, each arrival of new data transforms the previous posterior distribution into a new prior distribution, which is then combined with the new data to form an updated posterior distribution. This iterative process continues, progressively refining the model parameters. Online learning can enhance the SLAM system’s ability to generalize to previously unseen environments. Through continuous learning and integration of new data, the model can persistently optimize and adjust its internal parameters, thereby effectively managing diverse scenarios.
3. Method
The research builds upon the deep learning-based SLAM system, DROID-SLAM [Reference Teed and Deng41], with the objective of enhancing the model’s generalization to unknown environments using a single training set. This is achieved through the integration of EMA and Bayes online updating.
In selecting techniques to enhance the model’s generalization capability, we adopt the EMA and Bayes online learning methods, primarily due to their advantages in online adaptability [Reference Cai, Ravichandran, Maji, Fowlkes, Tu and Soatto6]. In contrast, alternative methods typically necessitate additional modifications or extensions to the model during the training phase. For instance, data augmentation techniques [Reference Sheng, Lu and Li34] enhance data diversity by expanding the training dataset, while transfer learning [Reference Ada, Ugur and Akin1] necessitates additional training on top of pre-existing models to adapt to new tasks or environments. These methods generally incur additional computational resources and extended training time.
In contrast to these methods, our method enables real-time model updates during the inference phase. By employing EMA to smooth changes in historical data and dynamically adjusting the model’s confidence through Bayes methods, we effectively handle non-smoothness and noise in the data.
Moreover, this method incurs minimal computational overhead when processing large-scale real-time data. In our research, we will investigate the development of a more efficient SLAM system and explore the integration of EMA with Bayes online updating. Through these enhancements, we aim to enhance the system’s generalization capability and mitigate the accuracy degradation of SLAM systems when operating in unknown environments.
3.1. Framework
The work reconstructs scenes by analyzing a sequence of ordered images
$\{I_t\}_{t=0}^N$
. Specifically, for each image
$t$
, we maintain two unknown state variables: camera pose
$G_t \in SE(3)$
and inverse depth
$d_t\in \mathbb {R}_+^{H \times W}$
. The two-state variable sets, the pose set
$\{G_t\}_{t=0}^N$
and the inverse depth set
$\{d_t\}_{t=0}^N$
, are iteratively updated as each new frame is processed. EMA is integrated into the iterative network, performing smoothing updates at an intermediate layer. For the remainder of this paper, it is important to note that all references to depth pertain to the inverse depth parameterization. Utilizing inverse depth enhances the sensitivity and accuracy of detecting changes in scene depth while preserving the algorithm’s efficiency.
By constructing a frame-graph
$(\nu, \varepsilon )$
, which represents the covariance between frames, the edges
$(i,j)\in \varepsilon$
indicate that images
$I_i$
and
$I_j$
contain overlapping regions and share points. The frame graph is dynamically constructed during both the training and inference phases. Following each update of pose or depth information, the system undergoes a visibility re-evaluation process to update the graphical representation of the frames dynamically. When the camera’s field of view revisits a previously mapped region to achieve loop closure (i.e., identifying and correcting cyclic portions of the path to enhance the map’s consistency and accuracy), we introduce long-distance connecting edges into the frame graph. These edges span spatial regions that might otherwise be considered discontinuous, thereby establishing comprehensive spatial connections. Ultimately, to enhance the algorithm’s broad applicability and online adaptability, we incorporate an online learning method grounded in Bayes theory into the overall algorithm. This method enables the algorithm to continuously update its internal model or parameters based on newly acquired data while processing the data stream in real time, thus achieving a sensitive response to environmental changes and continuous optimization. Through the Bayes framework, we can rationally incorporate prior knowledge and dynamically adjust the model’s posterior distribution based on observed data, thereby enhancing the algorithm’s adaptability to new situations and unknown changes while maintaining stability.
3.2. Feature extraction and correlation
The feature extraction component of DROID-SLAM parallels the RAFT framework, which reconstructs scenes by parsing a sequence of ordered images
$\{I_t\}_{t=0}^N$
newly introduced into the system. It leverages RAFT’s dual-network strategy: constructing both a correlation volume network and a context network. The feature network comprises six residual blocks and three downsampling layers, effectively reducing the resolution of each input image to one-eighth of the original, thus generating a high-level feature map.
Constructing the correlation pyramid is a crucial step in enhancing performance. For each pair of pixel points
$(i,j)\in \varepsilon$
in an image pair, we compute a four-dimensional correlation volume, as illustrated in Eq. (1). This is accomplished by calculating the dot product of the feature vectors corresponding to feature points
$g_\theta (I_i)$
and
$g_\theta (I_j)$
in the two images. This process captures not only the similarity between pixels but also preserves the spatial location information. A four-layer correlation pyramid is constructed by average pooling the last two dimensions of the correlation volume.
\begin{equation} \ C^{ij}_{u_1 v_1 u_2 v_2} = (g_\theta (I_i)_{u_1 v_1}) \cdot (g_\theta (I_j)_{u_2 v_2}) \end{equation}
Upon completing the construction of the correlation pyramid, the next step involves performing a correlation lookup. Utilizing a lookup operator to index the correlation volume within a grid of radius
$r$
, the operator retrieves the corresponding correlation information tensor at various resolutions,
$L_r \,:\, \mathbb {R}^{H \times W} \times \mathbb {R}^{H \times W \times 2} \rightarrow \mathbb {R}^{H \times W \times (r+1)^2}$
.
3.3. Updating operators and training
The image shows how the input image is optimized from feature extraction, feature finding, and finally through the fusion of the EMA’s update operator to the DBA layer all in one process.
Integrating Exponential Moving Average (EMA) into the core component of DROID-SLAM, the update operator, facilitates smoother updates within the system. At the heart of the operator lies a
$3\times 3$
Convolutional Gated Recurrent Unit (ConvGRU) with a hidden state
$h$
. The operator updates the hidden state and generates a bit position
$\Delta \xi ^{(k)}$
along with an inverse depth update
$\Delta d^{(k)}$
, subsequently applying these updates to the current inverse depth and bit position estimations, as illustrated in Eqs. (2) and (3).
\begin{align} \ G^{(k+1)} = \text {Exp}(\Delta \xi ^{(k)}) \circ G^{(k)}\\[1pt]\nonumber \end{align}
\begin{align} \ \quad d^{(k+1)} = \Delta d^{(k)} + d^{(k)} \\[8pt]\nonumber\end{align}
The architectural diagram of the update operator for the fusion EMA is shown in Figure 1.
3.3.1 Updating operators for fusion of EMA
Correspondence is established prior to initiating the update process. For the pixel coordinate grid defined for the
$i$
-th frame image,
$p_i\in \mathbb {R}^{H \times W \times 2}$
, the correspondence field
$p_{ij}$
is calculated.
\begin{align} \ p_{ij} &= \Lambda _c(G_{ij} \circ \Lambda _c^{-1}(p_i, d_i)) \quad p_{ij} \in \mathbb {R}^{H \times W \times 2} \end{align}
\begin{align} \ G_{ij} &= G_j \circ G_i^{-1} \\[8pt]\nonumber \end{align}
Eq. (4) for
$\Lambda _c$
is the mapping of 3D points onto the image, and the corresponding position of each point in frame
$i$
in frame
$j$
is calculated by using the known positional transformations and the camera model. Eq. (5) is used to calculate the relative pose between two frames, which is obtained by the combination of the pose of frame
$j$
with the inverse of the pose of frame
$i$
.
The network retrieves correlation features from the correlation volume
$C_i\,j$
for each edge
$(i,j)\in \varepsilon$
by indexing the correlation volume with the corresponding field and computes them based on the optical flow differences induced by camera motion. The residuals from the previous Bundle Adjustment (BA) solution are linked to the optical flow field, allowing the network to receive feedback from previous iterations. The correlation features offer visual similarity information, while the optical flow provides smooth motion information, thereby enhancing robustness. The correlation and optical flow features are injected into the GRU after processing through two convolutional layers, with contextual features introduced via elementwise addition. The GRU operates locally, while the global context is extracted by averaging the hidden states across the spatial dimensions of the image, serving as an additional input to the GRU.
\begin{equation} \ \alpha _t = \beta \alpha _{t-1} + (1-\beta )\theta _t \end{equation}
Concurrently, this work introduces the EMA method within the network’s GRU component. The EMA method is implemented to address the jittering of model weights at the actual optimal point during the last n steps. By averaging these steps, the model becomes more robust. During the optimization process,
$\theta _t$
represents the model weight at the current moment,
$\alpha _t$
is the shadow weight averaged over the previous
$t$
moments, and
$\beta$
is the weighted weight value, as depicted in Eq. (6). We maintain the shadow weights throughout the training process; these weights are not involved in training but are utilized during the inference phase to enhance the system’s generalization capability. EMA mitigates error amplification through the weighted averaging of historical data, thereby reducing oscillations during the training phase, particularly in the initial stages of training. As the model adapts to new data distributions, EMA effectively prevents overfitting to anomalous or noisy data points. EMA is employed to smooth the model parameters, including pose estimation and feature confidence, which are vulnerable to noise and dynamic environments, and to update these parameters with each model iteration.
Finally, in generating the updated hidden state
$h_{k+1}$
, the modified flow field
$r_{ij}$
and the confidence map
$w_{ij}$
of the dense flow field are predicted. The modified flow field is used to correct the dense counterpart field shown in Eq. (7).
\begin{equation} \ p_{ij}^{*} = r_{ij} + p_{ij} \end{equation}
The pixel-level damping factor and the
$8\times 8$
mask used for the inverse depth estimation during up-sampling are predicted by aggregating features that share the same source view
$i$
. The inverse depth estimation is performed similarly to the inverse depth estimation during up-sampling.
\begin{equation} \ E(\mathbf {G}', \mathbf {d}') = \sum _{(i,j) \in \mathcal {E}} \left \| \mathbf {p}_{ij}^{*} - \Lambda _c \left ( \mathbf {G}_{ij}' \circ \Lambda _c^{-1} (\mathbf {p}_i, \mathbf {d}_i')\right ) \right \|_{\Sigma _{ij}}^2 \quad \Sigma _{ij} = \text {diag} \, \mathbf {w}_{ij} \end{equation}
The corrected
$p_{ij}^{*}$
, weighted by
$w_{ij}$
, is subsequently fed into the Dense Bundle Adjustment (DBA) layer. A set of flow revisions is transformed into a series of pose and pixel depth updates by optimizing the cost function, as delineated in Eq. (8). The DBA layer is integrated into the computational map, and backpropagation is conducted through this layer during training.
3.3.2 Training
The training process, conducted using PyTorch akin to DROID-SLAM, addresses the canonical degrees of freedom issue in monocular setups by fixing the first two poses of the training sequence. This method enhances both the conditional and gradient stability of the linear system. The training video is dynamically generated to maintain an average optical flow between 8px and 96px across neighboring frames. Supervision is achieved through a dual loss system, combining pose loss and optical flow loss, which serves to enhance both the stability and downstream performance of the model.
The figure prominently displays the key components of the SLAM system at the top, with the green area at the bottom illustrating the principal optimization process of the frontend, including the integration of Bayes online learning. The pink area delineates the maintenance status throughout the update process.
3.4. SLAM system
The operational process of our SLAM system bears similarity to DROID-SLAM [Reference Teed and Deng41]. However, our innovation incorporates Bayes online learning into the frontend of the SLAM system, as depicted in Figure 2. This enhancement significantly boosts the system’s generalization capabilities in unfamiliar scenarios.
3.4.1 Bayes online learning
Bayes online learning can be effectively applied in this context to adaptively update model parameters based on incoming data streams. Here’s how it can be integrated:
Bayes online learning maintains a posterior distribution over model parameters, which is updated sequentially as new data arrives. Given the model weight at time
$t$
,
$\theta _t$
, and the shadow weight averaged over previous moments,
$\alpha _t$
, Bayes inference provides a principled way to update these weights. The shadow weight
$\alpha _t$
can be updated recursively using the following equation:
\begin{equation} \alpha _{t} = \gamma \cdot (w_{target} - \alpha _{t-1}) + \alpha _{t-1} \end{equation}
Where
$\gamma$
is the learning rate of the update processing. This allows the system to track the optimal point more robustly by averaging recent updates, thereby reducing the impact of noise or rapid fluctuations in the model parameters.
During the inference phase, the shadow weights
$\alpha _t$
are utilized to enhance the system’s generalization capability without directly affecting the training process. This integration enhances the model’s stability and adaptability over time, which is crucial for tasks where continuous learning and adaptation to new data are required.
Algorithm 1. Bayes Online Learning for SLAM
Incorporating Bayes online learning into the optimization process involves adapting the cost function
$ E(\mathbf {G}', \mathbf {d}')$
from Eq. (8) to include the posterior distribution over model parameters. The updated cost function can be formulated as:
\begin{equation} \ E(\mathbf {G}', \mathbf {d}', \theta _t) = \sum _{(i,j) \in \mathcal {E}} \left | \mathbf {p}{ij}^{*} - \Lambda _c \left ( \mathbf {G}{ij}' \circ \Lambda _c^{-1} (\mathbf {p}_{ij}, \mathbf {d}i')\right ) \right |^{\Sigma {ij}^2} + \lambda \cdot R(\theta _t | D_t) \ \end{equation}
where
$R(\theta _t \mid D_t)$
is the regularization term based on the posterior distribution
$P(\theta _t \mid D_t)$
of parameters
$\theta _t$
given the current data
$D_t$
.
\begin{equation} \ R(\theta _t \mid D_t) = -\log P(\theta _t \mid D_t) \ \end{equation}
This defines
$R(\theta _t \mid D_t)$
as the negative logarithm of the posterior probability
$P(\theta _t \mid D_t)$
. This regularization term ensures that the model parameters remain within a reasonable range and prevents overfitting, which is especially important in online learning scenarios where new data continuously updates the model’s parameters.
Thus, by integrating Bayes online learning with the existing methods described, the model not only benefits from enhanced stability through shadow weights
$\alpha _t$
but also adapts more effectively to new data while maintaining the robustness and generalization capabilities. This method supports continuous improvement and adaptation in dynamic environments, making it suitable for applications such as real-time data processing and adaptive learning systems.
4. Experiments
The experiments are designed to assess the enhancement in generalization ability of our algorithm by utilizing a variety of datasets. The experimental framework is methodically structured into two core components: comparative analysis and ablation study. The principal evaluation metric employed in this study is the Absolute Trajectory Error (ATE) [Reference Sturm, Engelhard, Endres, Burgard and Cremers36] for comparative analysis. Consistent with our focus on enhancing generalization capability, the training strategy closely mirrors that of DROID-SLAM, utilizing monocular video sequences derived from the synthetic Tartan Air dataset. Specific implementation details are deliberately omitted for brevity.
4.1. Comparative analysis
The comparative experiments are assessed using the EuRoC and TUM-RGBD datasets. These datasets have established themselves as standard benchmarks for evaluating SLAM algorithms, owing to their extensive adoption within the research community. Leveraging these datasets enables a more comprehensive comparison of the performance across various algorithms, thereby ensuring the fairness, consistency, and reproducibility of the experimental results.
EuRoC [Reference Burri, Nikolic, Gohl, Schneider, Rehder, Omari, Achtelik and Siegwart5] dataset encompassing both monocular and binocular configurations, serves as a classical benchmark for SLAM systems, thereby enhancing the reliability of evaluations. This dataset comprises multiple sequences of data collected by a micro air vehicle equipped with a camera and an IMU. We employ the EuRoC dataset to assess the performance of both monocular and binocular systems, as detailed in Table I.
Many deep learning-based SLAM algorithms [Reference Czarnowski, Laidlow, Clark and Davison9, Reference Teed and Deng39, Reference Wang, Hu and Scherer43] exhibit weak generalization when tested on the EuRoC dataset, making it a more precise benchmark for evaluating generalization capabilities. Within a monocular setting, our primary focus is to compare the results yielded by DROID-SLAM [Reference Teed and Deng41] on the dataset. The method performs excellently on the V202 dataset, with a
$37\%$
reduction in error rate compared to DROID-SLAM, and a
$20\%$
reduction in overall error rate across all datasets. Although our method does not outperform DROID-SLAM on the MH
$05$
and V
$102$
datasets for visual odometry, this discrepancy is primarily attributable to rapid camera motion and abrupt illumination changes within these datasets, which lead to low correlation between adjacent frames and complicated feature matching, thereby impacting the accuracy of visual odometry. However, our method demonstrates a significant improvement in visual odometry performance, with an overall error rate reduction of
$31\%$
. Despite mediocre performance on individual datasets, the substantial overall reduction in error demonstrates a significant enhancement in the generalization ability of our method.
In the evaluation of monocular SLAM on the EuRoC datasets, the symbol
$^{\dagger }$
represents visual odometry methods. The underlined data points represent the primary data we are comparing against.
We benchmark the performance of our method against data from DROID-SLAM and other classical algorithms. When comparing DeepV2D trained on the TartanAir [Reference Wang, Zhu, Wang, Hu, Qiu, Wang, Hu, Kapoor and Scherer44] dataset to versions trained on NYUv2 [Reference Silberman, Hoiem, Kohli and Fergus35] and ScanNet [Reference Dai, Chang, Savva, Halber, Funkhouser and Nießner10], the error rate is markedly higher, and both the generalization ability and robustness are inferior to our algorithm, which exhibits fewer issues with drift.
TUM-RGBD [Reference Sturm, Engelhard, Endres, Burgard and Cremers36] The TUM-RGBD dataset is a widely utilized resource within robotics and computer vision research, featuring a diverse array of complex scenes. It provides timestamped RGB and depth image sequences and ground truth data for camera poses. We present a comparison against other deep learning SLAM systems in Table II, highlighting our algorithm’s enhanced generalization capabilities. We also cross-reference data from DROID-SLAM [Reference Teed and Deng41] to demonstrate the effectiveness of our algorithm.
For the nine sequences successfully tracked within the TUM-RGBD dataset, the teddy sequences have an ATE that is
$58\%$
lower than that of DROID-SLAM. While some sequences show less optimization, the overall ATE is
$35\%$
lower compared to DROID-SLAM. Concurrently, for the performance across these nine sequences, our ATE is
$90\%$
lower than that achieved by DeepFactor [Reference Czarnowski, Laidlow, Clark and Davison9].
4.2. Ablation study
To evaluate the impact of individual model components on overall performance, we conduct an in-depth analysis of their contributions to the final results, systematically removing the EMA and Bayes online learning components. This process further aids in identifying which components are most critical for enhancing the model’s generalization ability. In this section, we utilize the ETH3D [Reference Schöps, Schönberger, Galliani, Sattler, Schindler, Pollefeys and Geiger32] (RGB-D) dataset to assess the impact of each individual component.
We compare the Absolute Trajectory Error (ATE) of deep learning SLAM systems based on nine sequences from the TUM-RGBD dataset against the ATE of our algorithm.
The images illustrate the visualization of the latent space for varying
$\beta$
-parameters.
By analyzing the latent space representation presented in Figure 3, we determined the optimal
$\beta$
value for EMA. The results indicate that a
$\beta$
value of
$0.6$
yields the most pronounced feature differentiation, thereby enhancing the network’s ability to represent features more effectively. In contrast, at other
$\beta$
values, the feature differentiation of the embedding is less pronounced, which fails to significantly improve the network’s representational capacity.
Finally, we conducted ablation experiments utilizing six subsets of the ETH3D (RGB-D) dataset, each varying in illumination conditions and camera transformation speed. We employed ATE to assess the impact of each component on the algorithm, thereby demonstrating the contribution of our method to the system by comparing the results after systematically removing different components.
The comparison results are presented in Table III, where the ATE is reduced by
$6.4\%$
and
$7.1\%$
, respectively, upon incorporating the EMA and Bayes online learning methods. Overall, the EMA method contributes more significantly to the improvement of system performance. However, in certain datasets (indicated by the lower horizontal line in Table III), the presence of low-texture regions, rapid camera motion, insufficient prior information, and low correlation between adjacent frames led to suboptimal performance of both components. Our method integrates both the EMA and Bayes online learning components, achieving an overall ATE reduction of
$21\%$
, thereby outperforming each component individually. The interaction between the two components effectively addresses issues related to insufficient prior information and dataset limitations, ultimately demonstrating superior performance across all sequences.
5. Conclusion
In this study, we improve the generalization capacity of the DROID-SLAM system across diverse location scenarios using non-training datasets, by integrating Exponential Moving Average (EMA) and Bayes online learning techniques. In non-training environments, previously trained models frequently struggle to adapt to novel and unseen conditions. The integration of EMA and Bayes methods facilitates real-time adaptation during the inference phase, thereby effectively mitigating the challenges posed by data variations and uncertainties in real-world scenarios. We observe that the enhanced DROID-SLAM system demonstrates substantial improvements over traditional SLAM systems, other deep learning-based SLAM methods, and previous versions of DROID-SLAM.
Through comprehensive experiments, we find that the enhanced DROID-SLAM system shows notable improvements compared to traditional SLAM systems, other deep learning-based SLAM methods, and previous versions of DROID-SLAM. However, our improvements still have certain limitations. In the absence of Visual Odometry focusing on global maps and loopback detection, the system may fail to achieve the desired robustness in environments with drastic lighting changes or significant motion blur. Future research may explore further integration with other advanced technologies, as well as the selection of more sophisticated feature extraction and adaptive methods to further enhance the system’s performance in complex environments. Additionally, the impact of the algorithms in practical applications could be assessed through hardware deployment, with a focus on optimizing computational efficiency and real-time performance, while enhancing robustness under more realistic and complex environmental conditions. Further optimization of these methods may yield more substantial performance improvements.
The table presents the Absolute Trajectory Error (ATE) results for the six sequences derived from the ETH3D (RGB-D) dataset in the ablation study.
Author contributions
Jun Liu and Haihang Deng conceived and designed the study. Haihang Deng performed statistical analyses and wrote the article.
Financial support
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Competing interests
The authors declare no conflicts of interest exist.
Ethical approval
Not applicable.
