2.1. Dynamic object elimination

Instance segmentation is performed using Mask R-CNN [Reference He, Gkioxari, Dollár and Girshick21] to identify dynamic objects and filters their feature points. The model is implemented with the Detectron2 framework (PyTorch 1.13.0, CUDA 11.7) and uses the COCO pre-trained weights (R50-FPN backbone) without fine-tuning to maintain generalizability. Figure 2 illustrates dynamic feature detection results of a COCO-pretrained model evaluated on a TUM dataset sample:

Figure 2.

Dynamic mask based on prior model.

As shown in Figure 2(c), parts of the person’s body are still exposed outside the mask, which may lead to incomplete removal of dynamic points. Therefore, this paper performs a "dilation" operation on the original mask. The dilated result in Figure 2(d) fully encapsulates the person’s contour within the mask, eliminating erroneous extraction of partial dynamic object features caused by incomplete mask coverage. This can improve the system’s accuracy.

Figure 3.

Dynamic feature detection based on multi-view geometry.

This paper employs multi-view geometry to further determine potentially mobile objects that either lack prior models or exhibit low motion probability in existing prior models, thereby achieving enhanced dynamic object elimination performance. The multi-view geometry algorithm follows this workflow:

1. 1. Select a subset of images with the greatest overlap with the current frame (CF) as key frames (KF), and set the number of key frames.

2. 2. Project each feature point $x$ from the previous KF onto the CF, denoted as $x'$ , and calculate their projected depth $z_{pro}$ , as shown in Figure 3.

3. 3. For each feature point, use its corresponding 3D coordinate $X$ to calculate the parallax angle $\alpha$ between points $x , x'$ , and $X$ , and set the threshold $\tau _{\alpha }$ .

4. 4. Based on the depth map obtained from the RGB-D camera for the current frame, the depth value $z'$ corresponding to point $x'$ is obtained. The difference between the projected depth and the real depth, $\bigtriangleup z=z_{pro}-z'$ , is calculated, and a threshold $\tau _{z}$ is set.

5. 5. Determine whether the point is a dynamic feature point. If the difference between the projected depth and the real depth is small, i.e., $\bigtriangleup z=0\ll \tau _{z}$ and the disparity angle $\alpha \lt \tau _{\alpha }$ , the point is considered static, as shown in Figure 3(a). Otherwise, it is classified as a dynamic keypoint, as shown in Figure 3(b).

6. 6. The thresholds for the multi-view geometry check, specifically the parallax angle $\tau _{\alpha }=15^{\circ}$ and the depth difference $\tau _{z}=0.5m$ , were determined through a grid search on the TUM RGB-D datasets to maximize the F1-score for dynamic point detection. A parallax angle exceeding $15^{\circ}$ indicates significant motion inconsistency, while a depth difference larger than $\tau _{z}=0.5m$ substantially surpasses the typical depth noise (e.g., < 4 cm for RGB-D cameras like Intel RealSense D435) of the sensor. These two criteria provide strong evidence for the identification of a dynamic point. These values have been validated across multiple sequences and proved effective in distinguishing true motion from sensor noise.

Thus, the dynamic object elimination strategy presented in this section operates through a dual-stage pipeline to ensure comprehensive filtering. In the first stage, a pre-trained Mask R-CNN model is employed to perform instance segmentation and mask a priori known dynamic objects (e.g., pedestrians), which effectively removes the large-scale dynamic interferences based on semantic priors. However, this semantic approach alone is insufficient to handle objects lacking pre-defined models or those that may still exhibit movement despite having low motion probability in the prior knowledge base. To mitigate this limitation, the second stage incorporates a multi-view geometry constraint. Consequently, this synergistic combination ensures the robust elimination of both macroscopic dynamic entities and finer-grained dynamic feature points, thereby substantially enhancing the system’s resilience in complex dynamic environments.

Figure 4.

Image enhancement flow chart.

2.2. Improved ORB feature extraction

The ORB-SLAM2 system utilizes ORB feature extraction in its visual odometry module for efficient computation. However, its fixed thresholds restrict the adaptability to illumination changes, while texture-dependent feature extraction results in uneven feature distribution – specifically, feature clustering in texture-rich areas vs. sparse features in texture-poor regions – thereby degrading the accuracy of pose estimation. Our solution proposes two key improvements: (1) adaptive gamma correction (AGC) for illumination-robust image enhancement and (2) an optimized quadtree algorithm designed to ensure uniform feature distribution. These two components significantly enhance the robustness and localization accuracy of the ORB algorithm.

The adopted image enhancement algorithm follows the workflow illustrated in Figure 4. The RGB image is first converted to grayscale, with each pixel intensity denoted as $I(x,y)$ . The standard deviation of the grayscale values is computed as:

(1) \begin{equation} \lambda =\sqrt{\frac{1}{M\cdot N}\sum _{x=1}^{M}\sum _{y=1}^{N}\left(I(x,y)-\overline{I}\right)^{2}} \end{equation}

where $\overline{I}$ is the mean intensity. If $\lambda \lt 0.25$ , the image is considered to have sufficient quality. Otherwise, it undergoes further processing. If $\overline{I}\lt 128$ , the image is deemed under-exposed and enhanced via AGC; otherwise, it is inverted before AGC.

AGC automates gamma adjustment using image statistics to improve contrast across varying illumination. The probability density of grayscale level $i$ is:

(2) \begin{equation} P(i)=\frac{n_{i}}{M\cdot N} \end{equation}

where $n_{i}$ is the number of pixels at level $i$ . A weighted probability density is then derived:

(3) \begin{equation} P_{\omega }(i)=P_{\max }\cdot \left(\frac{P(i)-P_{\min }}{P_{\max }-P_{\min }}\right)^{\alpha } \end{equation}

Here, $P_{\max }$ and $P_{\min }$ are the maximum and minimum probability densities, and $\alpha =0.2$ is a smoothing factor. The weighted distribution function is:

(4) \begin{equation} C_{\omega }(i)=\sum _{i=0}^{255}\frac{P_{\omega }(i)}{\sum P_{\omega }} \end{equation}

The gamma parameter is adapted as $\gamma =1-C_{\omega }(i)$ , and the corrected intensity is:

(5) \begin{equation} I'(x,y)=255\!\cdot\! \left(\frac{I(x,y)}{255}\right)^{\gamma } \end{equation}

To resolve the uneven feature distribution caused by varying image textures, this study enhances the camera pose estimation accuracy by implementing an improved quadtree algorithm that prevents redundant feature clustering and achieves feature homogenization. Traditional quadtree algorithms lack constraints on node splitting depth during homogenization. Excessive splitting leads to increased memory consumption and computational complexity, thereby reducing computational efficiency.

Our method introduces a constraint on the maximum node splitting depth to curb computational costs. Building upon the mathematical properties of the quadtree, where each splitting level increases the number of potential nodes by a power of four, we establish a theoretical relationship between target feature quantity $n_{\max }$ and maximum depth $d_{\max }$ . This relationship is analytically formulated as:

(6) \begin{equation}d_{max}=[\!\log _{4} n_{max}]\end{equation}

In the above equation, $n_{\max }$ represents the number of feature points to extract, and $d_{\max }$ is the split depth threshold. This conditional check is incorporated into the original node splitting step to verify whether the current splitting depth $d$ is less than the predefined maximum depth threshold $d_{\max }$ . Nodes are permitted to undergo further splitting only when $d\lt d_{\max }$ , thereby preventing excessive subdivision and ensuring computational efficiency.

In the improved ORB feature extraction process of this section, image preprocessing and feature extraction are carried out sequentially. First, AGC is applied to the RGB images acquired from the Intel RealSense D435 camera to enhance image quality by mitigating uneven illumination and improving contrast. Subsequently, the enhanced images are processed with the improved quadtree algorithm to achieve a more uniform spatial distribution of ORB feature points. This sequential processing pipeline ensures both improved image quality and reliable feature distribution.

2.3. Map fusion

The two key steps in map fusion are map alignment and information fusion [Reference Bosse and Zlot22]. The goal of map alignment is to ensure that both maps have the same format, scale, and resolution. Alignment algorithms [Reference Li, Wang and Kaess23, Reference Huang, Scherer and Kaess24] compute coordinate transformations between submaps, establishing the foundation for subsequent information association. Information association, after map alignment, is the process of merging information between maps to generate the final scene map. Figure 5 illustrates the complete fusion process.

Figure 5.

Map fusion flow chart.

In the map alignment phase, this paper proposes a combined alignment algorithm tailored for corridor environments. In Harris corner detection, corner responses change with map resolution, which may cause original corners to become unrecognizable. To address this, we incorporate scale invariance into the Harris corner detector.

First, the first-order image derivatives at point $(x,y)$ using the differential scale $\sigma _{D}$ are defined as:

(7) \begin{align} I_{x}(x,y,\sigma _{D})&=\frac{\partial }{\partial x}\left[g(\sigma _{D})\otimes I(x,y)\right]\\[-10pt]\nonumber \end{align}

(8) \begin{align} I_{y}(x,y,\sigma _{D})&=\frac{\partial }{\partial y}\left[g(\sigma _{D})\otimes I(x,y)\right] \end{align}

where $g(\sigma _{D})$ is a Gaussian kernel with standard deviation $\sigma _{D} , \otimes$ denotes the convolution operation, and $I(x,y)$ is the original image.

Next, the structure tensor composed of quadratic terms of the image derivatives is constructed:

(9) \begin{equation} S(x,y,\sigma _{D})=\left[\begin{array}{ll} I_{x}^{2}(x,y,\sigma _{D}) & I_{x}I_{y}(x,y,\sigma _{D})\\[5pt] I_{x}I_{y}(x,y,\sigma _{D}) & I_{x}^{2}(x,y,\sigma _{D}) \end{array}\right] \end{equation}

Finally, the scale-invariant Harris second-moment matrix is obtained by convolving the structure tensor $S$ with a Gaussian kernel $g(\sigma _{I})$ of standard deviation $\sigma _{I}$ , which yields:

(10) \begin{align} M(x,y,\sigma _{D},\sigma _{I})&=\sigma _{D}^{2}\cdot \left[g\left(\sigma _{I}\right)\otimes S(x,y,\sigma _{D})\right]\nonumber\\[5pt] &=\sigma _{D}^{2}\left[\begin{array}{ll} g\left(\sigma _{I}\right)\otimes I_{x}^{2}(x,y,\sigma _{D}) & g\left(\sigma _{I}\right)\otimes I_{x}I_{y}(x,y,\sigma _{D})\\[5pt] g\left(\sigma _{I}\right)\otimes I_{x}I_{y}(x,y,\sigma _{D}) & g\left(\sigma _{I}\right)\otimes I_{y}^{2}(x,y,\sigma _{D}) \end{array}\right] \end{align}

where $\sigma _{D}$ is the differential scale window, and $\sigma _{I}$ is the integral scale window. The differential scale window determines the variation of the differential values around the corner, filtering out noise around the corner through Gaussian smoothing. The integral scale window defines the current scale of the corner, which corresponds to the size of the filtering window $(x,y)$ .

The framework first constructs a multi-scale space $\sigma _{n}=\sigma _{I},k\sigma _{I},\ldots ,k^{n}\sigma _{I}$ . At each scale, it calculates the corner response value and applies non-maximum suppression within the 8-neighborhood of candidate corner points exceeding the predefined threshold. Subsequently, the algorithm computes the Laplacian of Gaussian (LoG) response at candidate corner positions across all scales:

(11) \begin{equation} \left| LoG\!\left(x,\sigma _{n}\right)\right| =\sigma _{n}^{2}\left| I_{xy}\left(x,\sigma _{n}\right)+I_{yy}\left(x,\sigma _{n}\right)\right| \end{equation}

Compare the $LoG$ values [Reference Mikolajczyk and Schmid25] in adjacent scale spaces:

(12) \begin{equation} \left| LoG\!\left(x,\sigma _{n}\right)\right| \gt \left| LoG\!\left(x,\sigma _{i}\right)\right| ,\forall \sigma _{i}\in \left\{\sigma _{n-1},\sigma _{n+1}\right\} \end{equation}

When Eq. (12) holds, the candidate corner point remains valid until the algorithm identifies the maximum $LoG$ value at its position across multiple scales. This maximum $LoG$ value determines the optimal scale for the corner point. Following Harris corner detection, a similarity-based filtering mechanism is applied to remove redundant corner points. In corridor environments, wall junctions exhibiting distinct geometric features are regarded as inner corner points, selected based on the occupancy ratio of grid states within a radius $r$ . Figure 6 illustrates the screening results after applying these criteria. The red crosses in the figure represent the positions of the Harris corner points. As seen in Figure 6(a), the original detected corner points contain redundancy. Figure 6(b) demonstrates the filtered results containing internal corner points, where only wall-corner keypoints remain after screening. Subsequently, the centroid method assigns a dominant orientation to internal corner points to ensure rotational invariance.

Figure 6.

From redundant Harris corner detection to interior corner extraction results.

To endow the Harris corner with a rotational descriptor, this paper employs an improved BRIEF feature as the inner corner descriptor. The improved BRIEF descriptor incorporates the main direction information of the feature points to achieve improved rotation invariance. A binary descriptor is adopted, and Hamming distance is used as the metric. Furthermore, for each pair of points, the Hamming distance is computed twice to account for the potential variations caused by the order of points, with the minimum value taken as the final descriptor distance. If the Euclidean and Hamming distance differences between point pairs lie within a specified tolerance range, the correspondence is validated. Otherwise, the algorithm initiates an iterative refinement process for inner corner screening.

Upon successful matching of the internal corner points, the positions of the corresponding points in submap $a$ and submap $b$ are denoted as $m_{a}=[x_{a},y_{a},1]^{T}$ and $m_{b}=[x_{b},y_{b},1]^{T}$ , respectively. The mapping relationship between the two submaps is given by:

(13) \begin{equation} m_{a}=\lambda Rm_{b}+l \end{equation}

Here, $\lambda$ represents the scale change, which we can calculate from the distance ratio between matched point pairs. $R$ is the rotation matrix, and $l$ is the translation matrix.

In the information fusion part, this paper proposes a method based on Bayesian inference to integrate grid information so that the fused map can better mirror the actual situation. The extra probability grids from Bayesian inference can lead to inaccurate map information expressions. To address this issue, the extra probability grids are represented in the closest status, ensuring that the grid types remain unchanged after fusion, thus making the fused map clearer and more accurate. The specific formula is as follows:

(14) \begin{equation} p_{f}=\left\{\begin{array}{l} p_{free}, if | p-p_{free}| =\min (| p-p_{free}| ,| p-p_{occ}| ,| p-p_{un}| )\\[2pt] p_{occ}, if| p-p_{\mathrm{occ}}| =\min (| p-p_{free}| ,| p-p_{occ}| ,| p-p_{un}| )\\[2pt] p_{un}, \textit{otherwise} \end{array}\right. \end{equation}

where $p_{f}$ is the final grid probability, and $p$ is the grid probability after fusion through Bayesian inference, with the predefined thresholds $p_{free}$ , $p_{occ}$ , and $p_{un}$ conventionally set to 0.2, 0.8, and 0.5, respectively. Considering the mapping characteristics of the two types of sensors, this paper defines a fusion rule, as shown in Table I.

Table I.

Map fusion rules.

The design of these rules is governed by the following principles:

1. 1. The Principle of Certainty: A high-confidence observation from any single sensor overrides conflicting or uncertain information from another. For instance, if one sensor classifies a grid as occupied ( $p_{occ}$ ), the fused result is always occupied, regardless of whether the other sensor reports free ( $p_{free}$ ) or unknown ( $p_{un}$ ). This is because missed obstacles (false negatives) pose a far greater safety risk to robot navigation than false positives.

2. 2. The Principle of Conservatism: In cases of high uncertainty, the most conservative assumption is adopted. Specifically, for the unknown-unknown fusion, neither sensor provides reliable information, indicating a highly ambiguous region. Arbitrarily classifying it as free could lead to ignoring undetected obstacles and potential collisions. Therefore, the safest and most rational strategy is to preserve its unknown status, prompting subsequent navigation planning or perception modules to allocate more attention or initiate re-exploration of this area.

3. 3. The Principle of Efficiency: A free-free fusion result is confirmed as free, indicating a consistent agreement between both sensors that the region is safely traversable, and the result is trustworthy.

These rules are designed to prioritize the safety and reliability of mobile robot operation in dynamic environments.