3.1. BWCR navigation problem

The BWCR navigation means that the biped wall-climbing robot can build a map of the curtain wall environment, plan a 3D climbing path according to the set destination, and track the resulted path until reaching the target.

A curtain wall environment can be represented as,

(1) \begin{align} \mathsf{M}=\left\{{\boldsymbol{\varepsilon }}_{i}\,|\,i=1,2,\cdots, N_{e}\right\} \end{align}

where $\varepsilon _{i}$ is the structural elements such as wall planes, frames (i.e., beams and columns), and features detected for localization, and N _e denotes the number of the elements. The environment map can be obtained through SLAM using appropriate sensors or CAD models of buildings.

Figure 2 shows a scenario in which a BWCR navigates in a 3D curtain wall environment. After constructing the environment map, one feasibility climbing path is generated based on a hierarchical 3D climbing path planner [Reference Zhu, Lu, Gu, Wei and Guan6], which consists of a sequence of footholds (gray triangles in Figure 2) connecting the start point and the destination, and single-step collision-free motion of each adjacent footholds (solid curves in Figure 2). The robot is placed in the starting position with an initial configuration and begins to climb along the planned path. Due to the influence of initial position deviation, motion deviation, environmental geometric errors, and the elastic deformation of the suction cup at the base, the robot may deviate from the predetermined path. Therefore, BWCR needs to constantly estimate its global position and correct the error in time, while meeting the kinematic constraints and avoiding obstacles. The resulted path-tracking model can be represented as,

Figure 2.

BWCRs navigate in a curtain wall environment.

(2) \begin{align} min\,d=\left\| f_{p}-f_{a}\right\| \end{align}

where $f_{p}$ denotes the planed foothold through the proposed climbing path planner [Reference Zhu, Lu, Gu, Wei and Guan6], and $f_{a}$ is the actual adsorption point of BWCR, which can be obtained by minimizing the deviation $d$ between $f_{p}$ and $f_{a}$ . Therefore, it is necessary to determine the pose of the swinging end using an appropriate sensing system and a corresponding method.

3.2. Proposed solution

Figure 3 depicts an overview of the proposed localization and navigation framework. To address the limitations of vision and other sensors in perceiving glass curtain walls, geometric information about environmental elements is extracted from the corresponding BIM to generate an environment map. Wall and frame information is used to plan a collision-free climbing route, and marker features are utilized for robot localization. A camera integrated into the suction module detects the ArUco marker, enabling the determination of the local pose of the swinging end within the marker coordinate frame. The global pose is then calculated based on the environment map. If deviations exist, appropriate motion correction strategies are employed to generate corrected footholds and plan collision-free single-step motions for various climbing states, such as moving on a single wall or transitioning between walls. Control parameters are sent to the robot controller to move the swinging end to the new foothold and ensure reliable adhesion to the wall surface. These operations are repeated until the robot reaches its destination.

Figure 3.

Overview of BWCR navigation method.

4.1. Environment map and its mathematical description

Curtain walls are widely used in large modern architecture, such as airports, train stations, cultural venues, financial centers, and commercial hotels. To facilitate understanding, the environment can be simplified as a combination of walls and supporting frames. Thus, the map $\mathrm{M }$ can be represented in the world coordinate frame {W} as,

(3) \begin{align} \mathrm{M } & = \left\{\left\{\boldsymbol{\omega }_{i}\,|\,i=1,2,\cdots, N_{m},N_{m}\geq 1\right\}\right.,\nonumber \\[3pt] & \quad \left\{\boldsymbol{\alpha }_{j}\,|\,j=1,2,\cdots, N_{a},N_{a}\geq 1\right\},\\[3pt] & \quad \left.\left\{\boldsymbol{\varsigma }_{k}\,|\,k=1,2,\cdots, N_{s},N_{s}\geq 0\right\}\right\} \nonumber \end{align}

where $\boldsymbol{\omega }_{i}$ denotes the i-th wall, $\boldsymbol{\alpha }_{j}$ is the j-th marker, and $\boldsymbol{\varsigma }_{k}$ is the k-th frame. A flat wall can be represented as a convex polygon with vertices,

(4) \begin{align} \boldsymbol{\omega }_{i}=\left({}^{W}{\boldsymbol{\nu }}_{\gamma },\boldsymbol{{n}}_{i}\right) \end{align}

where ${}^{W}{\boldsymbol{\nu }}_{\gamma}$ is the ordered vertices, and ${}^{W}{\boldsymbol{\nu }}_{\gamma } = \{{}^{W}{P}{_{i}^{}}\,|\,i=1,2,\cdots, N,N\gt 3\}$ . $\boldsymbol{n}_{i}$ is the normal vector of $\boldsymbol{\omega }_{i}$ , and indicates the surface of the wall allowed to climb on. Note that, a concave polygon can always be divided into a limited number of convex polygons.

The ArUco marker is selected for locating the BWCR due to its effectiveness, robustness against detection errors and occlusion, and capability to directly estimate the camera’s relative pose [Reference Romero-Ramirez, Munoz-Salinas and Carnicer32]. A mark on the wall $\boldsymbol{\omega }_{i}$ can be expressed as,

(5) \begin{align} \boldsymbol{\alpha }_{i}=\left\{ID,\,i,\,{}^{W}{\boldsymbol{\vartheta }}_{a},\,{\vphantom{T}}^{W}_{A}{\boldsymbol{T}}_{j}\right\} \end{align}

where ID is the unique identifier of a marker, i represents the serial number of the wall where the marker is located, ${}^{W}{\boldsymbol{\vartheta }}_{a}=\{{}^{W}{\boldsymbol{P}}_{i}\,|\,i=1,2,3,4\}$ is the set of the four corners in a clockwise direction, while ${\vphantom{T}}^{W}_{A}{\boldsymbol{T}}{_{j}^{}}$ denotes 4 × 4 homogeneous transformation matrix from the ArUco marker coordinate frame to the world coordinate frame, which constitutes the ArUco marker database.

For a BWCR, the walls are the passable area, while the frames and connectors are the obstacles, which can be described as a minimum envelope cylinder. Thus, a frame can be expressed with the following parameter equation

(6) \begin{align} \boldsymbol{\varsigma }_{k}=\left\{{}^{W}{\boldsymbol{P}}_{s},{}^{W}{\boldsymbol{P}}_{e},r\right\} \end{align}

Where ${}^{W}{\boldsymbol{P}}_{s}, {}^{W}{\boldsymbol{P}}_{e}$ and denote the two endpoints and radius of the simplified cylinder respectively.

4.2. ArUco marker database

Figure 4 shows us the model of solving the transformation matrix ${\vphantom{T}}^{W}_{A}{\boldsymbol{T}}{}$ . The coordinates of the four corners ${}^{W}{\boldsymbol{\vartheta }}_{a}$ of the ArUco marker can be extracted from the BIM model. Each point ${}^{W}{\boldsymbol{P}}_{i}$ can also be calculated through the corresponding corner ${}^{A}{\boldsymbol{P}}_{i}^{\prime}$ described in the marker coordination frame {A}. The transformation formula can be written as,

Figure 4.

Calculation of transformation matrix from the marker to the world coordinate frame.

(7) \begin{align} \left[\begin{array}{l} {}^{W}{\boldsymbol{P}}_{i}\\[3pt] 1 \end{array}\right]=\left[\begin{array}{l@{\quad}l} \boldsymbol{R} & t\\[3pt] \mathbf{0} & 1 \end{array}\right]\left[\begin{array}{l} {}^{A}{\boldsymbol{P}}_{i}^{\prime}\\[3pt] 1 \end{array}\right] \end{align}

where $\boldsymbol{R}$ and t are rotation component and translation component of ${\vphantom{T}}^{W}_{A}{\boldsymbol{T}}{}$ . To solve those two components, the iterative closest point (ICP) algorithm [Reference Besl and McKay33] can be used by converting the calculation process into the pose estimation problem with known feature point pairs.

First, calculating the centroids ${}^{W}{\boldsymbol{P}}{_{c}^{}}$ and ${}^{A}{\boldsymbol{P}}_{c}^{\prime}$ sets of corresponding points

(8) \begin{align} {}^{W}{\boldsymbol{P}}{_{c}^{}}=\frac{1}{n}\sum _{i=1}^{n}\boldsymbol{P}_{i},\,{}^{A}{\boldsymbol{P}}_{c}^{\prime}=\frac{1}{n}\sum _{i=1}^{n}\boldsymbol{P}_{i}^{\prime} \end{align}

Then, the centroid-removed coordinates of each point can be obtained

(9) \begin{align} {}^{W}{\boldsymbol{Q}}{_{i}^{}}={}^{W}{\boldsymbol{P}}{_{i}^{}}-{}^{W}{\boldsymbol{P}}{_{c}^{}},\,{}^{A}{\boldsymbol{Q}}{_{i}^{}}=\,{}^{A}{\boldsymbol{P}}{_{i}^{}}-{}^{A}{\boldsymbol{P}}{_{c}^{}} \end{align}

After that, a 3 × 3 matrix can be calculated by

(10) \begin{align} \boldsymbol{Q}=\sum _{i=1}^{n}{}^{W}{\boldsymbol{Q}}{_{i}^{}}\,{}^{A}{\boldsymbol{Q}}{_{i}^{T}} \end{align}

The SVD method [Reference Pomerleau, Colas and Siegwart34] is used to decompose the matrix $\boldsymbol{Q}$ . The decomposed diagonal matrix is represented by $\boldsymbol{U}$ and $\boldsymbol{V}$ respectively, and the singular value is represented by $\boldsymbol{\sigma }$

(11) \begin{align} \boldsymbol{Q}=\boldsymbol{U}\left[\begin{array}{l@{\quad}l@{\quad}l} \sigma _{1} & 0 & 0\\[3pt] 0 & \sigma _{2} & 0\\[3pt] 0 & 0 & \sigma _{3} \end{array}\right]\boldsymbol{V}^{T} \end{align}

Finally, the rotation matrix component can be calculated as

(12) \begin{align} \boldsymbol{R}=\boldsymbol{V}\times \boldsymbol{U} \end{align}

And, the translation matrix component can be obtained

(13) \begin{align} \boldsymbol{t}={}^{W}{\boldsymbol{P}}{}-\boldsymbol{R}\cdot {}^{A}{\boldsymbol{P}}_{i}^{\prime} \end{align}

After extracting the coordinates of the markers and the coordinate wall in the world coordinate frame, the ICP algorithm can be used to solve the coordinates of the marker in the wall, and then the mark can be arranged in the real environment.

4.3. Data extraction from BIM

According to the International Organization for Standardization (ISO 19650-1:2018), BIM is defined as the utilization of a shared digital representation of a constructed asset to streamline design, construction, and operational processes, thereby providing a dependable foundation for decision-making. In the BIM modeling phase, designers can readily create standardized components, including curtain walls and frames. However, for distinctive elements like ArUco markers, customized component libraries can be developed and seamlessly incorporated into the design workflow. Nevertheless, generating the map for BWCR directly from BIM is not feasible. Instead, an extraction algorithm must be meticulously crafted to retrieve specific data from the 3D building model in BIM, enabling the construction of the curtain wall environment map.

The diagram of our proposed BIM data extraction method is shown in Algorithm 1. First, the various types of environment components are found through a filter. Then, all the components are traversed to extract the corresponding geometric data and ID information according to their internal structure. The sets of curtain walls, ArUco markers, and frames $\boldsymbol{\varsigma }$ can be obtained, respectively. Note that, other components are enveloped as cylinders through $\text{CYLINDRICALENVELOPE}()$ and considered cylinder-type obstacles. Finally, all extracted data will be stored to build the curtain wall environment map $\mathsf{M}$ .

Algorithm 1:

Data extraction from BIM

5.1. Global localization

During climbing in a curtain wall scenario, BWCR needs to obtain its own global position and posture information to evaluate whether it deviates from the planned route. Figure 5 illustrates the basic idea of the visual positioning scheme of BWCR based on ArUco markers. The swinging end of the robot moves along the planned single-step path to the next foothold. The onboard monocular camera installed at the suction module perceives the environment. Once the robot detects and obtains the pose of the markers ${\vphantom{T}}^{A}_{C}{\boldsymbol{T}}{}$ , where {C} and {A} are the coordinate frames of the camera and the ArUco marker, we can calculate the global pose of the swinging end {E} with the relationship between the coordinate frames,

Figure 5.

Global visual localization of BWCR.

(14) \begin{align} {\vphantom{T}}^{W}_{E}\boldsymbol{T} = {\vphantom{T}}^{W}_{A}{\boldsymbol{T}}\,{\vphantom{T}}^{A}_{C}{\boldsymbol{T}}\,{\vphantom{T}}^{C}_{E}{\boldsymbol{T}}{} \end{align}

where ${\vphantom{T}}^{W}_{A}{\boldsymbol{T}}{}$ is the transformation matrix from the marker coordinate frame to the world coordinate frame, which can be extracted from the environment map. ${\vphantom{T}}^{C}_{E}{\boldsymbol{T}}{}$ denotes the transformation matrix from the swinging end of the robot to the monitoring camera, which can be obtained through eye-in-hand calibration. Thus, obtaining ${\vphantom{T}}^{A}_{C}{\boldsymbol{T}}{}$ with an appropriate algorithm is the most critical step of the global localization of BWCR.

5.1.2. ArUco marker detection algorithm

The ArUco marker detection process is outlined in Algorithm 2. First, the visual image $\boldsymbol{I}$ is collected by the monitoring camera on the swinging end, and transmitted to the host PC through the wireless network. Then, the operator $RGB2GRAY()$ converts the RGB image into a grayscale image. After filtering the image with a Gauss filter $\text{IMAGEF}\text{ILTER}()$ , an adaptive threshold method $\text{ADAPTIVES}\text{EGMENT}()$ segments the markers. Next, the contours of squares and convex hulls are detected using an approximation polygon algorithm $\text{APPROXC}\text{ONTOUR}()$ . If the contour perimeter exceeds the set threshold, the corresponding marker is eliminated. At this stage, all candidate markers are identified, and the front view of the image is generated using a perspective transformation algorithm $\text{PERSPECTIVET}\text{RANSFORM}()$ . Afterward, the Otsu method $\text{OTSUB}\text{INARYZATION}()$ is used to for thresholding. The marker information can be extracted by matching it with the marker dictionary. In certain detection poses, the robot may detect multiple markers, including those behind the glass. To ensure accurate and reliable positioning, the marker closest to the image center is selected to calculate the global localization of the swinging end of the BWCR using the proposed pose estimation algorithm. Additionally, the system can leverage the pre-extracted BIM-based environment map to validate the detected markers. By comparing the detected markers with their expected positions on the map, the system ensures that only relevant markers are utilized for localization.

Algorithm 2:

ArUco marker detection

As the robot’s tasks within wall environments are often dynamic, it cannot be ensured that markers remain within the camera’s field of view throughout traversal. The marker detection actions can be intentionally increased during the robot movement. Even if the robot’s pose deviates slightly from the planned path, executing detection movements significantly enhances the probability of locating markers for self-localization.

5.2. Climbing path correction of BWCR

The BWCR traverses the curtain wall environment following the planned path. Deviations in the initial pose and climbing motion may cause the BWCR to deviate from the planned path during its movement. As the number of climbing steps increases, foothold deviations accumulate, potentially causing the robot’s landing point to exceed the wall’s range, leading to a fall hazard. Thus, timely monitoring and correction of the robot’s motion deviations are required to ensure the BWCR continuously follows the planned path.

5.2.1. Climbing path correction model

The model of climbing path correction is depicted in Figure 6. A foothold received from the planner can be expressed in the base coordinate frame,

Figure 6.

Model of climbing path correction.

(15) \begin{align} {\vphantom{T}}^{W}_{F}{\hat{\boldsymbol{T}}}{}=\left[\begin{array}{l@{\quad}l} \hat{\boldsymbol{R}} & {}^{W}{\hat{\boldsymbol{P}}}{_{f}^{}}\\[3pt] \mathbf{0} & 1 \end{array}\right]=\left[\begin{array}{l@{\quad}l@{\quad}l@{\quad}l} \begin{array}{l} \hat{n}\\[3pt] 0 \end{array} & \begin{array}{l} \hat{o}\\[3pt] 0 \end{array} & \begin{array}{l} \hat{a}\\[3pt] 0 \end{array} & \begin{array}{l} {}^{W}{\hat{\boldsymbol{P}}}{_{f}^{}}\\[3pt] 1 \end{array} \end{array}\right] \end{align}

where the sign ^ denotes the ideal foothold. To make the actual foothold ${\vphantom{T}}^{W}_{F}{\boldsymbol{T}}{}$ as close to the ideal as possible, we have to minimize the distance of the attaching orientation and the attaching position, respectively.

Since the walls in this paper are plane, according to the geometric constraints of the robot’s adsorption, the attaching orientation of the robot is fixed at any position in the plane. Therefore, the corrected attaching orientation is consistent with the ideal,

(16) \begin{align} {}^{W}_{F}{\boldsymbol{R}}{}=\left[\begin{array}{l@{\quad}l@{\quad}l} \hat{n} & \hat{o} & \hat{a} \end{array}\right] \end{align}

For the position component, the attaching point must be on the wall and closest to the idea,

(17) \begin{align} \begin{array}{l} \underset{x,y,z}{\textit{argmin}}\,f\!\left(s,y,z\right)=\left\| {}^{W}{\hat{\boldsymbol{P}}}{_{f}^{}}-{}^{W}{\boldsymbol{P}}{_{f}^{}}\right\| \\[3pt] s.t.\,\,\left\{\begin{array}{l} {}^{W}{\boldsymbol{P}}{_{f}^{}}\in \omega _{i}\\[3pt] \exists \boldsymbol{q}=IK\left({\vphantom{T}}^{B}_{F}{\boldsymbol{T}}{}\right),\boldsymbol{q}\in \left[\underline{\boldsymbol{q}},\overline{\boldsymbol{q}}\right]\\[3pt] \Re \left(\boldsymbol{q}\right)\cap \boldsymbol{E}_{w}=\varnothing \end{array}\right. \end{array} \end{align}

where denotes the joint angle vector of BWCR and $[\underline{\boldsymbol{q}},\overline{\boldsymbol{q}}]$ are the joint limits, $IK()$ represents the robot inverse kinematics, while $\Re$ denotes the robot geometry calculated by its kinematics. The constraint $\Re (\boldsymbol{q})\cap \boldsymbol{E}_{w}=\varnothing$ means that the robot does not collide with the environment. Notably, ${\vphantom{T}}^{B}_{F}{\boldsymbol{T}}{}$ signifies the actual foothold in the base coordinate frame and is obtained

(18) \begin{align} {\vphantom{T}}^{B}_{F}{\boldsymbol{T}}{}=\left({\vphantom{T}}^{W}_{E}{\boldsymbol{T}}\,{\vphantom{T}}^{E}_{B}{\boldsymbol{T}}^{-1}\right)^{-1} {\vphantom{T}}^{W}_{F}{\boldsymbol{T}} \end{align}

where ${\vphantom{T}}^{W}_{E}{\boldsymbol{T}}{}$ can be calculated by Equation (14), while ${\vphantom{T}}^{B}_{E}{\boldsymbol{T}}{}$ can be read from the robot controller directly.

5.2.2. Deviation correction method

Figure 7 illustrates the proposed climbing path correction method. BWCR utilizes the onboard monitoring camera to detect the ArUco marker for global localization during each climbing step. If the marker cannot be detected, the robot initiates dead reckoning using forward kinematics for auxiliary positioning. Upon detecting a climbing motion deviation, the robot calculates the corrected foothold based on its current motion state and re-plans the swinging end’s trajectory to the new target. As vacuum adsorption necessitates precise alignment of all suction module suckers with the target surface to create an airtight chamber for vacuum generation, an autonomous pose detection and alignment method utilizing ultrasonic sensors is employed to ensure accurate wall adherence by the swinging end [Reference Zhu, Guan, Wu, Zhang, Zhou and Zhang37].

Figure 7.

The proposed climbing path correction method.

Several strategies are defined to enable the robot to perform long-duration climbing navigation:

• • Visual-based localization is the primary, supplemented by odometer positioning: The robot utilizes ArUco markers for visual localization, with the odometer activated simultaneously. If visual detection fails, the system automatically switches to dead reckoning to estimate the robot’s global pose.

• • Reliable adsorption must be ensured: Negative pressure adsorption imposes strict geometric constraints between the suction cups and the target surface. Therefore, the suction module’s reliable adhesion must be maintained during climbing, particularly when transitioning between walls.

• • Deviation correction within one step is the primary, supplemented by multi-step correction: If the motion deviation becomes excessive due to prolonged reliance on dead reckoning, the robot cannot correct the deviation in a single step and must invoke the path planner to compute multiple footholds for correction.

6.1. Robot system

The proposed algorithms were applied to localization and navigation within the self-developed W-Climbot, a modular biped climbing robot consisting of five joint modules and two suction modules [Reference Guan, Zhu, Wu, Zhou, Jiang, Cai, Zhang and Zhang5, Reference Zhu, Lu, Gu, Wei and Guan6]. Its configuration is depicted in Figure 1. A host personal computer (PC) serves as the robot controller, communicating with the distributed servo drivers of joint modules via the CAN bus, as illustrated in Figure 8. The suction module uses STM32F103 as the primary control chip to collect the sensor data, including pressure sensor, ultrasonic sensor, IMU, and camera sensors, and to transmit control signals for the vacuum pump and solenoid valve via wireless data and image transmission modules.

Figure 8.

Architecture of the robotic control system.

The BIM-based environment mapping method, robot localization algorithm, and climbing path correction method were executed directly on the host PC. A hierarchical motion planning method was implemented to generate the climbing path, connecting the starting point and the destination within the wall environment. The single-step motion planner was deployed to re-plan the collision-free movement swinging the robot suction module from the current pose to a pre-suction point, which was generated by offsetting the corrected foothold in the wall’s normal direction by a safe distance. The robot then detected the wall using the sensing system and aligned the suction module with the target wall.

6.2. Image wireless transmission and decoding

To enhance the climbing robot’s flexibility and minimize the impact of wired connections on its movement, a wireless high-definition image transmission system was developed. The hardware system primarily includes monitoring cameras, switches, and wireless image transmission modules. Monitoring cameras mounted on the two suction modules capture images and encode them into video streams compliant with the Real-Time Streaming Protocol (RTSP). These streams are transmitted via the Internet Protocol network to a wireless image transmission transmitter, where they are modulated into data packets using orthogonal frequency division multiplexing (OFDM). The packets are then sent to a ground-based wireless image transmission receiver. After OFDM demodulation, the receiver restores the data packets into a video stream with the RTSP protocol and transmits it to the computer. On the remote terminal controller of the W-Climbot, OpenCV image processing operators convert the video frames from YUV420 encoding to RGB format. These RGB frames are then utilized for video playback, image analysis, and robot localization.

6.3. Suction module control system

The W-Climbot control system on the ground side was developed using C++ and Python on the Linux platform, with the user interface (UI) designed using QT. The suction module control system integrates functions such as autonomous adsorption, video stream decoding, remote video monitoring, environmental structured map import, and status display. The monitoring video is displayed in real-time on the UI and can be switched based on the active adsorption module.

The control process of the suction module is shown in Figure 9. Initially, the robot control system executes initialization tasks, establishes communication with the suction modules, and loads the ArUco marker database. The appropriate suction module is then selected based on the generated climbing path. The alternate module serves as the swinging end, responsible for releasing pressure, disengaging from the wall surface, and transitioning to the next foothold. The UI plays the monitoring video captured by the camera in the corresponding suction module, while simultaneously loading the intrinsic calibration parameters and hand-eye calibration matrix of the camera. Upon detecting the ArUco marker, the swinging end autonomously localizes itself. After motion deviation correction, the swinging end reaches an intermediate observation point near each foothold, maintaining a safe distance from the curtain wall. The control system then collects data from various sensors in the suction module to compute deviations in autonomous adsorption, including the distance and angle between the suction module and the wall. Finally, the swinging end fine-tunes the angle deviation and moves parallel to the wall until the suction cup securely adheres to the surface. If the set pressure is not achieved, the suction module returns to the observation point and repeats the adsorption operation. The robot will continue transforming the suction end and performing the adsorption process until it reaches the destination.

Figure 9.

The suction module control method.

7.1. Map extraction from BIM

In this paper, Autodesk Revit, a BIM authoring tool, was used as the 3D modeling and secondary development platform to develop a Revit plug-in for automatically extracting the map of the curtain wall environment.

A total of 11 Revit models, containing architectural elements such as frames, ArUco markers, and glasses, were selected for testing, with some examples shown in Figure 10. The ArUco markers used in this study are black-bordered squares with an internal binary matrix and a unique ID, each measuring 40 mm × 40 mm. These markers were positioned arbitrarily on the glass surfaces.

Table I.

Results of data extraction based on BIM.

Figure 10.

Some testing scenarios for environment element extraction.

Table I presents the results of the data extraction. As shown, the extraction rates for frames, markers, and glass in all models are 100%. The time required for extraction increases progressively with the number of components. For instance, information extraction takes only 159.01 ms for the frame environment, which consists of 1680 elements. However, when the number of glass walls and markers reaches 100, the extraction time increases to 4275.05 ms. From the 10th data group, it is evident that the extraction time is primarily spent on the glass walls. Reducing the number of glass walls significantly decreases the time consumption. This is because the storage method for glass walls in Revit is complex. Moreover, a glass wall is essentially a cuboid made up of six surfaces. During extraction, the two largest adsorbable walls are selected first, followed by the selection of the side with the ArUco markers as the attachment surface.

7.2. Evaluation of positioning accuracy of BWCR

To verify the positioning accuracy and stability of global visual localization using ArUco markers, two sets of evaluation experiments were conducted, as shown in Figure 11. The UR3 robot, chosen as the experimental platform due to its superior motion precision and stability compared to W-Climbot, ensures precise and reliable results. A suction module, integrated with a manufactured probe, was mounted on the end of the UR3 robot to observe the ArUco marker placed on the workbench. After moving the robot’s Tool Center Point (TCP) to the end of the probe, the transformation matrix ${\vphantom{T}}^{A}_{C}{\boldsymbol{T}}{}$ between the camera and the new TCP was obtained via hand-eye calibration. By touching the four corners of the ArUco marker with the probe tip and reading its coordinates from the robot controller, the pose of the marker in the robot’s base coordinates frame ${\vphantom{T}}^{B}_{A}{\boldsymbol{T}}{}$ was determined. The suction module then moved to capture the relative pose of the camera using the proposed visual localization method. The positioning pose of the TCP was calculated by

Figure 11.

Experimental setup used to evaluate the location accurately.

(19) \begin{align} {\vphantom{T}}^{B}_{E}{\boldsymbol{T}}{}\boldsymbol{=} {\vphantom{T}}^{B}_{A} {\boldsymbol{T}}\,{\vphantom{T}}^{A}_{C}{\boldsymbol{T}}\,{\vphantom{T}}^{C}_{E} {\boldsymbol{T}}{} \end{align}

the positioning accuracy was evaluated by comparing it with the robot TCP pose ${\vphantom{T}}^{B}_{E^{\prime}}{\boldsymbol{T}}$ read from the robot controller directly,

(20) \begin{align} Err\boldsymbol{=} {\vphantom{T}}^{B}_{{E}^{\prime}}{\boldsymbol{T}} - {\vphantom{T}}^{B}_{E}{\boldsymbol{T}} \end{align}

To assess the stability of the positioning method, the suction module was placed in the same pose for multiple observations to examine the fluctuation in the localization data. The camera plane, positioned parallel to the marker, was placed at distances of 120 mm, 170 mm, and 220 mm with 10 repeated measurements for each. The results are presented in Table II. On average, the positional error of the TCP was within 2.2 mm, while the average orientation error was within 2 degrees. As the distance between the camera and the marker increases, the positioning error also increased. The maximum errors for both position and orientation occurred at the 220 mm, with values of 2.88 mm and 3.14 degrees, respectively. This increase in error is attributed to the decrease in image quality as the distance increases, which reduces positioning accuracy.

Table II.

Results of localization stability test.

To evaluate the positioning accuracy, the suction module was placed at different positions within the UR3 workspace for global visual localization, and the positioning error distribution was analyzed. During the experiment, the robot TCP was moved to the geometry center of the suction module’s bottom plane. The center point of the marker was considered the center of the sphere, with the distance between the robot TCP and the marker serving as the sphere’s radius. Observation points were selected along the spherical surfaces to obtain positioning data, as illustrated in Figure 12. In this experiment, 22 groups of data were collected along the x-axis and y-axis on two spherical surfaces with radii of 120 mm and 170 mm, respectively, and the corresponding positioning errors were recorded, as shown in Figure 13. Overall, the positioning accuracy was minimally affected by spatial angle, with the position error at all observed points remaining below 3 mm, and the angle error within 3 degrees.

Figure 12.

The accuracy evaluation of spatial multi-pose positioning.

Figure 13.

Evaluation results of positioning accuracy with a radius of 120 mm. (a) Position error. (b) Angle error.

7.3. Multi-step climbing

To evaluate the feasibility and effectiveness of the visual localization and path correction method based on ArUco markers and BIM, this subsection performed a comprehensive climbing comparison experiment involving W-Climbot climbing multiple steps on different wall surfaces.

Figure 14 depicts the experimental scenario. A 3D frame environment was constructed using poles of varying lengths connected by fasteners. Glass walls were affixed to the frame via surrounding connecting claws, and ArUco markers were applied to each wall surface in a grid pattern. The experimental environment was positioned within the working range (8 m × 7 m × 3 m) of the OptiTrack motion capture system, which offered a tracking accuracy of 0.1 mm. The motion capture system monitored the robot’s motion trajectory by tracking markers attached to both ends of the W-Climbot. Two sets of climbing experiments were conducted: one utilizing the proposed navigation scheme with visual localization and path correction, and the other serving as a comparative test without localization. Both experiments followed the same climbing path, allowing verification of the scheme’s effectiveness by comparing the actual trajectory with the theoretically planned trajectory and evaluating the deviations between the two sets of footholds. The robot’s climbing cycle is depicted in Figure 15.

Figure 14.

Multi-step climbing environment scenario.

Figure 15.

W-Climbot climbed multiple steps consecutively in a wall scenario.

Figure 16 shows the experimental results. The deviations observed in the footholds of both groups on wall w ₁ were attributed to initial placement errors, which were random and did not influence the experimental outcomes. As depicted in Figure 17, the deviation in the third foothold increased significantly, exceeding 120 mm. This was primarily due to the initial error and the transition between walls. In the visual localization experiment, the robot exhibited a larger deviation in the third step, which was attributed to the amplified initial error, indicating a noticeable error accumulation during the multi-step climbing process of BWCR. After incorporating visual localization, the robot’s foothold error decreased markedly, with a deviation of 34 mm in the fourth step compared to 82.5 mm in the experiment without visual localization. For the final foothold in the multi-step climbing sequence, the deviation without visual localization exceeded 170 mm, significantly diverging from the theoretical trajectory reference. In contrast, the foothold deviation in the visual localization experiment remained around 50 mm and demonstrated a trend toward stabilization.

Figure 16.

Comparison of the actual and the planned climbing motion. (a)Without visual localization and path correction. (b) With visual localization and path correction.

Figure 17.

Foothold errors during multi-step climbing for the two group experiments.

In summary, the robot’s foothold error is significantly larger in the absence of visual localization and path correction, with the error accumulation effect becoming increasingly pronounced as the number of climbing steps increases. Conversely, the robot employing visual localization maintains a single-step foothold error within 58.1 mm. Furthermore, the average deviation across the robot’s final six steps is reduced by 44.39%, while the maximum deviation decreases by 65.98%.