1. Introduction
Wireless capsule endoscope (WCE) is an effective non-invasive tool for screening gastrointestinal (GI) diseases and related cancers. With the camera placed on the small-sized WCE, continuous images from the GI tract can be captured and transmitted to a data logger carried by the patients [Reference Iddan, Meron, Glukhovsky and Swain1–Reference Martin, Scaglioni, Norton, Subramanian, Arezzo, Obstein and Valdastri3]. In recent years, WCE has been accepted as a routine clinical procedure alternative to conventional endoscopy. Nevertheless, the most current commercial capsule endoscopes provide only the functions of image transmission and passive navigation inside the human GI tracts [Reference Gao, Hu, Chen, Liu and Zhang4, Reference Menciassi5]. The accurate localization of WCEs in real-time is still missing, which could significantly enhance their functionalities in diagnostic and therapeutic applications, such as localization of the malignant lesion, closed-loop feedback control of WCEs when navigating in the GI tracts, biopsy, fixed-point drug delivery, and non-invasive surgery [Reference Ciuti, Valdastri, Menciassi and Dario6–Reference Gao, Yan, He, Xu and Wang13].
The clinical practice and state-of-the-art of localization methods of WCEs mainly include ultrasonic imaging, X-ray imaging, inertial navigation, and magnetic localization [Reference Figueiredo, Leal, Pinto, Figueiredo and Tsai14–Reference Than, Alici, Zhou and Li16]. Among them, positioning the WCE based on the magnetic field is the most widely used and efficient approach. Magnetic localization can be achieved by remote sensing the WCE that has an embedded small permanent magnet. As demonstrated in many previous studies, a magnetic sensor array can be placed externally to read the magnetic field information from the embedded permanent magnet in the WCE to solve the position [Reference Hu, Ren, You, Yang, Song, Xiang, He, Zhang and Meng17, Reference Wang, Song, Liu and Meng18]. Based on the magnetic dipole model, the pose of the embedded permanent magnet can be obtained by minimizing the difference between the measured data of each magnetic sensor and the theoretical magnetic field, which is equivalent to the position of the capsule. However, the positioning of WCEs by a magnetic sensor array is mainly for passive capsules. The positioning system is relatively complex and bulky, which requires a wearable sensor array.
Alternatively, magnetic localization can be achieved by onboard sensing of an external magnetic source that generates either static magnetic fields (permanent magnet), alternating magnetic fields (electromagnetic magnets), or hybrid static and alternating magnetic fields [Reference Lin, Zhang, Lin, Chang and Xu19–Reference Karargyris and Koulaouzidis22]. In many previous studies, the permanent magnet is placed outside the WCE to make it more compatible with external magnetic manipulation of WCEs. The pre-calculated magnetic field information can be used to search the closest position [Reference Salerno, Ciuti, Lucarini, Rizzo, Valdastri, Menciassi, Landi and Dario23, Reference Di Natali, Beccani and Valdastri24]. For positioning based on the electromagnetic wave, radio frequency identification (RFID) is a well-used method. With RFID technology, the capsule could be attached to a tag (transponder) and localized by an external reader. However, due to the fluctuation of path loss in the digestive tract environment, the positioning errors are inconsistent, which could be either less than 3 mm or more than 20 mm [Reference Mateen, Basar, Ahmed and Ahmad25]. In addition, the balance between the choice of working frequency and data transmission has always been a challenge [Reference Xiao, Liang and Yang26]. Another drawback is that the attenuation of radio frequency signals within the human body is irregular. Therefore, it cannot be adequately compensated and calibrated. In order to solve this problem, it is necessary to increase the size of the transmitter and receiving antenna, which poses great challenges for the manufacture of capsules.
Apart from these, Islam et al. achieved the positioning using the electromagnetic fields [Reference Islam and Fleming27]. The positioning system consists of three orthogonal transmission coils and three orthogonal sensing coils, where the sensing coils are embedded into the capsule while the transmission coils are placed outside the body. Each transmission coil works as a magnetic dipole to generate a magnetic field. Thus, the position can be solved through the magnetic field data received by the sensing coils. For this method, the size of the inner sensing coils and the design of the outer transmission coils have limitations on the miniaturization of the WCE and insufficient workspace. The positioning method based on the electromagnetic fields shows a more accurate position result than that based on the static magnetic fields [Reference Franz, Haidegger, Birkfellner, Cleary, Peters and Maier-Hein28]. Taddese et al. have proposed a positioning based on hybrid magnetic fields (an assembly of static and time-varying magnetic field sources outside the WCE), where magnetic sensors and permanent magnets are embedded in the capsule endoscope to be localized and actuated [Reference Taddese, Slawinski, Pirotta, De Momi, Obstein and Valdastri29]. The hybrid approach eliminates the singularity of the workspace and uses particle filters to perform the pose estimation. This method is beneficial for the active manipulation of the capsule, and a positioning accuracy within 10 mm can be achieved. To sum up, magnetic localization of WCEs based on onboard sensing units has shown significant advantages in terms of accuracy, ease of deployment, and compatibility with external magnetic manipulation. However, a well-calibrated analytical model is still missing for the accurate localization of WCEs by the measurements from different onboard sensing units.
In this paper, we employ a localization strategy based on an external permanent magnet (EPM) mounted on a robotic arm and a WCE with an embedded magnetic sensor and an inertial measurement unit (IMU). The EPM on the robotic arm manipulates the WCE in the GI tract, meanwhile, provides a reference for localization of the WCE. The orientation of the WCE can be obtained by the IMU module. The position is solved by the Jacobian-based iterative method. An error model of the localization system based on the law of error propagation is built, and the accuracy of the error analysis model is verified through experiments. The magnetic field distribution around the used EPM was carefully measured with a high-resolution 3D magnetic field measuring instrument. Additionally, sensor calibration and parameter optimization further improve the positioning accuracy of the positioning system. The maximum average error was about 6.29 mm when the magnetic source-sensor distance was from 150 to 350 mm. The average error for the capsule orientation angles, obtained by fusing the accelerometer and gyroscope, was 2.93°. The main contributions of this work include the following:
-
(a) Compared with the external magnetic sensor array method, our onboard sensor fusion method provides a simple, compact, and real-time approach that does not require multiple magnetic sensors for attitude calculation and needs only one six-axis IMU module and one three-axis magnetic sensor. Besides, there is no need for patients to wear redundant sensor array vests. The proposed method is compatible with magnetic manipulation systems and could be easily deployed and integrated with other diagnostic and therapeutic platforms for GI tracts.
-
(b) Most magnetic field models were built based on finite element simulation data or limited experimental data. In this study, the actual magnetic field information from the whole working space was measured with a magnetic field resolution of 0.1
$\mu$
T and spatial resolution of 10
$\mu$
m to optimize the magnetic field models. It is found that the magnetic dipole model is better than the finite element integration method in the current experimental settings. Meanwhile, the Jacobian-based iterative method could use the real-time magnetic field data to get the inverse solution to position directly to increase the calculation speed. -
(c) The magnetic positioning of the WCE is generally based on particle filtering, particle swarm optimization, and other methods for location prediction and neighborhood search. These methods could only be verified by obtaining the positioning information from the hardware platform, and it is difficult to predict and evaluate through simulation. In addition, there is no general error model when analyzing positioning errors, which means that the optimization can only be deduced from the final positioning results. In this paper, we apply a theoretical error model to analyze the magnetic positioning method, which provides a straightforward approach and demonstrates the effectiveness and feasibility of error propagation.
The organization of this paper is as follows. In Section 2, we introduced the principle and model of the localization system and proposed the error transfer model. Furthermore, we optimized the parameter in the magnetic dipole model. In Section 3, we presented the hardware and experimental setup. In Section 4, the localization errors in different motion experiments were analyzed. We compared different magnetic field analytical models further to prove the effectiveness of the magnetic dipole model. Besides, the error model was verified by performing both rotational and translational motions. At last, the discussion and conclusion are followed in Section 5.
2. Method and modeling
2.1. Localization method
Localization of a magnetic field source has been widely employed for magnetic tracking and motion capture [Reference Eshkevari and Sadough30–Reference Cavlu and Brennan32]. Our localization system consists of an IMU, a triaxial magnetic sensor embedded inside the capsule endoscope, and an external cylindrical permanent magnet. Within the operation range, the magnetic field generated around the cylindrical permanent magnet can be used to manipulate the magnetic WCE. Meanwhile, the relative position and attitude can be derived from the onboard sensing units in the WCE. In this paper, we focus on the localization of WCEs by determining the relative position between the EPM and the WCE based on the embedded magnetic sensor. An appropriate equivalent model is crucial for the accurate prediction of the magnetic field and field gradient at a specific position relative to the permanent magnets, hence the position estimation or the magnetic interaction between permanent magnets. Till now, different magnetic field models have been developed to express the magnetic fields produced by a magnetic source, such as the magnetic dipole model, equivalent magnetic current model, and equivalent magnetic charge model. The magnetic dipole model is a simplified mathematic model used for calculating the magnetic strength of the magnet source. However, the magnetic field produced by the dipole model is considerably unreliable at intermediate and shorter distances than their own size. Here, we use the magnetic dipole model to explain the localization method. The fitness error of the magnetic field produced by the used permanent magnet and the magnetic dipole model was experimentally quantified at a specified working space.
Based on the magnetic dipole model (a time-invariant nonlinear mathematical expression) shown in Eq. (1), the position of the magnetic sensor inside the capsule is mapped to the magnetic field data that was read from the capsule endoscope.
\begin{equation} \boldsymbol{{B}}=\frac{\mu _0}{4\pi }\frac{||\boldsymbol{{M}}_{\boldsymbol{{E}}}||}{||\boldsymbol{{r}}||^3}\left(3\tilde{\boldsymbol{{r}}}\tilde{\boldsymbol{{r}}}^T\tilde{\boldsymbol{{M}}_{\boldsymbol{{E}}}}-\tilde{\boldsymbol{{M}}_{\boldsymbol{{E}}}}\right)=g(\boldsymbol{{r}}) \end{equation}
where
$\mu _0$
is the vacuum permeability,
$\boldsymbol{{r}}$
is the vector from the EPM to the capsule,
$\tilde{\boldsymbol{{r}}}$
is the unit vector along
$\boldsymbol{{r}}$
,
$\boldsymbol{{M}}_{\boldsymbol{{E}}}$
is the dipole moment of the EPM, and
$\tilde{\boldsymbol{{M}}_{\boldsymbol{{E}}}}$
is the unit vector along
$\boldsymbol{{M}}_{\boldsymbol{{E}}}$
.
The implementation of Eq. (1) requires that the magnetic field data
$\boldsymbol{{B}}$
, target point position
$\boldsymbol{{r}}$
, and magnetic dipole moment
$\boldsymbol{{M}}_{\boldsymbol{{E}}}$
are under the coordinate system of the permanent magnet. Therefore, we transform all the parameters to the coordinate system of the permanent magnet. Figure 1 shows the relative positioning between the EPM, magnetic sensor, and IMU sensor. Figure 2 shows the flow chart for the position tracking based on the onboard magnetic sensor and IMU sensor. After initialization, the coordinate transformation matrix relative to the initial state can be obtained based on the IMU information, which is the attitude of the capsule. Then, the attitude transformation matrix
$R_{\text{IMU}}$
is used to transform the magnetic sensor data
$\boldsymbol{{B}}_{\boldsymbol{{s}}}$
to the permanent magnet coordinate
$\boldsymbol{{B}}$
. Finally, according to the Jacobian-based iterative method, the location of the magnetic sensor center can be iteratively derived under the permanent magnet coordinate, equivalent to obtaining the position of the WCE. The Jacobian-based iterative method is performed as follows [Reference Di Natali, Beccani, Simaan and Valdastri33]. Assuming the capsule moves from position
$\boldsymbol{{r}}_{\boldsymbol{{i}}}$
to
$\boldsymbol{{r}}_{\boldsymbol{{i}}+1}$
in a short time, the distance
$\boldsymbol{\Delta} \boldsymbol{{r}}$
produces a magnetic field change from
$\boldsymbol{{B}}_{\boldsymbol{{i}}}$
to
$\boldsymbol{{B}}_{\boldsymbol{{i}}+1}$
. Based on the first-order Taylor expansion, we can obtain:
\begin{equation} \boldsymbol{{B}}_{\boldsymbol{{i}}+1}=\boldsymbol{{B}}_{\boldsymbol{{i}}}+\frac{\partial \boldsymbol{{B}}_{\boldsymbol{{i}}}}{\partial \boldsymbol{{r}}} \cdot \boldsymbol{\Delta} \boldsymbol{{r}}=\boldsymbol{{B}}_{\boldsymbol{{i}}}+J \cdot \boldsymbol{\Delta} \boldsymbol{{r}} \end{equation}
where
\begin{align*} \dfrac{\partial \boldsymbol{{B}}_{\boldsymbol{{i}}}}{\partial \boldsymbol{{r}}}& =J=\begin{bmatrix} \dfrac{\partial B_x}{\partial r_x} & \quad \dfrac{\partial B_x}{\partial r_y} & \quad \dfrac{\partial B_x}{\partial r_z}\\[10pt] \dfrac{\partial B_y}{\partial r_x} & \quad \dfrac{\partial B_y}{\partial r_y} & \quad \dfrac{\partial B_y}{\partial r_z}\\[10pt] \dfrac{\partial B_z}{\partial r_x} & \quad \dfrac{\partial B_z}{\partial r_y} & \quad \dfrac{\partial B_z}{\partial r_z} \end{bmatrix} \end{align*}
(a) The schematic of the experimental platform for localization of wireless capsule endoscope based on onboard sensing. (b) The coordinate system of the external permanent magnet
$\{EPM\}$
. (c) The coordinate system within the capsule, where
$\{C\}$
represents the WCE coordinate system,
$\{Sensor\}$
represents the internal magnetic sensor coordinate system, and
$\{IMU\}$
represents the IMU coordinate system.
Flow chart for position tracking based on magnetic sensor and IMU sensor.
The inverse solution relation of the magnetic field at the present position of WCE can be obtained by reversing Eq. (2). Then, the relationship becomes:
\begin{equation} \boldsymbol{{r}}_{\boldsymbol{{i}}+1}=\boldsymbol{{r}}_{\boldsymbol{{i}}}+J^{-1} \cdot \boldsymbol{\Delta} \boldsymbol{{B}}_{\boldsymbol{{i}}} \end{equation}
where
$\boldsymbol{\Delta} \boldsymbol{{B}}_{\boldsymbol{{i}}} = \boldsymbol{{B}}_{\boldsymbol{{i}}+1}-\boldsymbol{{B}}_{\boldsymbol{{i}}}$
,
$J^{-1}$
is the inverse of the Jacobian,
$\boldsymbol{{r}}_{\boldsymbol{{i}}}$
is the previous position, and
$\boldsymbol{{r}}_{\boldsymbol{{i}}+1}$
is the current position to be solved. The maximum refresh rate of the IMU is 200 Hz, which is fast enough to ensure small movements of the WCE during subsequent attitude measurements.
2.2. Parameter optimization of magnetic dipole model
An accurate analytical magnetic field model is extremely important to obtain closed-form and fast estimates of the magnetic field. By incorporating high-resolution spatial experimental magnetic field mapping, we are able to optimize the magnetic field model and assess the relative error. Finally, an iterative magnetic localization of the WCE can be achieved by the closed-form expression for the Jacobian of the magnetic field relative to changes in the WCE pose. The Jacobian-based iterative method for magnetic localization provides an iterative localization at a short computational time. However, the accuracy can still be significantly affected by the magnetic field model. Different dimensions of EPM have different fitness errors from the magnetic dipole model. For the axial magnetized cylindrical permanent magnet with an aspect ratio of 1, the matching error is approximately 7
$\%$
when the positioning distance is greater than twice the radius [Reference Petruska and Abbott34, Reference Fang and Son35]. The magnetic dipole moment is the only parameter in the magnetic dipole model, which needs to be accurately determined for the used permanent magnet. Unlike conventional studies that obtain the magnetic dipole moment either from the datasheet by the permanent magnet manufacturer or from finite element simulation, we employ a high-resolution gauss meter probe mounted on motorized stages for measurement of the 3D magnetic field at different regions above the EPM. A cost function is established to obtain the optimal magnetic dipole moment by minimizing the cost function. The cost function is constructed by the difference between the magnetic field data pre-calculated by the magnetic dipole model and the magnetic field collected by the sensor:
\begin{equation} \min J(\boldsymbol{{M}}_{\boldsymbol{{E}}})=\sum _{i=1}^N{\Big\|r_i^3({\boldsymbol{{B}}_{\boldsymbol{{i}}}(\boldsymbol{{M}}_{\boldsymbol{{E}}},\boldsymbol{{r}}_{\boldsymbol{{i}}})-\boldsymbol{{b}}_{\boldsymbol{{i}}}})\Big\|} \end{equation}
where
$\boldsymbol{{M}}_{\boldsymbol{{E}}}$
is the magnetic dipole moment,
$\boldsymbol{{B}}_{\boldsymbol{{i}}}$
is the magnetic field data pre-calculated by the magnetic dipole model (Eq. (1)),
$r_i$
is the distance between the sampling point and the EPM center,
$\boldsymbol{{b}}_{\boldsymbol{{i}}}$
is the magnetic field collected by the magnetic sensor, and
$N$
is the number of sampling points. Since
$\boldsymbol{{B}}$
is proportional to the
$r^3$
, we add a gain
$r_i^3$
to the cost function to ensure the same weight of each sample point in the optimization region.
We sampled the space of 200
$\times$
200
$\times$
150 mm around 7 cm above the surface of the EPM with an interval of 10 mm as a data set to optimize the
$\boldsymbol{{M}}_{\boldsymbol{{E}}}$
using the above cost function. Then, we used the “fmincon” function in MATLAB to obtain the optimal magnetic dipole moment. Based on the measured data, the average fitness error of the EPM is 7.58
$\%$
in the whole workspace.
2.3. Theoretical error analysis
Theoretical error analysis can provide broad insights for prediction and guidance. In an engineering system, there are usually two kinds of variables: direct measurement quantity and indirect measurement quantity. The direct measurement quantity is the data that can be obtained by the sensor as the input of the system. The indirect measurement is the output of the system, that is, the desired result. The indirect measurement can usually be constructed through various models to establish the relationship with the direct measurement. In order to analyze the errors of different models or different parameters in an individual model, it is often necessary to build a hardware platform for the verification of the preliminary test. However, an appropriate theoretical error model could save time and cost to predict the system error only through simple operations. The influence of direct measurement error on indirect measurement error is based on error propagation. The propagation of errors is essential for understanding how uncertainties in a parameter affect calculations using that parameter. Error is propagated to the solution through a set of rules.
The error analysis is based on two theoretical premises. For the given positioning system of the WCE, there are three direct measurements, namely magnetic field information
$B$
, angular velocity
$\omega$
, and acceleration data
$a$
. The indirect measurement is the position of the WCE. There is a specific relationship between the indirect measurement, position
$P$
, and the direct measurements
$B$
,
$\omega$
and
$a$
, namely
$ P=f(B,\omega,a)$
, where magnetic field
$B$
, acceleration
$a$
, and angular velocity
$\omega$
are independent with each other. Then, the error of the indirectly measured result
$P$
is the total differential of the error of
$B$
,
$\omega$
and
$a$
:
\begin{equation} e_P=\left|\frac{\partial f}{\partial B}\right|e_{B}+\left|\frac{\partial f}{\partial \omega }\right|e_{\omega }+\left|\frac{\partial f}{\partial a}\right|e_{a} \end{equation}
where
$e_{B}$
,
$e_{a}$
, and
$e_{\omega }$
are the errors of each direct measurement,
$e_P$
is the error of the indirect measurement
$P$
.
For the error of each direct measurement, there is systematic error
$e_s$
and random error
$e_r$
. Generally, we consider that the random error follows a normal distribution, in which the mathematical expectation of the random error
$E(e_r )=0$
. The correlation between random error and systematic error is small and can be ignored after some measurements. Therefore, the error of the directly measured quantity can be regarded as its systematic error,
$e=e_s$
[Reference Wu, Tang and Tang36].
We derived the transfer relationship from the error of the original measurement data to the error of the final positioning. The absolute positioning error of the WCE is equivalent to the positioning error of the magnetic sensor.
The attitude matrix is a function of quaternion, which is calculated from the IMU accelerometer and gyroscope:
\begin{equation} R_{\text{IMU}} = f(q)=f(q(a,g)) \end{equation}
\begin{equation} f(q) = \begin{bmatrix} q^2_0+q^2_1-q^2_2-q^2_3 & 2\!\left(q_1q_2+q_0q_3\right) & 2\!\left(q_1q_3-q_0q_2\right)\\[7pt] 2\!\left(q_1q_2-q_0q_3\right) & q^2_0-q^2_1+q^2_2-q^2_3 & 2\!\left(q_2q_3+q_0q_1\right)\\[7pt] 2\!\left(q_1q_3+q_0q_2\right) & 2\!\left(q_2q_3-q_0q_1\right) & q^2_0-q^2_1-q^2_2+q^2_3 \end{bmatrix} \end{equation}
\begin{equation} q_{t_0+\Delta T}(g)= q_{t_0} + \begin{bmatrix} -q_{1,t_{0}} & \quad -q_{2,t_{0}} & \quad -q_{3,t_{0}}\\[5pt] q_{0,t_{0}} & \quad -q_{3,t_{0}} & \quad q_{2,t_{0}}\\[5pt] q_{3,t_{0}} & \quad q_{0,t_{0}} & \quad -q_{1,t_{0}}\\[5pt] -q_{2,t_{0}} & \quad q_{1,t_{0}} & \quad q_{0,t_{0}} \end{bmatrix} \cdot g \cdot \Delta T \end{equation}
\begin{equation} g_{t_0+\Delta T}(a)= g_{t_0}+K_p \cdot e_{t_0}\cdot a +\sum _{k=0}^{t_0} K_i \cdot e_k\cdot a \end{equation}
where
\begin{equation*}e_k=\begin {bmatrix} 0 & \quad S_{z,k} & \quad -S_{y,k}\\[5pt] -S_{z,k} & \quad 0 & \quad S_{x,k}\\[5pt] S_{y,k} & \quad -S_{x,k} & \quad 0 \end {bmatrix} \end{equation*}
and
\begin{equation*}\begin {bmatrix} S_x\\[5pt] S_y\\[5pt] S_z \end {bmatrix} = \begin {bmatrix} 2\!\left(q_{1,t_{0}}q_{3,t_{0}}-q_{0,t_{0}}q_{2,t_{0}}\right)\\[5pt] 2\!\left(q_{0,t_{0}}q_{1,t_{0}}+q_{2,t_{0}}q_{3,t_{0}}\right)\\[5pt] q^2_{0,t_{0}}-q^2_{1,t_{0}}-q^2_{2,t_{0}}+q^2_{3,t_{0}} \end {bmatrix} \end{equation*}
$q$
is the quaternion,
$t_0$
is the current time,
$g$
is the reading of the gyroscope,
$\Delta T$
is the time interval (constant), and
$a$
is the reading of the accelerometer.
$K_p$
is the proportional parameter, and
$K_i$
is the integral coefficient of the PI compensation, respectively.
Because of the PI compensation, the quaternion calculation is a process of continuous iteration within the set time interval. The quaternion at the present moment is formed by the function of the quaternion at the previous moment and the angular velocity, while the angular velocity is calculated by the quaternion at the previous moment and the acceleration. Obviously, there is a cumulative error in quaternion calculation over time. Hence, the application of the error transfer model fails. As for the attitude, we conducted error analysis and optimization according to the actual experimental data.
The position solution can be obtained by the Jacobian-based iterative method based on the magnetic dipole model (Eq. (1)), where
$\boldsymbol{{M}}_{\boldsymbol{{E}}}$
can be considered as a constant for a given magnet. Thus, the magnetic field
$\boldsymbol{{B}}$
is merely a function
$g$
of position
$\boldsymbol{{r}}$
. Further, the position
$\boldsymbol{{r}}$
can be regarded as a function of the magnetic field data
$\boldsymbol{{B}}$
, that is,
$\boldsymbol{{r}}=g^{-1}(\boldsymbol{{B}}_{\boldsymbol{{E}}})$
. Then, the error is:
\begin{equation} \boldsymbol{{e}}_{\boldsymbol{{r}}} = \frac{\partial g^{-1}}{\partial \boldsymbol{{B}}_{\boldsymbol{{E}}}}\boldsymbol{{e}}_{\boldsymbol{{B}}_{\boldsymbol{{E}}}} \end{equation}
where
\begin{align*} \dfrac{\partial g^{-1}}{\partial \boldsymbol{{B}}_{\boldsymbol{{E}}}}& \quad =J^{-1}=\begin{bmatrix} \dfrac{\partial r_x}{\partial B_x} & \quad \dfrac{\partial r_x}{\partial B_y} & \quad \dfrac{\partial r_x}{\partial B_z}\\[8pt] \dfrac{\partial r_y}{\partial B_x} & \quad \dfrac{\partial r_y}{\partial B_y} & \quad \dfrac{\partial r_y}{\partial B_z}\\[8pt] \dfrac{\partial r_z}{\partial B_x} & \quad \dfrac{\partial r_z}{\partial B_y} & \quad \dfrac{\partial r_z}{\partial B_z} \end{bmatrix} \end{align*}
3. Hardware and experimental setup
As shown in Fig. 3(a), the capsule endoscope used in our experiments is 28.44 mm in length, 11.68 mm in diameter, and 2.37 g in weight, which satisfies the general requirements for WCEs in size. The capsule is a prototype from JIFU Medical, which is equipped with a 3-axis magnetic field sensor (AK09970N), an IMU (BMI160), and a camera. The magnetic field sensor has a wide sensing range of ±39 mT, a sensitivity of 3.1
$\mu$
T per least significant bit, and time for measurement of less than 1 ms, which provides a fast response and a sufficient sensing range and precision for localization of magnetically actuated WCEs or closed-loop remote magnetic manipulation of a WCE inside the patient’s body. The capsule can transmit images, magnetic field, acceleration, angular velocity, and other information wirelessly to the external receiver. Then, the information can be processed to determine the position of the capsule. The EPM used in this study is a rubidium-iron-boron N52 cylindrical permanent magnet, which is 50 mm in diameter and 50 mm in height. At 100 mm away from its surface, it can produce a magnetic field of approximately 10 mT, which is sufficient for external magnetic manipulation. Apart from the capsule endoscope and magnet in the positioning system, a three-axis rotating stage (Fig. 3(a)), a high-resolution three-dimensional magnetic field measuring instrument (CH-Magnetoelectricity Technology Co., Ltd, F-30, displacement resolution is 10
$\mu$
m, and magnetic field resolution is 0.1
$\mu$
T.), and a Helmholtz coil (Fig. 3(b)) were used for the implementation of experiments. The 3D magnetic field measuring instrument (Fig. 3(c)) could set sampling space, sampling interval, and sampling track, automatically scan the spatial magnetic field, and save the collected 3D magnetic field data and corresponding spatial information.
Hardware platform. (a) Three-axis rotating stage and the capsule endoscope. (b) Uniaxial Helmholtz coil. (c) Three-dimensional magnetic field measurement instrument that incorporates fine positioning stages and high-resolution three-axis hall magnetic sensor probe.
4. Experiments and error analysis
4.1. Sensor calibration
For a low-cost generic IMU, the accuracy is often affected by inaccurate scaling factors, non-verticality of the sensor axes, and non-zero biases. IMU calibration is the process of defining these quantities [Reference Tedaldi, Pretto and Menegatti37]. Assuming that we ignore ambient noise, the complete sensor error model is as follows:
\begin{equation} a_c = T \cdot K \cdot (a_o+b) \end{equation}
where
\begin{align*} T=\begin{bmatrix} 1 & \quad -\beta _{yz} & \quad \beta{zy}\\[5pt] 0 & \quad 1 & \quad -\beta _{zx}\\[5pt] 0 & \quad 0 & \quad 1 \end{bmatrix}, K=\begin{bmatrix} k_x & \quad 0 & \quad 0\\[5pt] 0 & \quad k_y & \quad 0\\[5pt] 0 & \quad 0 & \quad k_z \end{bmatrix}, b =\begin{bmatrix} b_x\\[5pt] b_y\\[5pt] b_z \end{bmatrix} \end{align*}
$a_o$
is the original uncalibrated data,
$T$
is the conversion matrix of the sensor axis,
$K$
is the scale factor matrix of the reading,
$b$
is the bias, and
$a_c$
is the calibrated sensor data.
$\beta$
is the deflection angle from the actual coordinate system to the ideal vertical coordinate system. The details of IMU calibration can be found below [Reference Tedaldi, Pretto and Menegatti37]. Step 1: Set the sensor for 5 s to obtain zero offsets; Step 2: Rotate the sensor and place it in different attitudes, and stand for at least 3 s in each attitude; Step 3: Rotate the IMU to at least six different attitudes; Step 4: Perform the estimation algorithm to obtain the
$T$
,
$K$
, and
$b$
. Similar to the calibration of IMU, the magnetic sensor was placed in a uniform magnetic field generated by the Helmholtz coils, as shown in Fig. 3(b). Steps 1 to 4 were repeated to obtain the calibration parameters
$T$
,
$K$
, and
$b$
of the magnetic sensor.
4.2. Comparison of different magnetic field analytical models
As discussed in the previous section, an accurate analytic relationship between position and the magnetic field is of great importance for the proposed localization of WCEs. Here, we used the measured magnetic field data as the ground truth to compare two magnetic field analytical models. One is the magnetic dipole model we applied to the localization method. The other is the magnetic current model based on the finite element integration method. The magnetic field in a 200
$\times$
200
$\times$
150 mm space 7 cm above the surface of the EPM was measured. The interval along the Z direction (Corresponding to the axial direction of the cylindrical permanent magnet) is 10 mm. The sample points with the same Z distance composed a layer. The intervals along the X and Y direction in each layer are also 10 mm (Choose the center of the EPM as the origin, sampling from −100 to 100 mm).
By fitting the measurement data, we found that both magnetic field models agreed well with the experimental measurement. We selected a set of measurement data from the same distance (one layer) above the permanent magnet as a typical magnetic field distribution. As shown in Fig. 4(a), both models demonstrated a high matching degree of the measured data. The fitness error of the magnetic dipole model is smaller. Further, we calculated all the layers’ matching errors, as shown in Fig. 4(b). When the distance in the Z direction is less than 80 mm, the matching error of the magnetic current model is smaller. However, when more than 80 mm, the dipole model is more accurate. Besides, when the distance in the Z direction is more than 100 mm, the larger the Z distance, the larger the matching error. In addition, the dipole model is mathematically simpler for analyzing and calculating magnetic fields and can provide analytical solutions in 3D space [Reference Paperno, Sasada and Leonovich38]. Therefore, we chose the magnetic dipole model at the given working range in our localization system. The calibrated analytical model for magnetic localization of WCEs based on onboard sensing enables the real-time tracking of the pose (i.e., position and orientation) of the WCE, which has high adaptiveness and compatibility to the remote magnetic manipulation of a WCE inside the patient’s body in closed-loop feedback control. However, the dipole model provides limited localization performance when the WCE is close to the magnetic field source, which could compromise the magnetic coupling between the driving magnet and the WCE, hence the maneuverability.
Comparison of different magnetic field models. (a) Magnetic field comparison in one layer. (b) Matching errors of the two magnetic field models in all Z layers.
4.3. Magnetic localization under rotational motion of the WCE
Typically, the WCE will undergo rotational motion, translational motion, and both rotational and translational motion when performing GI diagnosis or treatment. We first check the localization error of the WCE during rotational motion. The capsule was placed on a three-axis rotating device, as shown in Fig. 3(a), which provides a rotation accuracy of 0.01° and no translational motion. Attitude changes in three different orientations were carried out by manually controlling the rotation speed of approximately 10°/s. The theoretical change was compared with the actual change. Table I shows the experimental results of 6 groups of different attitude changes. As shown in Table I, the average attitude errors of the six groups increase with the increase of attitude angle. The attitude calculation error accumulates with time. As the attitude angle approaches 90°, the quaternion solution of the Euler angle gradually becomes nearly singular, which leads to an increase in error. The average attitude errors of three different heading angles are 2.92°, 3.80°, and 2.06°, respectively, which remain in a relatively stable range, and the roll angle is more accurate. The overall average error of the six experiments is approximately 2.93°, which meets the actual demand.
Attitude change comparison.
4.4. Magnetic localization under translational motion
During the translational motion, the magnetic sensor was fixed on a graph paper to determine the relative position with the external magnet while maintaining the attitude unchanged. The posture of both the magnet and the sensor was kept unchanged. The external magnet was moved in the space of 200
$\times$
200
$\times$
200 mm by the robotic arm, and a trajectory of 45 points was obtained. The path calculated from the reading of the onboard sensing was compared with the path set to the robotic arm, as shown in Fig. 5(a). To reduce the sensor fusion error caused by the misalignment of the magnetic field sensor and IMU and reduce the overall size, we used a 9-axis magnetic and IMU MPU-9250. The MPU-9250 has a sensitivity of 0.6
$\mu$
T per least significant bit, and the sampling frequency is 100 Hz, which could satisfy the response speed and measurement precision of the localization algorithm.
(a) Path comparison: real position versus optimized calculated position. (b) Positioning errors in X, Y, Z directions. From the nearest to the furthest, the layers were denoted as Z1 to Z5, respectively.
In the magnetic localization under the translational motion of the WCE, we divided the working space into five layers according to the distance in the Z direction. Each layer was separated by 50 mm. Nine points were sampled in each layer, and the X and Y directions were separated by 100 mm, respectively. As shown in Fig. 5(b), with the increase of the distance in the Z direction, the positioning errors in the three directions gradually increase. After the calibration of the sensors and optimization of
$\boldsymbol{{M}}_{\boldsymbol{{E}}}$
in the magnetic dipole model, the overall mean errors of 5 layers are 3.60, 4.54, 5.59, 7.33, and 10.40 mm, respectively. The mean positioning error of the 45 points is 6.29 mm.
4.5. Error analysis with theoretical error model
Before we analyze the error, some variables are defined. The actual position of the magnetic sensor relative to the permanent magnet is
$Pr$
, and the position calculated by the magnetic field data is
$Pc$
. The difference between the two is the actual positioning error
$Er$
. The actual magnetic field data read by the magnetic sensor is
$Bs$
. The theoretical magnetic field data
$Bt$
is calculated based on the magnetic dipole model (Eq. (1)). The difference between the two is the magnetic field error
$E_B$
, which can be considered as the input error of the direct measurement. The theoretical positioning error (
$Et$
) is calculated from the magnetic field error
$E_B$
based on the error model (Eq. (10)).
It can be seen from Fig. 6 that the calculated error
$Et$
based on the theoretical error model is consistent with the actual positioning error
$Er$
. In general, the absolute value of theoretical error is less than the actual error, which indicates that experimental operation and environmental factors may bring extra positioning error. The positioning error is highly dependent on the magnetic field model and the accurate measurement from the magnetic sensor.
Error comparison.
$Et$
is calculated based on the theoretical error model,
$Er$
is the actual positioning error.
The above proves that the theoretical error model established in this study is compelling. The error model can be used as an instruction to reduce the magnetic field error
$E_B$
and the position error
$Er$
by calibrating the sensor and modifying the model parameters of the position method. One additional benefit of the error analysis is that accurate modeling can speed up the localization simulations to define a better working range by generating the locations with appropriate error distributions without running the localization algorithm [Reference Slijepcevic, Megerian and Potkonjak39].
4.6. Magnetic localization under rotational and translational motion
After improving the positioning accuracy of translational motion, we also design a path to change the position and attitude simultaneously. Based on the experimental platform in Fig. 7(a), we fixed the EPM, and the sensor module was fixed at the end of the manipulator, which moved along the six marker points from P1 to P6 shown in Fig. 7(b). The six points on the path are selected as the characteristic clinical detection markers based on the anatomical model of the stomach, which corresponds to the cardia (P1), fundus of stomach (P2), greater curvature of the stomach (P3), lesser curvature of the stomach (P4), pyloric sinus (P5), and pylorus (P6) [Reference Valdastri, Simi and Webster40, Reference Troccaz, Dagnino and Yang41].
(a) Experimental platform. (b) Stomach phantom with marker points. (c) Path comparison.
Figure 7(c) shows the average error and variance of the localization results in six experiments for these six points. The average position errors in the X, Y, and Z directions are 5.79, 8.72, and 14.90 mm, respectively, while the average error is 10.12 mm. Besides, the variance in the X, Y, and Z directions is 1.87, 2.00, and 0.24, respectively. It can be found that the average error in the Z direction is the largest, but the variance is the smallest. The reason might be that the motion of the six points in the Z direction is much smaller than the motion in the X and Y directions. In addition, the error at some points is significantly smaller than that of other points, which could be the consequence that the accuracy of the magnetic dipole model is related to the distance between the magnetic sensor and the magnet. While the distances between different points and the magnet are different. Therefore, the error of the point with a closer distance is smaller than that of other points. One drawback of this study is that the drift of the inertial data from IMU will become an issue over time, which can be improved by introducing a reference magnetic field in the localization system, such as integrating a pair of Helmholtz coils to provide a time-varying uniform magnetic field. Thus, absolute localization is still feasible by the proposed method for long-term magnetic localization and magnetic manipulation.
4.7. Motion in liquid environment
WCEs would work in a liquid environment in practical applications. Thus, we set up another set of experiments to verify the feasibility of magnetic localization by the onboard sensing units. As shown in Fig. 8(a), the sensor module was fixed to the end of a 3D-printed polymeric rod that was mounted to the robotic manipulator to obtain the magnetic field and the real-time position information. The sensor module was moved along the profile of the stomach model. A trajectory containing 9 points was recorded underwater, and the position was also estimated by the magnetic field. As shown in Fig. 8(b) and (c), the average position errors along X, Y, and Z directions are 5.06, 1.31, and 2.45 mm, respectively. A similar experiment was also conducted in the air. The result shows that the average positioning errors of the same path in the air are 3.98, 0.95, and 2.48 mm, respectively. The positioning error in this set of experiment is smaller than that in Fig. 5 because we adjusted the workspace to a space with a less matching error of the magnetic dipole model. The results further proved that the magnetic positioning method has environmental adaptability and effectiveness. Besides, it could provide accurate positioning results in the given workspace.
(a) Experimental platform and schematic diagram. (b) Position errors in water. (c) Error comparison in water versus in air.
5. Conclusions
Localization of capsule endoscopes during the examination of the GI tract is of great importance for precise diagnosis. In this paper, we have constructed a well-calibrated analytical model to perform the localization of the WCE based on onboard sensing, which is compatible with magnetic manipulation. For the positioning method, the fundamental limitation is the accuracy of the magnetic field model, reflecting the one-to-one relationship between the position and the magnetic field. In order to reduce the matching error of the magnetic field model, we optimized the magnetic dipole moment in the magnetic dipole model by collecting abundant sample points through a high-precision three-dimensional magnetic field measuring instrument. Meanwhile, we found that the different working range distributes to a different matching error of the magnetic field model, which also reflects the position error. In addition, based on the error propagation theory, we established a theoretical error model to analyze and predict the errors in the positioning system. Through the positioning experiments of various motion modes under different conditions, the accuracy of our localization system was evaluated.
Due to the size limitation of the capsule endoscope and the accuracy of the used sensor, the existing IMU module with an appropriate size for the WCE is not accurate enough to provide the highest accuracy. In the future, other magnetic sources could be added in vitro as an auxiliary magnetic field to further reduce attitude error. The incorporation of a dynamic magnetic field or more magnetic sensors inside the capsule endoscope could also contribute to accurate positioning. In addition, the matching fitness of the magnetic dipole model itself is another crucial point. For our proposed positioning method, we rely on the mapping relationship of the model. That is, the position and spatial magnetic field have a one-to-one mapping. However, the experimental results indicate that the dipole model has different accuracy within different spatial distributions. Hence, the relationship between different spaces and the accuracy of the magnetic dipole model could be further explored.
Author contributions
YL and CH conceived and designed the study. YL performed the study and statistical analyses. YL and ZH conducted data gathering. XL, YJ, and CS assisted in the use of robotic manipulation. YL and CH wrote the article.
Financial support
The financial support from the National Natural Science Foundation of China (61903177), the Shenzhen Science and Technology Program (Grant No. JCYJ20190809144013494), and the Science and Technology Program of Guangdong (Grant No. 2021A1515011813) are gratefully acknowledged. This work was in part supported by the Science, Technology, and Innovation Commission of Shenzhen Municipality [Grant ZDSYS20200811143601004], in part by the Southern Marine Science and Engineering Guangdong Laboratory (Guangzhou), and in part by the SUSTech-Jifu Medical Joint lab.
Conflicts of interest
The authors declare no conflicts of interest exist.
Ethical approval
Not applicable.
