1. Introduction
Traditional fracture reduction surgery heavily relies on the surgeon’s manual reduction skills, often involving the use of intramedullary nails for internal fixation. This method boasts a high fracture healing rate (90%∼99%) [Reference Gosling, Westphal, Hufner, Faulstich, Kfuri, Wahl and Krettek1] and a lower postoperative infection rate [Reference Kempf, Grosse and Beck2]. However, both the surgeon and the patient are exposed to radiation from the fluoroscopy machine for a significant duration during surgery, ranging from 158 to 316 s [Reference Sugarman, Adam and Bunker3]. Unfortunately, the use of intramedullary nails frequently results in severe malreduction, with sagittal and coronal plane misalignment occurring in approximately 2%∼18% of cases [Reference Grover and Wiss4]. Furthermore, the incidence rate of rotational deformity exceeding 10° exceeds 40% [Reference Prevot, Charissoux, Fiorenza, Arnaud and Pecout5]. Poor reduction can alter the normal alignment of the lower limbs, potentially leading to disability [Reference Yang, Han, Jahng, Shin and Park6]. Additionally, surgeons must manually maintain the reduction position for a period of time after the initial manual reduction to allow for the intramedullary nail to penetrate the medullary cavity and expand it. This process often demands significant physical strength from the surgeon, particularly in long bone fracture surgeries. The maximum force and torque generated by the surrounding muscles around the fractured bone are reported to be 604.52N and 74N
$\cdot$
m [Reference Georgilas, Dagnino, Tarassoli, Atkins and Dogramadzi7], while the maximum average reduction force and torque are 264N and 39N
$\cdot$
m [Reference Gösling, Westphal, Faülstich, Sommer, Wahl, Krettek and Hufner8]. With the advancement of robotic technology, robotic reduction combined with external fixation using bone spicules has become increasingly sophisticated. Currently, there are three types of robots used in fracture reduction surgery: passive robots, autonomous robots, and teleoperated robots.
Passive robots, which offer high flexibility during surgery, are fully controlled by the surgeon. However, they lack a dedicated control system for ensuring the safety of the operation, and the accuracy of the reduction largely depends on the surgeon’s experience. As a result, they are mainly used to assist surgeons in maintaining the fracture reduction position [Reference An, Ping, Li and Chai9].
Autonomous robots, on the other hand, are capable of performing the entire surgery without surgeon intervention [Reference Zheng, Lei, Hu and Zhang10]. The surgeon’s role is limited to determining the reduction path before surgery. The control system then sends motion control instructions to each joint of the robot, guiding it along the desired path for fracture reduction. Parallel robots, known for their high load capacity and precision, are widely employed in fracture reduction surgery [Reference Wang, Han and Lin11]. Notably, Lihai Zhang utilizes 3D reconstruction based on CT images to plan the reduction path [Reference Du, Hu, Li, Wang, Zhao, Li, Mao, Liu, Zhang, He, Zhang, Hou, Zhang and Tang12]. Although autonomous robots offer high surgical accuracy and controllability, they lack intraoperative flexibility and cannot effectively address unforeseen emergencies. The ultimate reduction outcome heavily relies on the accuracy of preoperative planning and registration.
The teleoperated robotic system consists of a master robot operated by the surgeon and a slave robot that responds to the surgeon’s actions on the other side, following its own motion protocol. It combines the flexibility of manual surgery with the precision of robotics. Wei Han conducted an experimental study on a teleoperated fracture reduction surgical robot for long bones under no load to evaluate accuracy and operability [Reference Han, Wang, Lin and Wang13]. The resulting data supports the feasibility of the teleoperated surgical method in fracture reduction surgery. Changsheng Li used two Stewart parallel mechanisms as the master and slave robots for teleoperated bone fracture reduction, confirming the load capacity of parallel mechanisms in fracture reduction surgery [Reference Li, Wang, Hu, Tang, Wang, Zhang, Guo and Tan14]. The robot can provide a maximum load capacity of 300N. Abedin-Nasab designed a complete long bone fracture reduction surgery system [Reference Saeedi-Hosseiny, Alruwaili, McMillan, Iordachita and Abedin-Nasab15]. To address the problem of the small workspace of Gough-Stewart Platform (GSP), a kind of reduction robot with 3-RRPS configuration is designed, and the symmetric 3-armed robot has a 15 times larger rotational workspace compared to that of the GSP. In cadaveric experiments, the larger workspace allowed the robot to drag the distal fractured bone in any direction to perform fracture reduction. However, they lack a path-planning algorithm to assist the surgeon in finding the optimal path. In addition, there are some research in the fields of master-slave control, intraoperative navigation and perception [Reference Zhao, Li, Ren, Hao, Zhang and Tang16].
For the teleoperated reduction surgery, it is worth noting that the surgeon’s manual reduction path often deviates from the planned path generated. Surgeons need extensive practice to adapt to this discrepancy. Currently, the primary method for monitoring reduction state during teleoperation is using a CT machine for fluoroscopy, which requires suspending all operations. Similar to traditional fracture reduction surgery, multiple fluoroscopy procedures are often needed to achieve precise fracture reduction, resulting in limited efficiency gains. In traditional surgery, surgeons can directly sense the reduction force, avoiding intraoperative errors. However, during teleoperation, surgeons lack effective intraoperative perception of changes in reduction force, potentially leading to operational mistakes.
The main contributions of this paper are as follows:
-
1. For teleoperated surgical robots used in long bone fractures, a constraint strategy of position and orientation based on virtual fixture is adopted. Position of the master robot is constrained based on the planned path and orientation of the master robot is constrained with the 3D visualization of the fracture model. The accuracy and stability for operation of the master robot are enhanced.
-
2. Force mapping is performed in Cartesian space. The large reduction force of the slave robot is fed back to the master robot, which enables surgeons to perceive changes in reduction force in real time and enhances their sense of operational presence.
-
3. A kind of fractured model with simulating large muscle forces is presented for experiment. The teleoperated fracture reduction experiment is conducted to verify the effectiveness of force feedback.
2. Teleoperated fracture reduction surgical robot system
2.1. System composition
The teleoperated robotic system primarily consists of the master robot, the slave robot, a six-dimensional force sensor, a fractured model with simulating large muscle force, an upper personal computer (PC), and a lower PC, as shown in Fig. 1. The upper PC is responsible for controlling the master robot and the lower PC is responsible for controlling the slave robot. Where
$q_{i},\dot{q}_{i},\ddot{q}_{i}$
are control variables of the master robot and
$L_{i},\dot{L}_{i},\ddot{L}_{i}$
are control variables of the slave robot.
System architecture of the teleoperated surgical robot.
The master robot, known as Omega.7, is a Delta mechanism equipped with force feedback for three degrees of freedom. It employs three base joints to collect position information and three wrist joints to collect orientation information of the surgeon’s hand.
The slave robot is designed with a 6-UPU Stewart parallel mechanical structure, which can meet the requirements of large workspace and substantial load capacity. The robot’s movement range along x, y and z axis is 200 mm, 200 mm and 100 mm and its rotational range around x, y and z axis is 40°, 40° and 70°. The robot can provide the maximum force of 300N and the maximum torque of 45N
$\cdot$
m. Communication between the master and slave robots is facilitated using the EtherCAT bus.
The fractured model is composed of fractured bone and simulated muscle as shown in Fig. 2. Firstly, the digital model of fractured bone is modeled using reverse reconstruction from CT images. Then, the fractured bone is made by 3D printed using resin material. During fracture reduction surgery, muscles exert force to restrict the robot’s movement. This force is referred to as muscle force. The muscle force magnitude mainly depends on elongation and category of the muscle. Several 4.5 mm silicone strips are employed to simulate the muscle tissue. These silicone strips are strategically positioned around the fractured bone model, and the intensity of the muscle force could be adjusted by varying the number of silicone strips.
Fractured model with simulating large muscle forces.
The clamping mechanism includes distal and proximal clamping instruments, several bone spicules and connectors. The proximal fractured bone is fixed on the operating table and the distal fractured bone is fixed on the moving platform of the slave robot with the bone spicules inserted into the fractured bone.
The six-dimensional force sensor is installed between the moving platform of the slave robot and distal clamping instruments to measure the reduction force. The spectra passive tool is installed on the moving platform of the slave robot to measure the position and orientation of the slave robot.
2.2. System dynamics model
The dynamics model of the master robot in cartesian space is:
\begin{equation}\boldsymbol{\Lambda }_{m}\left(\boldsymbol{q}_{m}\right)\ddot{\boldsymbol{x}}_{m}+\boldsymbol{\mu }_{m}\left(\boldsymbol{q}_{m},\dot{\boldsymbol{q}}_{m}\right)\dot{\boldsymbol{x}}_{m}+{\boldsymbol{F}_{m}}_{G}\left(\boldsymbol{q}_{m}\right)+{\boldsymbol{F}_{m}}_{f}\left(\boldsymbol{q}_{m}\right)={\boldsymbol{F}_{m}}_{c}+\boldsymbol{F}_{h}\end{equation}
where
$\boldsymbol{x}_{m}\in \mathbb{R}^{6}, \dot{\boldsymbol{x}}_{m}\in \mathbb{R}^{6}, \ddot{\boldsymbol{x}}_{m}\in \mathbb{R}^{6}$
are position vector, velocity vector and acceleration vector of the master robot in cartesian space,
$\boldsymbol{q}_{m}\in \mathbb{R}^{6}, \dot{\boldsymbol{q}}_{m}\in \mathbb{R}^{6}$
are joint variable and joint velocity,
$\boldsymbol{\Lambda }_{m}\in \mathbb{R}^{6\times 6}$
is inertia matrix,
$\boldsymbol{\mu }_{m}\in \mathbb{R}^{6\times 6}$
is centripetal force and coriolis force matrix,
${\boldsymbol{F}_{m}}_{G}\in \mathbb{R}^{6}$
is gravity,
${\boldsymbol{F}_{m}}_{f}\in \mathbb{R}^{6}$
is friction,
$\boldsymbol{F}_{h}\in \mathbb{R}^{6}$
is the force exerted by the operator on the master robot,
${\boldsymbol{F}_{m}}_{c}\in \mathbb{R}^{6}$
is driving force, which is used to overcome additional interference force such as gravity and friction. Adjusting
${\boldsymbol{F}_{m}}_{c}$
in real time can almost promise the surgeon a standard second-order linear system in an ideal state when operating. It can ensure the stability of the system during the entire surgery and assist the surgeon to better realize remote control, making
\begin{equation}{\boldsymbol{F}_{m}}_{c}=\boldsymbol{\eta }_{m}\left(\boldsymbol{q}_{m},\dot{\boldsymbol{q}}_{m}\right)-\boldsymbol{\Lambda }_{m}\left(\boldsymbol{q}_{m}\right)\boldsymbol{M}_{d}^{-1}\boldsymbol{B}_{d}\dot{\boldsymbol{x}}_{m}-\boldsymbol{\Lambda }_{m}\left(\boldsymbol{q}_{m}\right)\boldsymbol{M}_{d}^{-1}\boldsymbol{K}_{d}\boldsymbol{x}_{m}+\left(\boldsymbol{\Lambda }_{m}\left(\boldsymbol{q}_{m}\right)\boldsymbol{M}_{d}^{-1}-\boldsymbol{I}\right)\boldsymbol{F}_{h}\end{equation}
where
\begin{equation}\boldsymbol{\eta }_{m}\left(\boldsymbol{q}_{m},\dot{\boldsymbol{q}}_{m}\right)=\boldsymbol{\mu }_{m}\left(\boldsymbol{q}_{m},\dot{\boldsymbol{q}}_{m}\right)\dot{\boldsymbol{x}}_{m}+{\boldsymbol{F}_{m}}_{G}\left(\boldsymbol{q}_{m}\right)+{\boldsymbol{F}_{m}}_{f}\left(\boldsymbol{q}_{m}\right)\end{equation}
The desired dynamics model for the end of the master robot is:
\begin{equation}\boldsymbol{M}_{d}\ddot{\boldsymbol{x}}_{m}+\boldsymbol{B}_{d}\dot{\boldsymbol{x}}_{m}+\boldsymbol{K}_{d}\boldsymbol{x}_{m}=\boldsymbol{F}_{h}\end{equation}
Then, it can be approximately equivalent to a mass-damper-spring second-order linear system, where
$\boldsymbol{M}_{d}\in \mathbb{R}^{6\times 6}, \boldsymbol{B}_{d}\in \mathbb{R}^{6\times 6}, \boldsymbol{K}_{d}\in \mathbb{R}^{6\times 6}$
are desired inertia, desired damping and desired stiffness. It is generally hoped that the master robot is a free-operating system that does not need to apply back-to-center traction, so set
$\boldsymbol{K}_{d}$
to zero matrix. Introducing the time domain and (4) can be written as:
\begin{equation}\boldsymbol{F}_{h}\left(t\right)=\boldsymbol{M}_{d}\ddot{\boldsymbol{x}}_{m}\left(t\right)+\boldsymbol{B}_{d}\dot{\boldsymbol{x}}_{m}\left(t\right)\end{equation}
Establishing the dynamics model of the slave robot in cartesian space:
\begin{equation}\boldsymbol{\Lambda }_{s}\left(\boldsymbol{q}_{s}\right)\ddot{\boldsymbol{x}}_{s}+\boldsymbol{\mu }_{s}\left(\boldsymbol{q}_{s},\dot{\boldsymbol{q}}_{s}\right)\dot{\boldsymbol{x}}_{s}+{\boldsymbol{F}_{s}}_{G}\left(\boldsymbol{q}_{s}\right)+{\boldsymbol{F}_{s}}_{f}\left(\boldsymbol{q}_{s}\right)={\boldsymbol{F}_{s}}_{c}-\boldsymbol{F}_{ext}\end{equation}
where
$\boldsymbol{x}_{s}\in \mathbb{R}^{6},\dot{\boldsymbol{x}}_{s}\in \mathbb{R}^{6},\ddot{\boldsymbol{x}}_{s}\in \mathbb{R}^{6}$
are position vector, velocity vector and acceleration vector of the slave robot in cartesian space,
$\boldsymbol{q}_{s}\in \mathbb{R}^{6}, \dot{\boldsymbol{q}}_{s}\in \mathbb{R}^{6}$
are joint variable and joint velocity,
$\boldsymbol{\Lambda }_{s}\in \mathbb{R}^{6\times 6}$
is inertia matrix,
$\boldsymbol{\mu }_{s}\in \mathbb{R}^{6\times 6}$
is centripetal force and coriolis force matrix,
${\boldsymbol{F}_{s}}_{G}\in \mathbb{R}^{6}$
is gravity,
${\boldsymbol{F}_{s}}_{f}\in \mathbb{R}^{6}$
is friction,
${\boldsymbol{F}_{s}}_{c}\in \mathbb{R}^{6}$
is driving force for the robot,
$\boldsymbol{F}_{ext}\in \mathbb{R}^{6}$
is external force on the robot called the reduction force.
3. Six-dimensional constraints of master robot
3.1. Generation of virtual fixture and virtual axis
Traditional virtual fixtures can be categorized into guided virtual fixtures and forbidden virtual fixtures [Reference Nia Kosari, Rydén, Lendvay, Hannaford and Chizeck17]. Guided virtual fixtures provide specific guidance to the end of the robot during operations [Reference Zainan Jiang, Liu and Zou18]. In the context of operating the slave robot where manual control may lack precision, the virtual fixture (VF) can assist operators in achieving precise trajectory control, which enhance the performance of teleoperation.
Due to the complex musculoskeletal environment of fractured tissues, the soft tissue traction during the reduction process inevitably affects the reduction process, so the reasonable reduction path can avoid excessive stretching of soft tissues. An improved A* algorithm is studied to avoid secondary damage. The ICP algorithm based on fracture cross-section point cloud is used to register the proximal and distal fracture cross-sections, then the fracture deviation can be determined. Considering the changes in muscle length during the reduction process, an improved A* algorithm is presented to plan the reduction path [Reference Xu, Lei, Hu and Zhang19].
Based on the planned reduction path, a pipe-type virtual fixture with spherical surfaces is generated, with each path point as the center and the smooth enveloped surface formed by connecting all spherical surfaces constitutes the virtual fixture surface
$\mathbf{\Phi }=\{{\mathbf{\Phi }_{1}},{\mathbf{\Phi }_{2}},\cdots \mathbf{\Phi }_{i}\}$
, where
$\mathbf{\Phi }_{i}$
can be described as follows:
\begin{align}\begin{cases} x=P_{ix}+R\cos k\dfrac{\pi }{m}\cos k\dfrac{2\pi }{m}\\[12pt] y=P_{iy}+R\sin k\dfrac{\pi }{m}\sin k\dfrac{2\pi }{m}\\[12pt] z=P_{iz}+R\cos k\dfrac{\pi }{m} \end{cases} k=0,1,\cdots m \end{align}
where
$\boldsymbol{P}_{i}=(P_{ix},P_{iy},P_{iz})$
represents the reduction path point,
$i$
is the point number along the reduction path,
$m$
is the mesh density of the sphere. The larger
$i$
and
$m$
, the smoother the generated virtual fixture, with the actual value determined by the hardware’s performance.
$R$
is the radius of the virtual pipe. In terms of fracture reduction accuracy, clinical requirements typically dictate that the offset should not exceed 5 mm [Reference Yanpeng20]. To ensure the accuracy of fracture reduction, the error between the actual robot movement path and the planned path must also be limited within this range. Hence, we set
$R_{max}$
as 5 mm.
As shown in Fig. 3 (a), the display interface of the virtual fixture is generated using OpenGL. The planned reduction path is represented by the yellow center line and surrounded by the blue virtual pipe. The real position of the master robot is represented by the small orange ball.
Display interface.
The planned reduction orientation is expressed using the zyx-Euler in which (
$\alpha$
,
$\beta$
,
$\gamma$
) is the rotation angles around the z, y and x axis. The coordinate system corresponding to the reduction orientation needs to be generated at each reduction path point. Suppose that
$\boldsymbol{P}_{i}$
is the reduction path point, {
A
} is the virtual world base coordinate system, {
B
} is the coordinate system with
$\boldsymbol{P}_{i}$
as the origin. Orientation of {
B
} relative to {
A
} is the orientation of reduction path point
$\boldsymbol{P}_{i}$
, which is expressed by rotation matrix:
\begin{equation}{B}^{A}{\boldsymbol{R}}{}=\left[\begin{array}{c@{\quad}c@{\quad}c} c\alpha c\beta & {c}{\alpha }s\beta s\gamma -s\alpha c\gamma & {c}{\alpha }s\beta c\gamma +s\alpha s\gamma \\[4pt] s\alpha c\beta & {s}{\alpha }s\beta s\gamma +c\alpha c\gamma & {s}{\alpha }s\beta c\gamma -c\alpha s\gamma \\[4pt] -s\beta & c\beta s\gamma & c\beta c\gamma \end{array}\right]=\left[\begin{array}{c@{\quad}c@{\quad}c} {}^{A}{\boldsymbol{x}}{}_{B} & {}^{A}{\boldsymbol{y}}{}_{B} & {}^{A}{\boldsymbol{z}}{}_{B} \end{array}\right]\end{equation}
where
${B}^{A}{\boldsymbol{R}}{}$
is rotation matrix of {
B
} relative to {
A
},
$s\theta =sin(\theta ),c\theta =cos(\theta )$
,
${}^{A}{\boldsymbol{x}}{}_{B},{}^{A}{\boldsymbol{y}}{}_{B},{}^{A}{\boldsymbol{z}}{}_{B}$
is axis of {
B
}.
As shown in Fig. 3 (b), the axis of { B } corresponding to each reduction path point is also superimposed on the display interface of virtual fixture. The virtual axis can provide orientation guidance during reduction so that the surgeon can operate in the correct order to avoid collision with some important muscle tissues. Virtual axis is the planned orientation in each reduction path point. Real axis is the real orientation of the master robot. The red line is the x axis while the green line is the y axis and the blue line is the z axis. The number of virtual axes displayed can be adjusted as required. When continuous and precise orientation references are needed, the display density can be increased. If the virtual axes become too dense and obstruct the display of other elements, the density can be reduced.
3.2. Method of position constraints
In the context of fracture reduction surgery, where acceleration is nearly zero and negligible, and restoring force is unnecessary, we can simplify the mathematical model as follows:
\begin{equation}\boldsymbol{G}\left(f\right)\cdot \boldsymbol{F}=\boldsymbol{B}\cdot \boldsymbol{V}\end{equation}
where
$\boldsymbol{F}\in \mathbb{R}^{3}$
is input force,
$\boldsymbol{B}\in \mathbb{R}^{3\times 3}$
is generalized damping,
$\boldsymbol{V}\in \mathbb{R}^{3}$
is output velocity of the robot,
$\boldsymbol{G}(f)\in \mathbb{R}^{3\times 3}$
is virtual fixture rule. Jake J. Abbott proposed that as long as the movement in the non-ideal direction is restricted, the robot will move along the desired trajectory, based on which the mathematical expression of the virtual fixture rule is presented [Reference Abbott, Marayong and Okamura21]. Assuming that a time-varying matrix
$\boldsymbol{D}(t)\in \mathbb{R}^{3\times 3}$
is given to represent the instant direction of the motion of the robot, which has full rank. Define the projection operator:
\begin{equation}\boldsymbol{D}_{\tau }+\boldsymbol{D}_{\tau }=\boldsymbol{I}\end{equation}
where
$\boldsymbol{D}_{\tau }\in \mathbb{R}^{3\times 3}$
is instant ideal direction of motion,
$\boldsymbol{D}_{r}\in \mathbb{R}^{3\times 3}$
is instant forbidden direction of motion, the coefficient
$g\in [0,1]$
is introduced to control the guiding effect of the component force in the non-ideal direction, (10) is transformed to:
\begin{equation}\left(\boldsymbol{D}_{\tau }+g\cdot \boldsymbol{D}_{r}\right)\cdot \boldsymbol{F}=\boldsymbol{B}\cdot \boldsymbol{V}\end{equation}
When the position of the master robot is within the confines of the virtual pipe, the master robot can move freely, and
$g=0$
; when the position of the master robot is outside the virtual pipe, its movement must be constrained to restrict non-ideal directional motion, resulting in
$0\lt g\lt 1$
; Complete restriction of the master robot’s movement in non-ideal directions implies a rigid virtual fixture, with
$g=1$
. To maintain the desired flexibility of the virtual fixture, it is crucial to ensure that
$0\lt g\lt 1$
. In the design of the flexible virtual fixture presented in this paper, the parameter
$g$
is controlled by the stiffness
$K$
of the virtual spring, and it is defined as
$g=K/K_{max}$
.
The utilization of a flexible virtual fixture aims to maintain the master robot’s movement in the ideal direction within the virtual pipe. This objective is realized by generating opposing forces.
As shown in Fig. 4, the Haptic Interaction Point (HIP) is created to represent position of the master robot in the virtual environment. The proxy point (Proxy) is consistent with HIP when HIP is inside the pipe. However, when the master robot makes contact with the surface of the virtual fixture and moves outward, HIP tracks the master robot’s movement, while Proxy remains situated on the surface of the virtual fixture. During this scenario, HIP continues its motion, and Proxy consistently adheres to the surface of the virtual fixture, following only the motion of HIP in the ideal direction
$\boldsymbol{D}_{\tau }$
. Essentially, Proxy serves as the projection of HIP onto the outer surface of the virtual fixture. This relationship can be mathematically expressed as:
\begin{equation}\boldsymbol{X}_{\textit{Proxy}}=\begin{cases} \boldsymbol{X}_{HIP}, \quad\boldsymbol{X}_{HIP}\in pipe\\[5pt] \boldsymbol{X}_{HIP}\cdot \mathbf{\Phi },\quad\boldsymbol{X}_{HIP}\notin pipe \end{cases}\end{equation}
Rule of flexible virtual fixture.
The constraint force is determined based on the positional relationship between HIP and Proxy:
\begin{equation}\boldsymbol{F}_{i}=K\cdot \left(\boldsymbol{X}_{\textit{Proxy}}-\boldsymbol{X}_{HIP}\right)+\boldsymbol{B}\cdot \dot{\boldsymbol{X}}_{HIP}\end{equation}
3.3. Method of orientation constraints
In the absence of a motor to provide force constraint on the rotating joint, visual guidance is employed for orientation constraints. During the reduction, surgeons can observe any deviations in the actual orientation of the master robot compared to the planned orientation using the virtual axis display interface and make manual adjustments as needed. However, the fractured bone is surrounded by muscle tissue and its reduction status cannot be discerned with the naked eye. The CT machine cannot display real-time reduction status due to the substantial radiation involved. So we conduct the three-dimensional reconstruction of fractured bone in the virtual world. This reconstructed virtual fractured bone is then matched with the actual fractured bone. During the operation, the real-time reduction status of the fractured bone can be observed on the display interface, facilitating necessary adjustments during the final stages of fracture reduction. This human-machine interaction (HMI) interface improves operational efficiency and eliminates the need for multiple CT scans.
The origin of the fractured bone’s coordinate system in the virtual world is consistent with the insertion point of the actual fractured bone. Two spectra passive tools installed on the proximal fractured bone and distal fractured bone are used to track orientation information. The initial orientation of the virtual fractured bone aligns with the one of the actual fractured bone using orientation measured by NDI track. During the operation, the proximal fractured bone remains stationary while the distal fractured bone moves in tandem with the slave robot. The virtual fractured bone in virtual world can provide visual references for surgeons during reduction as shown in Fig. 5 (a), (b).
Display of the fracture reduction in 3D virtual world.
4. Force feedback of slave robot
The control block diagram of the slave robot is shown in Fig. 6. Assuming that the surgeon’s hand is rigidly connected to the end of the master robot during the surgery. The planned position and orientation data are imported to generate the virtual fixture. The constraint force is calculated based on the force
$\boldsymbol{F}_{i}$
, which is then transmitted to the master controller. The slave robot responds to the movement of the master robot in accordance with its control law and commences motion along the optimized trajectory. Position and orientation information from the slave robot is relayed to the master controller via the position sensor. It can be deduced from (6) that the reduction force can be calculated based on the motion state and impedance parameters of the slave robot:
\begin{equation}\boldsymbol{F}_{ext}={\boldsymbol{F}_{s}}_{c}-\boldsymbol{\Lambda }_{s}\left(q_{s}\right)\ddot{\boldsymbol{x}}_{s}-\boldsymbol{\mu }_{s}\left(\boldsymbol{q}_{s},\dot{\boldsymbol{q}}_{s}\right)\dot{\boldsymbol{x}}_{s}-{\boldsymbol{F}_{s}}_{G}\left(\boldsymbol{q}_{s}\right)-{\boldsymbol{F}_{s}}_{f}\left(\boldsymbol{q}_{s}\right)\end{equation}
The motion state of the slave robot can be acquired from internal sensors, but accurately determining the impedance parameters remains a challenge. Using internal sensors alone to estimate the force may result in relatively large errors, which do not meet the accuracy requirements for force feedback during reduction surgery [Reference Lei, Zheng, Hu, Zhang and Wang22]. As a solution, a force sensor is used to directly measure the reduction force
$\boldsymbol{F}_{ext}$
.
Control block diagram.
The method of direct force feedback involves the master robot reproducing this portion of force to implement the force feedback function. The force balance relationship can be inferred as follows:
\begin{equation}{M}^{M_{0}}{\boldsymbol{R}}{}\cdot { }^{M}{\boldsymbol{F}}{}_{f}={S}^{S_{0}}{\boldsymbol{R}}{}\cdot {\textit{sensor}}^{S}{\boldsymbol{R}}{}\cdot \lambda \cdot { }^{\textit{sensor}}{\boldsymbol{F}}{}_{ext}\end{equation}
The feedback force of the master robot can be deduced as:
\begin{equation}{ }^{M}{\boldsymbol{F}}{}_{f}={M_{0}}^{M}{\boldsymbol{R}}{}\cdot {S}^{S_{0}}{\boldsymbol{R}}{}\cdot {\textit{sensor}}^{S}{\boldsymbol{R}}{}\cdot \lambda \cdot { }^{\textit{sensor}}{\boldsymbol{F}}{}_{ext}\end{equation}
where
${ }^{M}{\boldsymbol{F}}{}_{f}\in \mathbb{R}^{3}$
is the feedback force in master robot’s moving coordinate system,
${M}^{M_{0}}{\boldsymbol{R}}{}\in \mathbb{R}^{3\times 3}$
is the rotation matrix converted from master moving coordinate system to base coordinate system,
${S}^{S_{0}}{\boldsymbol{R}}{}\in \mathbb{R}^{3\times 3}$
is the rotation matrix converted from moving platform coordinate system to static platform coordinate system of the salve robot,
${\textit{sensor}}^{S}{\boldsymbol{R}}{}\in \mathbb{R}^{3\times 3}$
is the rotation matrix converted from force sensor coordinate system to moving platform coordinate system of the slave robot,
${ }^{\textit{sensor}}{\boldsymbol{F}}{}_{ext}\in \mathbb{R}^{3}$
is the reduction force measured in coordinate system of the force sensor. For surgeon’s feeling of the feedback force, the joint torque
$\boldsymbol{\tau }_{m}\in \mathbb{R}^{3}$
is applied by controlling joint motors of the master robot. Force in cartesian space is mapped to joint space of the master robot through force Jacobian matrix
$\boldsymbol{J}^{T}\in \mathbb{R}^{3\times 3}$
.
\begin{equation}\boldsymbol{\tau }_{m}=\boldsymbol{J}^{T}\cdot \left({M}^{M_{0}}{\boldsymbol{R}}{}\cdot { }^{M}{\boldsymbol{F}}{}_{f}\right)\end{equation}
The Jacobian matrix of the master robot is:
\begin{equation}\boldsymbol{J}=\left[\begin{array}{c@{\quad}c@{\quad}c} u_{11} & u_{12} & u_{13}\\ u_{21} & u_{22} & u_{23}\\ u_{31} & u_{32} & u_{33} \end{array}\right]^{-1}\left[\begin{array}{c@{\quad}c@{\quad}c} w_{1} & 0 & 0\\ 0 & w_{2} & 0\\ 0 & 0 & w_{3} \end{array}\right]\end{equation}
where
\begin{align*}\begin{cases} u_{11}=2p_{x}+\left(R-r+mcos\theta _{1}\right)\\[4pt] u_{12}=2p_{y}-\sqrt{3}\left(R-r+mcos\theta _{1}\right)\\[4pt] u_{13}=2\left(p_{z}+msin\theta _{1}\right) \end{cases} \begin{cases} u_{21}=2p_{x}+\left(R-r+mcos\theta _{2}\right)\\[4pt] u_{22}=2p_{y}+\sqrt{3}\left(R-r+mcos\theta _{2}\right)\\[4pt] u_{23}=2\left(p_{z}+msin\theta _{2}\right) \end{cases}\end{align*}
\begin{align*}\begin{cases} u_{31}=2p_{x}-\left(R-r+mcos\theta _{3}\right)\\[4pt] u_{32}=p_{y}\\[4pt] u_{33}=p_{z}+msin\theta _{3} \end{cases} \begin{cases} w_{1}=m\left(2R-2r+p_{x}-\sqrt{3}p_{y}\right)sin\theta _{1}-2mp_{z}cos\theta _{1}\\[4pt] w_{2}=m\left(2R-2r+p_{x}+\sqrt{3}p_{y}\right)sin\theta _{2}-2mp_{z}cos\theta _{2}\\[4pt] w_{3}=m\left(R-r-p_{x}\right)sin\theta _{3}-mp_{z}cos\theta _{3} \end{cases}\end{align*}
where
$m$
is the length of the active arm,
$R$
is the radius of the distribution circle of the active arm hinge point,
$r$
is the radius of the distribution circle of the slave arm hinge point,
$\theta _{1},\theta _{2},\theta _{3}$
are the angle between the plane of the active arm and the static platform,
$p_{x},p_{y},p_{z}$
are the position of the end of the master robot. Combining (5), (13), (17), then the dynamic equation of master robot is:
\begin{equation}\boldsymbol{F}_{h}\left(t\right)-\left[\begin{array}{c} \boldsymbol{F}_{i}\left(t\right)\\[4pt] 0 \end{array}\right]-\left[\begin{array}{c} \boldsymbol{F}_{f}\left(t\right)\\[4pt] 0 \end{array}\right]=\boldsymbol{M}_{d}\ddot{\boldsymbol{x}}_{m}\left(t\right)+\boldsymbol{B}_{d}\dot{\boldsymbol{x}}_{m}\left(t\right)\end{equation}
5. Experiment
5.1. Experimental system
The experimental system is illustrated in Fig. 7. The master side comprises the master robot and the HMI interface. The interface displays the teleoperated operation area, video monitoring area, virtual fixture constraint area, and data scope area. The interactive interface offers two control modes: teleoperated control mode and point-to-point (PTP) control mode. The slave side is composed of the NDI track, the slave robot and a fractured model with simulating muscle forces.
Experimental system.
5.2. Force constraint experiment of the master robot
The force constraint experiment based on the virtual fixture is conducted independently on the master side without connecting the slave robot to verify the effectiveness of the designed virtual fixture. The planned path is imported to the upper PC, and the operator guides the master robot to move along the planned path. The position-tracking path of the master robot is shown in Fig. 8 (a). When the virtual fixture constraint is applied, the reduction path closely aligns with the planned path, and the final reduction position is maintained as expected. In contrast, without the virtual fixture, the reduction path deviates significantly from the planned path, leading to a substantial deviation in the final reduction position.
Experimental results of force constraint of the master robot.
As shown in Fig. 8 (b), (c), (d), the relationship between position and constraint force of the master robot is displayed in the x, y and z axis. If the master robot deviates from the expected path during reduction, a reverse constraint force is generated to guide it back to the planned path. When the path remains on track without deviation, the constraint force is set to zero.
5.3. Force feedback experiment of the slave robot
To assess the effectiveness of force feedback from the slave side, a force feedback experiment is conducted on the slave robot. The lower PC sends the reduction force data collected by the force sensor to the master side. The upper PC initially applies a low-pass filter method to deal with the force signal. Then the joint torque of the master robot is calculated based on the
${ }^{M}{\boldsymbol{F}}{}_{f}$
. Finally, the current command is transmitted to the driver for force feedback.
The reduction force of the salve robot and the feedback force of the master robot in x, y and z axis are shown in Fig. 9 (a), (b), (c). The errors in force feedback are shown in Fig. 9 (d), indicating that the accuracy is
$\pm 0.2N$
.
5.4. Teleoperated fracture reduction experiment
The experiment is performed using the fractured model with simulated muscle forces. The slave robot is reset to the zero position. To enable reconstruction, the CT machine captures images of the fractured area from various angles. The path-planning module calculates the planned reduction path, including both position and orientation. Subsequently, the upper PC automatically generates the virtual fixture based on the planned reduction path. The operator selects the teleoperated mode in the interface and powers up the six electric cylinders of the slave robot. Before taking measurements, the force sensor requires calibration. The operator sets the master robot to its zero position and activates the gripper on the master robot to initiate the teleoperated mapping process. The operator then guides the master robot along the planned path with the assistance of the virtual fixture. Upon reaching the final planned position and orientation, adjustments are made by observing the real-time 3D virtual scene. After completing these adjustments, the operator releases the gripper of the master robot and the slave robot maintains its current position and orientation regardless of the master robot’s movements. The fractured bone is fixed using the external fixation method. Finally, the operator selects the PTP mode to reset the slave robot to its initial state and then powers it off to conclude the surgery.
Position, orientation and errors after reduction.
Experimental results of force feedback of the slave robot.
Three groups reduction experiments are conducted. The position and orientation of the slave robot after reduction and errors of position and orientation between real and planned values are shown in Table I. The planned position and orientation after reduction are as follows:
\begin{equation*} \left[-27.66\textrm{mm} \quad 23.11\textrm{mm} \quad 13.28\textrm{mm} \quad 7.37^{\circ} \quad 2.34^{\circ} \quad 5.64^{\circ} \right] \end{equation*}
As shown in Table II, compared to the traditional fracture reduction method [Reference Follerås, Ahlo, Strømsøe, Ekeland and Thoresen23] and the teleoperated operating system that solely rely on CT fluoroscopy [Reference Han, Wang, Lin and Wang13], the fracture reduction method employed in this paper demonstrates significant advantages. It offers high precision and reduces the need for fluoroscopy.
Error statistics of three groups is shown in Fig. 10.
Comparison with traditional method and other teleoperated method.
Error statistics of the fracture reduction.
The group I of the experiment results are detailedly shown in Fig. 11.
-
1. Position and orientation of the master robot are shown in Fig. 11 (a), (b).
-
2. Position and orientation of the slave robot are shown in Fig. 11 (c), (d).
-
3. Errors of position and orientation between the master robot and the slave robot are shown in Fig. 11 (e), (f).
-
4. Feedback force and reduction force are shown in Fig. 11 (g), (h).
-
5. State of the fractured segments before and after reduction is shown in Fig. 11 (i), (j).
Experimental results of the fracture reduction.
From Fig. 11 (a), (b), (c) and (d), when master-slave mapping begins, the slave robot can quickly track the master robot. When the slave robot moves with multi-directions, the stroke of each cylinder increases. From 120s to 150s, the master robot provides a large range movement in the x y and z axis at the same time. Because the control software limits the maximum safe velocity for each electric cylinder to prevent operation mistakes, electric cylinders can’t keep up with the command. In addition, the slave robot is a parallel mechanism with joint coupled. When the position and orientation change simultaneously, the robot’s response velocity to the position is lower than that of the orientation. When the orientation of the salve robot changes quickly, the maximum position error reaches 8.7 mm and the maximum orientation error reaches 1.6°. The average position and orientation error is no more than 2 mm and 1°, respectively, as shown in Fig. 11 (e) and (f).
It can be seen in (g) that when the path of the master robot deviates from the expected path, the master robot exerts additional interactive force on the surgeon’s hand to guide him back to the expected path. Simultaneously, it can be seen in (h) that the master robot accurately reproduces the reduction force from the slave side, enhancing the surgeon’s perception of the operating environment. What’s more, this experiment requires only one fluoroscopy to assess the reduction effect, eliminating the need for multiple adjustments.
6. Discussion
This paper has the following shortcomings:
-
1. Robot joint coupling: The coupling between joints of the slave robot can reduce teleoperated followability, especially when the robot needs to move with multiple degrees of freedom in cartesian space.
-
2. Calibration errors: Currently, manual calibration by NDI track is used for initialization. Calibration errors between the virtual fracture model and the actual fracture model may impact the operator’s perception of the reduction status.
-
3. Constraint force and feedback confusion: The mixing of constraint forces and feedback forces of the master robot during the fracture reduction process can confuse the operator and reduce the transparency of the operation.
Based on the above shortcomings, we will continue the following work in the future:
-
1. Optimization of joint trajectory for the slave robot: Incorporating the structural motion characteristics of the slave robot into the reduction path-planning process is the next step. Optimization of the robot’s motion trajectory can reduce coupling effects and enhance the robot’s followability and precision during surgery.
-
2. Automatic calibration: Through the automatic calibration algorithm, the virtual fractured model bone is automatically aligned according to the fracture segments. Then it is initialized according to the calculated displacement and malalignment.
-
3. Separation of constraint force and feedback force: We consider to increase the priority of the constraint force for the master robot. If the constraint force is higher than the threshold, force feedback is turned off and then the constraint force is prioritized. If the constraint force is lower than the threshold, force feedback is turned on and the constraint force and feedback force exist together.
7. Conclusion
For the robot-assisted fracture reduction surgery with teleoperated operating mode, in order to improve the operating accuracy, a kind of position and orientation constraint method of the master robot based on virtual fixture is presented, and the virtual fixture is generated online according to the planned reduction path. If the master robot moves outside the virtual pipe, the force of the master robot is generated from the flexible virtual fixture to limit its position. Guided by the developed 3D visualization interface of the fractured model, the operator controls the orientation of the master robot through visual observation and tries to maintain the actual orientation as consistent as possible with the virtual orientation.
In order to enhance the surgeon’s sense of operational presence, the reduction force information of slave robot is collected by a six-dimensional force sensor. The reduction force is mapped to the master robot linearly. The required joint torque of the master robot is calculated and then force feedback is performed through motor stall control method.
The teleoperated fracture reduction experiments on the fractured model with simulating muscle force are conducted, and the experimental results show that the constraint method based on virtual fixture can significantly improve the operating accuracy. The force feedback method can achieve accurate force feedback under large loads
Author contributions
Jingtao Lei and Guangqing Song jointly carried out the research work, performed the experiments, wrote and revised the manuscript.
Financial support
This work is supported by the National Key R&D Program of China under 2020YFB1313803.
Competing interests
The authors declare no competing interests exist.
Ethical approval
None.
