2.1. Direct manual interaction

The direct manual interaction is required when the endoscope is to be placed out or in the surgical workspace (inside the nostril). During this operation, surgeon releases the brakes on the passive arm which carries the active arm via activating the related button on the user interface and deactivates the same button when the endoscope is to be placed back at the surgical workspace (https://youtu.be/kUtVecdl_gY). A prismatic joint is driven by a linear motor to adjust the height of the system before the surgery, and thereon it maintains its position throughout the surgery.

2.2. Master-slave teleoperation

The master-slave teleoperation subsystem of the NeuRoboScope is designed to have the master surgeon operating with surgical tools other than the endoscope, while the endoscope is handled by the slave system (https://www.youtube.com/shorts/b4amZrX_9M0). The slave system with the endoscope and the surgeon are cooperating on the patient at the same time as shown in Fig. 2.

Figure 2.

A screenshot indicating the master-slave teleoperation mode. A surgical tool, aspirator, is handled by the hand wearing the IMU-embedded ring. Robot and the surgeon cooperate on the patient.

The slave system is teleoperated by the surgeon, while the surgeon carries out the surgery on top of the patient with the other surgical tools. The information exchange between the master and the slave systems is described in Fig. 3. In Fig. 3, the “Endoscope Robot” term indicates the active arm that is placed on the passive arm. The active arm controls the pose of the endoscope inside the surgical workspace due its RCM capability of two rotational and one translational DoFs. The unilateral information flow from the master to the slave is initiated by pressing the foot pedal. The information sent via Bluetooth is the absolute orientation information received from the inertial measurement unit (IMU) that is embedded inside the wearable ring. When the foot pedal is not pressed, the slave system maintains its position and the teleoperation system does not issue any pose changing command to the slave system. Hence, any involuntary motion of the endoscope is restricted.

Figure 3.

Information exchange between the master system (ring and foot pedal) and the slave system (endoscope robot) of the NeuRoboScope.

This absolute orientation information is processed [Reference Dede, Kiper, Ayav, Özdemirel, Tatlıcıoğlu, Hanalioglu, Işıkay and Berker4] at the master side to generate speed demands ( $\omega _{\textrm{pitch}}, \omega _{\textrm{roll}}, v_{\textrm{dolly}}$ ) to drive the slave system. Therefore, the motion demand generated is not an absolute position demand but relative motion demand as a result of this procedure. In fact, during this operation, surgeon continuously observes how the slave system moves since the view captured by the endoscope and displayed at the monitor changes. The aim here is to minimize the efforts of the surgeon when the surgical workspace view needs to be changed. The mapping between the motion of the ring and the motion of the endoscope resembles the operation scenario between the gas pedal of an automobile and the change of its speed.

2.3. A comparison between the NeuRoboScope system and the other robotic endonasal surgery systems

A variety of methods have been developed to integrate robots into minimally invasive endonasal surgery procedures. These methods are (1) systems with autonomous tracking capability [Reference Dai, Zhao, Zhao, He, Sun, Gao, Hu and Zhang5], (2) direct master-slave teleoperation systems where the master system is located at a console away from the patient [Reference Chumnanvej, Pillai, Chalongwongse and Suthakorn6–Reference Muñoz, Garcia-Morales, Fraile-Marinero, Perez-Turiel, Muñoz-Garcia, Bauzano, Rivas-Blanco, Sabater-Navarro and de la Fuente10], (3) systems with supervisory level teleoperation with automated motion planning capabilities, (4) systems that require direct physical contact with the surgeon [Reference Colan, Nakanishi, Aoyama and Hasegawa11] and (5) systems that support the surgeon but do not directly handle any of the surgical tools [Reference Ogiwara, Goto, Nagm and Hongo12]. The NeuRoboScope system’s operation paradigm cannot be categorized into any of these methods. In the NeuRoboScope system, the surgeons can simultaneously operate with two surgical tools and the endoscope while they are operating on the patient via continuous control capability of the endoscope robot. Additionally, to the best of our knowledge, the personalized use options of these methods are limited and/or not have been reported.

1. 1. In ref. [Reference Dai, Zhao, Zhao, He, Sun, Gao, Hu and Zhang5], an endoscope robot is designed with the RCM mechanism. This robot is designed to navigate the endoscope autonomously during the surgery. The desired motion of the robot is coupled to the motion of surgical tool that the surgeon handles during the surgical operation. A vision system tracks the motion of the surgical tool and directs the endoscope robot with respect to this captured motion. The obvious problems of such a system are as follows: (1) markers should be attached to the surgical tool which is constantly swapped by another tool throughout the surgery; (2) room lighting conditions should be stable; and (3) surgeon does not always want the same tool to be followed. These problems are still to be addressed in future works.

2. 2. In ref. [Reference Chumnanvej, Pillai, Chalongwongse and Suthakorn6], a surgical robotic system is introduced for minimally invasive neurosurgery procedures. This robot control system guides the surgeon the endoscope is inserted inside the nostril. The surgeon is situated at a control station to control the motion of the robot throughout the surgery. This setting is a teleoperation setting. However, a surgeon is allocated to control the motion of the robot away from the patient similar to the operation routine of the da Vinci surgical system. Therefore, this method cannot be compared with the NeuRoboScope system where the surgeon controls endoscope and two other surgical tools simultaneously.

3. 3. In ref. [Reference He, Deng and Zhang13], an assistive surgical robot is proposed, which has similar features in terms of the NeuRoboScope robot’s mechanism with RCM. The teleoperation system’s master side involves the use of a commercial voice recognition system. This system is trained with eight commands to drive the surgical robot, while the surgeon continues the surgery by using both of their hands. The only indication given is how well these eight commands are identified by using ten test subjects. There is no indication about the ergonomics of use of such a teleoperation system during the surgery. In fact, during a surgery, not only the main surgeon speaks but the whole surgical team speak which poses a potential problem for this teleoperation setting. In addition, the eight commands (up and down, left and right, forward and backward, and clockwise and counter clockwise) are not enough to change the speed of operation or range of operation of the robot. Based on the received commands, the surgical system plans the path [Reference He, Zhang, Qi, Zhao, Li and Hu14] and the endoscope robot is driven with respect to this path. Further, a robot motion constraint model is developed based on a virtual fixture [Reference He, Hu, Zhang, Zhao, Qi and Zhang15]. In this way, the safety of operation is increased, while the endoscope is inserted into the nostril as the surgeon drags the endoscope. The endoscope is backdriven by using an admittance controller. The NeuRoboScope system is passively back-drivable, and the NeuRoboScope robot has passive gravity compensation system [Reference Dede, Kiper, Ayav, Özdemirel, Tatlıcıoğlu, Hanalioglu, Işıkay and Berker4]. Therefore, the NeuRoboScope system is inherently safe compared to the robot presented in ref. [Reference He, Hu, Zhang, Zhao, Qi and Zhang15].

4. 4. In ref. [Reference Colan, Nakanishi, Aoyama and Hasegawa11], a cooperative strategy in endoscopic endonasal surgery is proposed. The robot that handles the surgical tool is operated by a direct physical contact of the surgeon. This is not a teleoperation system but involves the direct physical interaction of the surgeon and the robotic tool holder. The aim is to increase the precision of the surgical operation. In ref. [Reference Huang, Lai, Cao, Burdet and Phee7], for the teleoperation, a flexible endoscope system hand and foot controllers are examined as the master system. In this scenario, the surgeon is situated at a control station to operate with the hand interface (commercial haptic devices) and a foot pedal with 4 DoFs. The test subjects used bot interfaces separately. The results indicate that hand controllers were found to be relatively more preferable due to the performance received by using them. Both systems presented in refs. [Reference Huang, Lai, Cao, Burdet and Phee7,Reference Colan, Nakanishi, Aoyama and Hasegawa11] have different working principle when compared to the NeuRoboScope system. In NeuRoboScope system, the surgeon can use two hands to operate two different surgical tools, while the endoscope is still under the control of the surgeon via the endoscope robot’s teleoperation system. However, the use of a foot interface could be adapted to the NeuRoboScope system.

5. 5. iArms [Reference Ogiwara, Goto, Nagm and Hongo12] is a robotic system that supports the forearm of the surgeon during the surgery. In this way, the hand trembling and fatigue of the surgeon are aimed to be prevented. iArms follows the surgeon’s arm, but it does not directly handle the surgical tools. Therefore, iArms operation paradigm is totally different from the NeuRoboScope system. Nevertheless, iArms procedure is found to prolong endoscope lens-wiping intervals in endoscopic sinus surgery which results in continuously clear endoscopic images in a study by ref. [Reference Okuda, Okamoto, Takumi, Kakehata and Muragaki16].

4.1. Preferred motion axes of the surgeons

Preferred motion axes of the surgeons refer to the motion characteristics of each surgeon. Surgeons have different handling of the surgical tools, and their postures during the operation are also different from each other. Consequently, their motion profiles when they try to move the surgical tool up and down and sideways are different. In order to identify the preferred motion axes of the surgeon, a methodology is developed. In this methodology, the aim is to identify the motion of the surgeons as they try to move the endoscope up and down, and sideways, respectively. Figure 6 shows these two motions. Blue arrows on the red workspaces indicates the direction of the requested motion of the endoscope. In order to move the endoscope in the desired direction, surgeons start their motion from an initial state which is identified as initial frame $\mathcal{F}_{\textrm{is}}$ and terminate their motion at a final state $\mathcal{F}_{\textrm{ts}}$ . For each of these motions, an axis of rotation is calculated which is indicated with the purple arrows in Fig. 6. As a consequence, an axis for each motion is identified and the motion signals issued for the endoscope are then calculated as the rotations measured about these preferred motion axes.

Figure 6.

Change of the IMU-fixed frame (X-Y-Z axes marked with red) during the initial and terminal states of the identification process. The identified motion axes are indicated with the purple arrows.

4.2. Procedure for identifying the preferred motion axes of the surgeons

In this subsection, the procedure to determine these axes is explained along with a necessary modification to the identified preferred motion axes. Initially, using the setup described in Fig. 4, the surgeon is asked to move sideways. The motion information acquisition by the IMU on the ring is initiated when the foot pedal is pressed. The orientation of the ring at that instant is recorded as the initial frame, and it is represented by a transformation matrix $\hat{C}^{(e,\textrm{is})}\in \mathfrak{R}^{3\times 3}$ which is calculated by using the measured Euler angles about X, Y, and Z axes ( $\theta_{x}^{\textrm{is}}, \theta_{y}^{\textrm{is}}, \theta_{z}^{\textrm{is}}$ ):

(1) \begin{equation}\hat{C}^{\left(e,\textrm{is}\right)}=\hat{R}_{x}\left(\theta _{x}^\textrm{is}\right)\hat{R}_{y}\left(\theta _{y}^\textrm{is}\right)\hat{R}_{z}\left(\theta _{z}^\textrm{is}\right)\end{equation}

Here, $\hat{R}_{i}\left(\theta _{i}\right)\in \mathfrak{R}^{3\times 3}$ defines a rotation matrix about $i$ axis by amount of $\theta _{i}$ , $e$ defines the earth frame, and $is$ defines the initial frame in sideways motion in $\hat{C}^{\left(e,\textrm{is}\right)}$ .

The motion information acquisition by the IMU on the ring is terminated when the foot pedal is released. The orientation of the ring at that instant is recorded as the terminal frame, and it is represented by a transformation matrix $\hat{C}^{\left(e,\textrm{ts}\right)}\in \mathfrak{R}^{3\times 3}$ which is calculated by using the measured Euler angles about X, Y, and Z axes ( $\theta _{x}^\textrm{ts}, \theta _{y}^\textrm{ts}, \theta _{z}^\textrm{ts}$ ):

(2) \begin{equation}\hat{C}^{\left(e,\textrm{ts}\right)}=\hat{R}_{x}\left(\theta _{x}^\textrm{ts}\right)\hat{R}_{y}\left(\theta _{y}^\textrm{ts}\right)\hat{R}_{z}\left(\theta _{z}^\textrm{ts}\right)\end{equation}

In order to determine the nominal rotation axes while moving from the initial frame to the final frame in sideways motion, the transformation matrix between these two frames, $\hat{C}^{\left(\textrm{is},\textrm{ts}\right)}$ , is calculated by (3). Then, by using the Rodrigues’ formula that is presented in matrix form in (4), the nominal rotation axis $\vec {n}_{s}$ is determined where $\vec {n}_{s}$ is a unit vector:

(3) \begin{equation}\hat{C}^{\left(\textrm{is},\textrm{ts}\right)}=\left(\hat{C}^{\left(e,\textrm{is}\right)}\right)^{-1}\hat{C}^{\left(e,\textrm{ts}\right)}=\hat{C}^{\left(\textrm{is},e\right)}\hat{C}^{\left(e,\textrm{ts}\right)}\end{equation}

(4) \begin{equation}\hat{C}^{\left(\textrm{is},\textrm{ts}\right)}=\hat{R}_{{n_{s}}}\!\left(\theta _{s}\right)=\hat{I}\cos \theta _{s}+\tilde{n}_{s}\sin \theta _{s}+\overline{n}_{s}{\overline{n}_{s}}^{t}\left(1-\cos \theta _{s}\right)\end{equation}

Here, $\hat{I}\in \mathfrak{R}^{3\times 3}$ is the identity matrix and $\tilde{n}_{s}\in \mathfrak{R}^{3\times 3}$ is the skew-symmetric matrix form of $\overline{n}_{s}\in \mathfrak{R}^{3\times 1}$ column matrix. The steps for the extraction of the rotation axis $\vec {n}_{s}$ from the Rodrigues’ equation are not given, since the derivation process is textbook information.

The preferred rotation axis identification procedure for the up-down motion is the same, but this time the motion request issued to the surgeon is to move the endoscope upwards or downwards. The resultant of this procedure is the nominal rotation axis $\vec {n}_{u}$ for the up-down motion where $\vec {n}_{u}$ is a unit vector.

To assess if the identified rotation axes are independent from each other, their orthogonality should be investigated by taking the dot product of these two vectors. If $\vec {n}_{s}\cdot \vec {n}_{u}=0$ , then these two axes are perpendicular to each other. If such a case can be reached, the identification of the preferred rotation axes is completed. Generally, such an ideal identification process is not possible. In this case, the common normal of these two vectors $\vec {n}_{s}$ and $\vec {n}_{u}$ is calculated via (5) and named as $\vec {n}_{t}$ :

(5) \begin{equation}\vec {n}_{t}=\vec {n}_{s}\times \vec {n}_{u}\end{equation}

The angle between $\vec {n}_{s}$ and $\vec {n}_{u}$ about $\vec {n}_{t}$ is calculated by using (6), and this angle is distributed evenly to satify the orthogonolity of all axes using (7) and (8) to determine the adjusted rotation axes $\vec {n}_{s'}$ and $\vec {n}_{u}'$ :

(6) \begin{equation}\beta =\cos ^{-1} \!\left(\vec {n}_{s}\cdot \vec {n}_{u}\right)\end{equation}

(7) \begin{equation}\overline{n}_{s'}=\hat{R}_{{n_{t}}}\!\left(-\beta /2\right)\overline{n}_{s}\end{equation}

(8) \begin{equation}\overline{n}_{u'}=\hat{R}_{{n_{t}}}\!\left(+\beta /2\right)\overline{n}_{u}\end{equation}

These preferred axes of rotation define the preferred frame of measurement for each surgeon. This preferred frame’s orientation (identified as $pf$ ) with respect to the earth frame (the frame of measurement of the IMU) is defined as $\hat{C}^{\left(e,\textrm{pf}\right)}=\left[\begin{array}{ccc} \overline{n}_{s'} & \overline{n}_{u'} & \overline{n}_{t} \end{array}\right]$ .

4.2. Implementation of the identification procedure

For the implementing the identification procedure, a new simulator with different hardware setup is developed to represent the surgical procedure of the minimally invasive endoscopic endonasal pituitary tumor surgery with the NeuRoboScope system. This VR screen includes the surgical tools that are used during this surgical process. Figure 7 shows the VR screen of the simulator during the preferred motion identification procedure. The screenshot at the top is captured during the up-down motion, and the one at the bottom is captured during the side-to-side motion. The blue colored objects are generic surgical tools which can be replaced by a specific tool representation. The background image is a photo taken during a surgical procedure, which can also be replaced depending on the location of the simulation. This simulator is not designed to train the surgeon to follow the surgical procedures and complete the surgery. The purpose of the simulator is to identify the surgeon’s preferred frame to acquire motion demands to be issued to the NeuRoboScope system’s active arm. Also, during the actual surgery, the surgeon observes the views captured by the endoscope from a 2-D screen. Therefore, some characteristics for an ideal VR screen such as the level of immersion into the VR and the level of reality become less relevant for this specific purpose.

Figure 7.

VR screen of the new simulator to identify the preferred frame of a neurosurgeon. The top screen capture indicates the up-down motion, and the bottom screen capture indicates the side-to-side motion.

The view changes as a result of the motion of the virtual endoscope modeled in the simulator. The motion of the endoscope is controlled by the virtual active arm. The virtual active arm receives the motion demands via the information flow described in Fig. 3.

The motion of the tools that appear on the VR screen is controlled via the motion information acquired from two haptic devices. Figure 8(a) shows how the haptic devices are places relative to the skull mock-up. This layout is specifically designed so that (1) the surgeon approaching to the patient from the patient’s right side would have enough free space to hold both the handles of the haptic devices, and (2) the haptic device’s motion limits are enough to cover the range of motion of the surgeon’s hand while the surgical tools are inside the surgical workspace. The interaction forces with the surgical environment (concha, tip of the nose, etc.) can also be provided by the use of these haptic devices.

Figure 8.

(a) Layout of the haptic devices with respect to the skull mock-up; (b) the new NeuRoboScope system simulator for the identification of the surgeon’s preferred axes of measurement.

The information exchange between the peripherals and the VR screen is summarized in Fig. 9. Aside from the haptic devices, the necessary input devices are the IMU-embedded wearable ring and the foot pedal of the NeuRoboScope system.

Figure 9.

Peripherals and their information exchange with the VR (virtual endoscope system) of the new simulator for the NeuRoboScope system.

During the preferred frame identification tests, the surgeon’s position with respect to the developed simulator system is shown in Fig. 8(b). This posture mimics the way the main surgeon operates during an endoscopic endonasal pituitary tumor surgery. This simulator also successfully recapitulates bimanual movements of hands and fingers of the surgeon performing the endoscopic endonasal transsphenoidal pituitary surgery.

4.3. Processing the acquired motion for controlling the NeuRoboScope’s active arm

NeuRoboScope system’s new simulator is not developed only to train a surgeon or evaluate the performance of a surgeon. This simulator is also developed to personalize the way that the NeuRoboScope robotic surgery system is used. This approach integrates the surgeon’s needs and behavior to the surgical system’s operation. The next step is to integrate this preferred measurement frame information to the control procedure of the NeuRoboScope system and hence, the control system replicated in the training simulator. Accordingly, the following procedure is defined to be integrated to the NeuRoboScope system’s operational procedure.

During the operation of the NeuRoboScope system, the change of orientation information of the ring, which is received in terms of Euler angles about X, Y, and Z axes, has to be transformed into rotations measured about the preferred axes. The procedure is as follows:

1. 1. Measure the orientation of the ring after the foot pedal is pressed at each time step. Formulate it for step $i$ as a transformation matrix $\hat{C}^{\left(e,i\right)}$ :

   (9) \begin{equation}\hat{C}^{\left(e,i\right)}=\hat{R}_{x}\!\left(\theta _{x}^{i}\right)\hat{R}_{y}\left(\theta _{y}^{i}\right)\hat{R}_{z}\left(\theta _{z}^{i}\right)\end{equation}

2. 2. Measure the orientation of the ring at the next time step and record it as a transformation matrix $\hat{C}^{\left(e,i+1\right)}$ :

   (10) \begin{equation}\hat{C}^{\left(e,i+1\right)}=\hat{R}_{x}\!\left(\theta _{x}^{i+1}\right)\hat{R}_{y}\left(\theta _{y}^{i+1}\right)\hat{R}_{z}\left(\theta _{z}^{i+1}\right)\end{equation}

3. 3. Calculate the transformation matrix that defines the rotation from the ring frame measured at step $i$ to the ring frame measured at step $i+1$ :

   (11) \begin{equation}\hat{C}^{\left(i,i+1\right)}=\left(\hat{C}^{\left(e,i\right)}\right)^{-1}\!\hat{C}^{\left(e,i+1\right)}=\hat{C}^{\left(i,e\right)}\hat{C}^{\left(e,i+1\right)}\end{equation}

4. 4. Transform $\hat{C}^{\left(i,i+1\right)}$ into the preferred frame so that the Euler angles solution for X-Y-Z Euler sequence can be applied to calculate the rotations about the preferred axes. In this calculation, previously determined $\hat{C}^{\left(e,\textrm{pf}\right)}$ transformation matrix is used which is identified for each surgeon:

   (12) \begin{equation}\hat{C}_\textrm{pf}^{\left(i,i+1\right)}=\hat{C}^{\left(\textrm{pf},e\right)}\hat{C}^{\left(i,i+1\right)}\hat{C}^{\left(e,\textrm{pf}\right)}=\hat{R}_{x}\left(\theta _{s}^{i+1}\right)\hat{R}_{y}\left(\theta _{u}^{i+1}\right)\hat{R}_{z}\left(\theta _{t}^{i+1}\right)\end{equation}

The calculated angles that define the rotation from step $i$ to step $i+1$ are issued as velocity demands ( $\omega _{\textrm{pitch}},\omega _{\textrm{roll}}, v_{\textrm{doly}}$ ) to the endoscope robot. Here, at step $i+1$ , $\omega _{\textrm{pitch}}^{i+1}=\theta _{s}^{i+1}+\theta _{s}^{i}$ is the angular speed demand issued for sideways motion, $\omega _{\textrm{roll}}^{i+1}=\theta _{u}^{i+1}+\theta _{u}^{i}$ is the angular speed demand issued for up-down motion, and $v_{\textrm{dolly}}^{i+1}=\theta _{t}^{i+1}+\theta _{t}^{i}$ is the speed demand issued for translational motion. These are the speed inputs issued to the endoscope robot as defined in Fig. 3.

Although the main aim of the simulator is for personalization of the NeuRoboScope system for every surgeon, it shown via the above-mentioned procedure, this new simulator can be modified for training and evaluation of the surgeons in using the teleoperation system of the NeuRoboScope system. Figure 10 shows a photo of a surgeon while having training on using the teleoperation system of the NeuRoboScope system. The surgical tools, skull mock-up, IMU-embedded ring, and the changing view of the VR screen can be observed on this figure. In this training session, the haptic devices were not used and instead actual surgical tools were handled by the surgeon working on the skull mock-up.

Figure 10.

A surgeon’s operation during a training session.

5.1. Participants

Ten participants were recruited as test subjects (three females, age: 20–44 years, M = 28.30, SD = 6.22) and gave written informed consent for their participation. Measurements of task completion duration and number of foot pedal pressings are recorded and used for all of the test subjects. The study was approved by the ethics committee of Izmir Institute of Technology.

5.2. Test procedure

A test scenario is generated to assess how trivial the use of NeuRoboScope system becomes when the personalization is used. The test screen is shown in Fig. 11. The crossed arrows in the middle indicated the mid-position of the endoscope view. The aim is to locate this mid-position to the circle that appears on the screen, which are called the target positions. Since 3-D feature is not present (it is also not present in the actual surgery), to indicate the depth of the target, a color code is used. If the color of the circle is red, the user should zoom out from the scene by retracting the telescope towards the pivot position and if the color of the circle is blue, the user should zoom in to the scene by moving the telescope away from the pivot point.

Figure 11.

Screen captures during the tests.

Each test subject used a non-personalized mode and the personalized mode to compare the effect of personalization. Ten repetitions are made with each mode by all the test subjects. The non-personalized mode is the mode which has a firm contact of the fingers with the tool, and the Euler angle measured about the Z axis is not used. The reasons to choose this non-personalized mode is that (1) the Euler angle measured about the Z axis and this motion is a very hard motion for the users, and (2) the surgeons, during the surgery, need a firm grasp of the surgical tool.

5.3. Test results

The results are obtained in terms of task completion durations presented in Fig. 12 and number of pedal pressings presented in Fig. 13.

Figure 12.

Average task completion durations and their standard deviations throughout the trials.

Figure 13.

Average number of pedal pressings and their standard deviations throughout the trials.

The average task completion duration of the non-personalized mode is calculated to be about 100 s in the first trial, and after the fifth trial, it converged to 40 s. The average task completion duration of the personalized mode is calculated to be about 60 s in the first trial, and after the third trial, it converged to 30 s.

During the tests with the non-personalized mode, the average number of pedal pressings decreased from 38 in the first trial to about 15 after the fifth trial. The average number of pedal pressing is measured to be consistently about 10 after the first trial.