3.1. Skill performance evaluation

Developing an effective training strategy requires an initial assessment of the trainee’s skill level. However, in real-world operational scenarios, it may be challenging to quantitatively express complex, multi-step processes’ desired trajectories in advance. The evaluation of skills in this study involves acquiring the trainee’s skill characteristic, which can be achieved through various studies tailored to different types of tasks. For example, in characteristic acquisition studies, the EGNN paradigm [Reference Liu, Yang, Wang, Lu and Li18], transfer feature learning [Reference Shi, Li, Yang, Wang and Hao19], and heterogeneous network representation [Reference Wang, Liu, Xu and Yan20] approaches can be utilized to uncover information about the features associated with the skill. This approach is applicable to more open-ended skill training scenarios, such as playing musical instruments without specific task objectives or engaging in ball sports. On the other hand, there are studies that focus on tasks with defined objectives, such as surgical scenarios, welding scenarios, or assembly processes. These studies typically examine the impact of operation presentation and construct characteristic based on the effects of the operations.

The research in this paper focuses on dual-user operation training under sharing control, so the characteristics of skills are all indicators under the conventional definition. According to the summary of the study [Reference Cotin, Stylopoulos, Ottensmeyer, Neumann and Dawson21], the length of the operation as well as the smoothness of the operation are selected as the trainee’s skill characteristics. This was also applied and analyzed in refs. [Reference Shahbazi, Atashzar, Ward, Talebi and Patel5, Reference Cotin, Stylopoulos, Ottensmeyer, Neumann and Dawson21–Reference Hogan and Flash23] to represent the operator’s skill performance, and after acquiring the skill characteristics, they were quantified into numerical values by evaluating them. In real-time training, this assessment should accompany the trainee’s real-time operation. Therefore, the assessment of the trainee’s skills should be done during the session rather than waiting for the end of the task. To address this issue, a task-independent assessment approach is adopted to objectively evaluate the trainee’s performance. This approach differs from absolute assessment methods and emphasizes the importance of evaluating the trainee’s relative performance level. The performance of the trainee’s skills is a dynamic assessment of the operation in progress with the expert, and its goodness also depends on the level of the expert at the time of cooperation.

A normalized task-independent metric, $\varPhi$ , has been defined to determine the desired quantitative performance of the trainee, as shown in Eq. (2).

(2) \begin{equation}{\varPhi }(t)=1-\left | \frac{{{X }_{E}}(t)-{{X }_{T}}(t)}{{{X }_{E}}(t)+{{X }_{T}}(t)} \right |, \end{equation}

The defined metric $\varPhi$ reflects the trainee’s relative skill performance compared to the expected level. It should be noted that for complex operations, the expected value should not be stereotyped, and expert operation data serves as a reference for determining the expected value in real time. Compared to absolute assessment methods, this quantitative performance approach provides a more objective evaluation of the trainee’s relative performance level. Furthermore, the normalization process facilitates calculations during agent learning, providing additional advantages.

In the development of a training strategy, the identification of relevant skill characteristics must be task-specific. For common operational tasks, an effective evaluation of a trainee’s skill level can be based on instrument motion alone. In this study, we selected path length and motion smoothness as the two features for performance evaluation. These characteristics were chosen based on two key considerations. Firstly, as both the trainee and expert perform the task simultaneously, they consume the same amount of time. Typically, the expert’s trajectory is more concise, indicating a deeper understanding of the task and resulting in a shorter path length compared to that of the trainee during the same time duration. Secondly, motion smoothness reflects the operator’s motor control ability, making it a crucial factor in evaluating operational performance. Therefore, these two features comprehensively cover most operational skill performance and are easily measurable.

The path length: This is a feature reflecting the operation trajectory, which is calculated as Eq. (3a), and the normalized task-independent value for $\varrho$ is given in Eq. (3b):

(3a) \begin{equation}{{\varrho }}=\int _{0}^{t}{\sqrt{{{\left ( \frac{d{x}}{dt} \right )}^{2}}+{{\left ( \frac{d{y}}{dt} \right )}^{2}}+{{\left ( \frac{d{z}}{dt} \right )}^{2}}}}d\tau, \end{equation}

and

(3b) \begin{equation}{{\varPhi }_{\varrho }}(t)=1-\left | \frac{{{\varrho }_{E}}(t)-{{\varrho }_{T}}(t)}{{{\varrho }_{E}}(t)+{{\varrho }_{T}}(t)} \right |, \end{equation}

where ${{\varrho }_{E}}(t)$ and ${{\varrho }_{T}}(t)$ denote the path length performance for expert and trainee, respectively.

Operation Smooth: Instantaneous jerk is a measure of the rate at which acceleration changes over time and is quantified as $J=\frac{{{d}^{3}}x}{d{{t}^{3}}} cm/s^{3}$ [Reference Hogan and Flash23]. The cumulative value of instantaneous jerk can be calculated using Eq. (4a) and is considered to be indicative of the smoothness of an operation during a task:

(4a) \begin{equation}{\nu }=\int _{0}^{t}{\sqrt{{{\left ( \frac{{{d}^{3}}{{x}}}{dt^{3}} \right )}^{2}}+{{\left ( \frac{{{d}^{3}}{{y}}}{dt^{3}} \right )}^{2}}+{{\left ( \frac{{{d}^{3}}{{z}}}{dt^{3}} \right )}^{2}}}}d\tau, \end{equation}

and the normalized task-independent value for $\nu$ is Eq. (4b):

(4b) \begin{equation}{{\varPhi }_{\nu }}(t)=1-\left | \frac{{{\nu }_{E}}(t)-{{\nu }_{T}}(t)}{{{\nu }_{E}}(t)+{{\nu }_{T}}(t)} \right |, \end{equation}

The value is represented by a ratio relation between the trainee and expert and falls within the range [0–1]. The performance parameters ${{\varPhi }_{\varrho }}(t)$ and ${{\varPhi }_{\nu }}(t)$ are then linearly fused to obtain the trainee’s skill state parameter, ${{\varPhi }_{\varDelta }}(t)$ , which is:

(5) \begin{equation}{{\varPhi }_{\varDelta }}(t)={{\varPhi }_{\varrho }}(t)\cdot{{\varPhi }_{\nu }}(t) \end{equation}

In order to further enhance the examination of the selected skill indicators, an additional approach can be employed wherein newly acquired quantitative eigenvalues are multiplied continuously. This innovative idea is inspired by the study [Reference Shahbazi, Atashzar and Patel24] and serves to amplify the analysis of the characteristics of these indicators.

The skill assessment phase serves as a crucial data preprocessing component within the interaction process of an intelligent agent. During this phase, the agent solely perceives and processes quantized skill-level information. Subsequently, the agent generates tailored training strategies based on the perceived skill level. By focusing on quantized skill-level information, the agent simplifies the complexity of the data, enabling efficient processing and analysis. This approach allows the system to distill the essential aspects of the trainee’s skill level, facilitating the generation of effective and personalized training strategies. Leveraging the perceived skill-level information, the intelligent agent leverages advanced algorithms and methodologies to develop training strategies that are specifically designed to optimize the trainee’s learning experience and enhance skill acquisition. These strategies are dynamically generated based on the real-time skill level, ensuring adaptability and responsiveness to the trainee’s needs.

3.2. Adaptive engagement regulation

Training is a systematic process that involves the acquisition of knowledge, skills, and attitudes to enhance performance in a specific setting. Adaptability is a critical element in ensuring training effectiveness. Kelly (1969) defined adaptive training as the variation of the problem, stimulus, or task based on the trainee’s performance. This approach consists of three primary components: performance measures, adaptive variables, and adaptive logic [Reference Kelley25]. Given the resource-intensive and time-consuming nature of training, an adaptive training protocol can significantly reduce training time compared to conventional step-by-step approaches. The integration of adaptation into training has been widely implemented across many domains, including older adult cognitive training [Reference Peretz, Korczyn, Shatil, Aharonson, Birnboim and Giladi26], professional athlete physical training [Reference Karanikolou, Wang and Pitsiladis27], game-based training [Reference Serge, Priest, Durlach and Johnson28], rehabilitation [Reference Rossol, Cheng, Bischof and Basu29], medical training [Reference Mariani, Pellegrini, Enayati, Kazanzides, Vidotto and De Momi30], and stress management training [Reference Popovic, Horvat, Kukolja, Dropuljić and Cosic31].

Personalizing training strategy and collecting user performance data requires an initial investment, but the benefits of reducing training time and matching learners’ needs outweigh the initial cost [Reference Bagnara, Tartaglia, Albolino, Alexander, Fujita, Bagnara, Tartaglia, Albolino, Alexander and Fujita32]. Furthermore, repetitive and ineffective training can hinder skill learning efficiency; thus, it is advisable to avoid such approaches.

Engagement is a critical factor in human motor skill training. It involves the active participation of trainees in skill acquisition, which enhances their performance [Reference Newton, Lepine, Ji, Wellman and Bush33]. Effective participation helps trainees master skills and knowledge comprehensively, motivates them to complete tasks with greater effort, provides feedback information, and more accurately measures their performance. Notably, engagement is a complex construct with context-dependent and multifaceted forms. Given that trainees’ control in shared control-based dual-user training systems is considered their engagement in the training process, this paper aims to enhance engagement by adaptively adjusting trainees’ control authority based on their skill level. This approach will contribute to establishing adaptive logic and promoting skill acquisition.

4.1. Learning problem formulation

In this paper, the adaptive engagement regulation problem is defined as a sequential decision-making process, and this is a finite MDP with time step $t = \left \{1,2,3,\ldots,T\right \}$ , which is reinforcement learning problem [Reference Sutton and Barto34]. Four-tuple $(S, U, P, r)$ are defined, where $S$ represents the observation space of the system,  $U$  denotes the permissible actions, $P$ represents the observation transition model, and $r$ is the immediate reward.

Figure 2 depicts the overall diagram of the DRL-based agent decisions for engagement regulation, which uses a simulated environment. The trainee’s skill information from the simulated environment is fed into an Actor-Critic network to approximate the action value of the agent decision $u$ . Skill information is acquired according to the calculations in Section 3.1. The decision with the highest action value, $u_i$ , is then selected for the agent.

4.1.2. Action space

Within the training policy, the agent’s actions have an impact on the control authority, which in turn influences both the slave robot’s performance and the trainee’s engagement. The agent’s decision is defined by a simple model, shown in Eq. (6).

(6) \begin{equation} \omega (t)=\zeta \cdot \hat{\omega }(t), \end{equation}

where $\zeta$ represents the maximum level of control authority that the trainee can engage with, as determined by the instructor to meet safety requirements. The action executed by the agent is denoted by $\hat{\omega }(t)$ and ranges from 0 to 1. Consequently, the action space is defined as [0, 1].

4.1.3. Transition probabilities

The transition from the current state $ s_{t}$ to the next $ s_{t+1}$ is defined as

(7) \begin{equation} s_{t+1} = f(s_{t}, \hat{\omega }_{t}). \end{equation}

The agent decision is updated based on a vague mechanism of human ability growth. Subsequently, the agent alters the trainee’s operation authority via a shared-control scheme. Modeling the accurate transition function for trainee performance change is challenging due to the highly complex nature of human skill progression. In this paper, a data-driven approach is proposed to provide the control action.

4.1.4. Reward

During the training process, the reward value obtained by the agent is based on the trainee’s engagement in the task and the operating trajectory of the slave robot. The reward function is composed of two components: the difference between the trainee’s engagement and their skill level at each moment, as well as the deviation between the slave robot’s operating trajectory and the desired trajectory. Therefore, the reward function is defined as Eq. (8)

(8) \begin{equation}{r}_{t} = \omega (t)-\Delta (t). \end{equation}

The performance of slave robot is defined as Eq. (9).

(9) \begin{equation} \Delta (t) ={{X}_{R}}(t) -{{X}_{D}}(t), \end{equation}

In this paper, $\Delta (t)$ represents the degree of similarity between the slave output curves and the expected curves.

The agent’s decision can be implemented by controlling the weight of authority to find an optimal policy related to $\omega$ for trainees with varying skill levels. When deploying the agent in the training system, at time $t$ , the agent observes the current skill level of the trainee, and based on this state the agent executes an action to change the trainee’s engagement in the training system. The objective is to maximize trainee engagement while minimizing deviations in performance.

4.2. Proposed approach

This paper utilizes Deep Deterministic Policy Gradient (DDPG), an actor-critic-based method, to regulate trainee engagement, as shown in Fig. 2. This structure comprises four neural networks: the actor network that outputs a continuous action $\hat{\omega }$ ; the critic network that evaluates the executed actions’ performance $Q-value$ , and two target networks for the actor and critic networks, respectively, ensuring convergence of the $Q$ function. It is worth noting that although this implementation involves four networks, the actor-critic method can share the same network structure with their respective target network in practice. The network structure is as follows:

1. (1) Actor Network: In the actor network, the output is a continuous action, which represents the agent’s decision. A deterministic policy, $\mu(s)=\hat{\omega}$ , is defined with parameters $\mu_{\theta}$ . For each state $s$ , this policy outputs the action $\hat{\omega}$ that maximizes the action-value function $Q(s,\hat{\omega})$ . Therefore, the optimal action can be represented as Eq. (10)

   (10) \begin{equation}{{\hat{\omega }}^{*}}=\max _{\theta }\mathbb{E}[{{Q}}(s,{{\mu }}(s|{\theta _\mu }))], \end{equation}

2. (2) Critic Network: The critic network’s output is $Q$ -value, which is approximated by the $Q-value$ with parameter $\theta _Q$ . Because the optimal policy is deterministic, the optimal action-value function can be described as Eq. (11)

   (11) \begin{equation} \!\!{Q}^*({{s}_{t}},\!\hat{\omega _t})\!=\!{{\mathbb{E}}_{{{s}_{t+1}}\sim \mu }}\!\left [ r({{s}_{t}},\!{\hat{\omega }_{t}})\!+\max _{\mu ({s}_{t+1})}\!\gamma \!\left [{{Q^*}}({{s}_{t+1}},\!{\mu ({s}_{t+1}|\theta _Q)}) \right ] \right ], \end{equation}

3. (3) Target Networks: The target network is a copy of the main network that is updated slowly using a soft update rule. Two target-networks are introduced, one for the $Q$ network and the other for the actor network. They are defined as $\theta '_\mu, \theta '_Q$ . To ensure stability during network updates, the target networks are updated less frequently than the main networks. The update methods for these networks are as follows:

   (12a) \begin{equation}{\theta '_Q}\leftarrow \tau{\theta '_Q}+(1-\tau ){\theta '_Q} \end{equation}
   and
   (12b) \begin{equation}{\theta '_\mu }\leftarrow \tau \theta '_\mu +(1-\tau ){\theta '_\mu } \end{equation}
   The term $\tau$ is a hyperparameter between 0 and 1, this term can make the update of the target network lag behind the main network. All of the hyperparameters used in our DDPG control algorithm are listed in Table I.

Table I.

Hyperparameters in DDPG controller.

Specifically, $\alpha _a$ and $\alpha _c$ represent the learning rate of actor and critic network, respectively. Furthermore, $\tau$ is target smoothing coefficient that we use to balance the weight updates of the two networks in order to optimize performance. Additionally, $N_a$ and $N_c$ represent the number of nodes in actor and critic network; the $\gamma$ is a discount factor and set as 0.99; the batch size (BS) is 256; and the update frequency for target networks is 3.

4.3. Training of the networks

The process of parameter learning is illustrated in Algorithm 1. This algorithm details the training procedures for both the Actor and Critic networks. $\gamma$ is discounted factor. The BS is the size of batchsize, and the UF is the update frequency of the target network.

Algorithm 1

Training of the Actor and Critic Network.

Algorithm 1 takes the training system state $s$ and reward $r$ as input, and returns the actor-network parameter ${\mu }_{\theta }$ and the critic-network parameter $\phi$ , respectively.

At the beginning of each episode, the environment is initialized. In line 1, all parameters are randomly initialized, followed by the initialization of all target networks. The iteration starts from line 3, and the parameters $\theta$ are updated through $N$ episodes. When observing the first state $s_1$ , the inner loop from line 5 to the line 6, the agent will select an action based on the current policy $\mu(s_t|\theta_{\mu})$ . To ensure the exploration, a noise $\mathcal{G}_{t}$ is used. This noise added can prevent the local optimum in training. Ornstein-Uhlenbeck Process (OUP) is chosen as the noise because of its temporally correlated nature. More details about OUP can be found in study [Reference Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa, Silver and Wierstra35]. After executing the action, an immediate reward is received. The buffer $\mathcal{B}$ , which is in line 3 and line 8, is used to store the interaction trajectories. This is the so-called experience replay approach. Specifically, in line 9, a minibatch of tuples $(s_{i},\omega _{i},r_{i},s_{i+1})$ is randomly sampled from the buffer $\mathcal{B}$ . From line 10 to line 13, the parameters are updated by minimizing the loss function. In line 13, two equations represent a soft update in target networks. $\tau$ is a very small factor, and it is set as 0.001, which is used to overcome the unsuitability of neural network training. This soft update method can improve the stability of the learning process.

Upon completion of the training process, the learned parameters $ \theta$ will be employed for real-world engagement adjustment scenarios.

4.4. Adaptive regulation for trainee engagement

Algorithm 2 describes the engagement adjustment process. Its input is the trainee’s skill state, and its output is the agent’s decisions regarding the trainee’s level of engagement. In line 1 of Algorithm 2, the trained parameters $\theta$ from Algorithm 1 are loaded. The initial level of engagement $\omega _{0}$ for the trainee is defined as an authority in the control workspace. Starting from line 3 of the algorithm, the agent is controlled by the DDPG algorithm to regulate the trainee’s engagement over a time period of T steps. At each interaction, the data of the system is obtained from the trainee’s operation device. This data is then fed into the DDPG algorithm to model the action and state of the system, as described in Section IV-A. In line 6, these states are used as inputs to the critic network, which calculates the action-value $Q(\boldsymbol{s}_{t},\boldsymbol{\hat{\omega }};\theta )$ for the agent’s decisions. Then, in line 7, the decision $\hat{\omega }$ is selected as $Q^*({\boldsymbol{s}_{t}},\boldsymbol{\hat{\omega }};\theta )$ . Finally, $\omega _t$ is outputted as the engagement policy.

Algorithm 2

Agent Decision-Making.

5.1. Setup design and implementation

In order to evaluate the proposed method, we set up a dual-console platform designed to gather operation information. The platform consists of two Omega 3 devices deployed on the master side and a Kuka iiwa robot on the slave side. In this setup, an expert manipulator and a trainee manipulator are connected to a computer, which is responsible for fusing the two control signals from the master side and transmitting the synthesized signals to the slave robot. The application software that communicates with the haptic devices is implemented in C++ using a Qt API. Meanwhile, the computer also records operation information from the two users at a sampling interval of 0.001 s. The slave robot receives the control command from the computer and the operation track in space is mapped in a 1:1 coordinate system. This experimental platform is applicable to the scenario of dual-user training, facilitating the collection of operation information during the training process and providing verification for future training strategies. Figure 3 displays the device composition. The instantaneous output $X_R$ corresponds to the synthetic track calculated by Eq. (1) and is generated jointly by the trainee and the expert on the master side. The control authority is determined by the agent.

Figure 3.

Dual-user training platform.

The parameters learning of agent is conducted via another computer. Specifically, the training and verifying of our proposed approach were performed on an Apple M1 GPU. The Actor and Critic networks were implemented in Python, while the combined control signal was developed using Gym.

5.2. Simulation scenarios

We conducted extensive simulations with a random initial shared policy as well as different skill levels of the trainee. Two operators participated in the experiments, simulating both an expert’s and a trainee’s behaviors. We considered three scenarios for interpretation, where the trainee’s skill level is represented as a normalized value showing their relative performance compared to that of the expert. As can be easily seen from Eq. (2), this value ranges from 0 to 1. In every episode start, the trainee’s control authority is set from [0–1] randomly. Furthermore, we define task safety as the root mean square error (RMSE) calculated from the deviation between the actual operational trajectory and the desired trajectory. Specifically, in this study, we investigate the impact of varying safety requirements on the effectiveness of the proposed approach in different scenarios. The safety requirements are manually set by experts based on the specific demands of each task. This enables us to explore the use of diverse safety requirements for different scenarios and evaluate the effectiveness of the proposed approach under various levels of safety requirements.

Scenario I: Two operators are required to perform the same task simultaneously, and there are no fixed requirements on how to perform the task. The trainees’ operations will be based on their own understanding of the task, which may result in a diverse range of trajectories and problem-solving methods. The expert’s operational trajectory will serve as a reference for comparison, enabling the evaluation of each trainee’s performance in relation to the established standard. The trajectories produced during operation are shown in Fig. 4 and Fig. 6.

Figure 4.

Operation track comparison in space without agent.

Scenario II: This scenario involves an operator performing a task within a specific restricted range, with the resulting trajectory displayed in Fig. 8(a), (b), and (c). The three cases presented correspond to three distinct skill levels, namely 0.9, 0.4, and 0.8. In this setting, an expert performs the operation only once, and their performance data is stored as a reference. Three different individuals then perform the task according to predefined instructions.

Operating within a restricted range provides the operator with guidance, which we leverage to enhance safety requirements during the operation. Specifically, we set the maximum RMSE requirement for all cases to be no higher than 0.15.

Scenario III: In the scenarios I and II, the reference trajectories for comparing is provided by the expert’s operation in real-time. In this scenario, we introduce a desired trajectory to evaluate the operation. The expert and trainee will perform the task operations simultaneously, similar to previous scenarios, while combined operation signals will be utilized to calculate deviations from the desired trajectory. This enables us to assess both training engagement and operational safety by examining deviations from the desired trajectory. The safety requirements in this scenario need to be determined depending on the different tasks, so the specific values will be illustrated in their case.

5.3. Learning process of agent

5.3.1. Scenario I

Case 1-1: In this case, Fig. 4(a) depicts information regarding the operation in space during task execution. The solid blue line represents the trajectory of the trainee at the end of the operated device, while the red dashed line shows the trajectory formed by the end of the operated device of expert. Figure 4(b) displays the difference in trajectory under 2-D plane, which clearly depicts the difference between the trainee and the expert in the operation process. During the task execution, it was observed that the trajectory formed by the trainees was less smooth than that of the experts. Furthermore, the trainees exhibited numerous redundant movements while attempting to complete the task. These observations suggest that trainees may have a less refined understanding of the motor skills required for the task compared to the experts. To evaluate the trainee’s skill, an analysis was conducted on the total length and smoothness of the trajectories of both the trainee and the expert. Based on this analysis, the trainee’s skill was assessed as 0.92. In the experiment, the RMSE of the output trajectory formed by the combined control signals of the trainee and the expert, and the trajectory between the experts, was utilized as a deviation measure. To ensure the end output remains within an acceptable safety range, the value of this RMSE must not exceed 0.2.

To determine the optimal trainee engagement based on the trainee’s skill evaluation, the agent was employed to observe the skill level to find the control authority that suit the trainee’s skill level.

We present the training performance by running Algorithm 1, the agent is trained for 200 episodes to learn the optimal decision about engagement. A total of 40,000 steps is run, and the simulation time for each episode is 200 s. The trainee’s engagement is randomly chosen from [0–1] in each episode. The evolution of the accumulated reward at T = 200 s over 200 episodes is shown in Fig. 5(a). The average decision in each episode is shown in Fig. 5(b)

Figure 5.

Evolution of the accumulated rewards at T = 200 s over 200 episodes during the training process. The right figure is the average decision in each episode.

Figure 6.

Operation track comparison in space without agent.

Figure 7.

Evolution of the accumulated rewards at T = 200 s over 200 episodes during the training process. The right figure is the average decision in each episode.

From the results, it can be seen that the rewards obtained by the agent gradually increase, and the decision of the agent gradually converges from 50 episodes.

Case 1-2: In this case, we intentionally tested the decision-making abilities of the agent by simulating a task operation resulting in a trainee performance level of 0.74. By observing the agent’s decision-making process under conditions of reduced operational skill, we aim to gain insights into its ability to adapt and perform effectively in different scenarios.

The trajectory of two operators is shown in Fig. 6.

Figure 6(a) shows the operational trajectory information in 3D space, and Fig. 6(b) is the 2D plane display of the operational information. From the trajectory information, it can be observed that in order to complete a task, the trainees need to execute a trajectory of higher length than that of the expert, and the acceleration variation generated by it is also higher than that of the expert. Finally, in this case, the trainee’s skill was evaluated as 0.74. In this case, the evolution of the accumulated reward is show in Fig. 7(a). Concurrently, Fig. 7(b) illustrates the mean decision value throughout the training process.

5.3.2. Scenario II

Case 2-1: In this case, the trainee’s skill level is evaluated as 0.9 as in Fig. 8(a). In the training process, the reward gained is shown in Fig. 9(a).

Figure 8.

Operation track comparison in three-dimensional space without agent.

Figure 9.

Accumulated rewards in three cases.

Case 2-2: In case 2, the trainees’ skills improved and the level skills are assessed as 0.4, and the trainee is asked to follow a predefined trajectory. As in Fig. 8(b), two users’ tracks are shown. Through 200 episodes, the learning process of the agent is shown in Fig. 9(b).

Case 2-3: In case 3, the trainee’s level is 0.8, his/her track as shown in Fig. 8(c). Figure 9(c) shows the accumulated rewards in this case during the training process.

From the accumulated rewards obtained by the agent over the 200 episodes, the accumulated rewards gradually increase and eventually converge. In this scenario, the average decision under different skill levels is shown in Fig. 10(a),(b), and (c). At three different skill levels, the average decision eventually converges around 0.90, 0.55, and 0.86.

Figure 10.

Average decision in three cases.

5.3.3. Scenario III

Case 3-1: This case has similarities with Scenario II, as both operators are required to operate within the designated range of operation. However, in this case, a desired operation trajectory is added, which is depicted by the green line in Fig. 11. The red and blue lines in Fig. 11 represent the expert and trainee operators, respectively. To evaluate the performance of the trainee operator, we compare their operation to that of the expert’s using relative scores. In the final evaluation of the operation, we measure the combined operation trajectory against the desired trajectory path, enabling us to establish the degree of deviation from the expected trajectory path.

Figure 11.

The operation trajectory with a expected reference.

In this case, we set the value of the safety factor so that the rmse value of the trajectory error does not exceed 0.5. For this case, the learning of the agent is shown in Fig. 12. Figure 12 presents the accumulated rewards obtained and the change in the average trainee engagement. From the results, it can be seen that the rewards gradually increased and the engagement level also converged. The convergence value of the engagement is finally equal to approximately 0.60.

Figure 12.

Evolution of the accumulated rewards at T = 200 s over 200 episodes during the training process. The right figure is the average decision in each episode.

Case 3-2: In this case, the operation without constraints and the trajectory is shown in Fig. 13. The green line is the desired trajectory. Similarly, red and blue are the operation trajectories of experts and trainees. It can be seen from the figure that the expert’s trajectory is more similar to the desired trajectory. In this case, the trainee’s skill level was assessed as 0.55.

Figure 13.

The operation trajectory with a expected reference.

The safety requirement is set as 0.33. The accumulated rewards in 200 episodes is shown in Fig. 14. The average engagement of trainee is converged to 0.55.

Figure 14.

Evolution of the accumulated rewards at T = 200 s over 200 episodes during the training process. The right figure is the average decision in each episode.

The results of this training demonstrate the effectiveness of the proposed approach in learning a training policy that maximizes trainee engagement under varying skill levels while ensuring safety. Through this approach, we were able to provide trainees with training scenarios that adapted to their individual skill levels, resulting in higher engagement and safety.

5.4. DRL control for engagement regulation

After the training, Algorithm 2 is run and presents the agent decision results in this section. The engagement will be controlled by the agent. The proposed approach is evaluated under all cases defined in Section V-A. In each case, we conducted extensive 10 simulations with random initial engagement.

Scenario I: In case 1, we evaluated the trainee’s skill level as 0.92 while randomly selecting their engagement level from the 10 available options. The results presented in Fig. 15(a) demonstrate that the engagement level can converge to 0.81 in all tests, considering the trainee’s skill level and safety requirements. These results suggest that the agent can successfully adjust the engagement level to 0.81 irrespective of the initial engagement level, based on the trainee’s skill level. This improves training outcomes by ensuring an appropriate level of engagement for optimal performance while maintaining safety standards. Figure 15(b) corresponds to the second case of scenario I, whose trainees’ skills were assessed as 0.74 and the optimal engagement was 0.57.

Figure 15.

The agent’s decision in Scenario I.

Scenario II: We loading the trained model into Scenario II, and 10 random initial engagements are deployed. The decision of the agent is shown in Fig. 16.

Figure 16.

The agent’s decision in Scenario II.

Figure 17.

The agent’s decision in Scenario III.

Scenario III: When load the model into Scenario III, the decisions of agent are presented in Fig. 17.

In case 1, the decision is converged to 0.61. In case 2, the decision is converged to 0.55.

The statistical results above reflect the method’s ability to adapt training strategies for different levels of trainees based on different safety requirements. When the trainee’s skills change, the agent is able to find the optimal training strategy based on the current skill level to ensure maximum engagement and task safety.

5.5. Error analysis for engagement

In this section, we compare the error in the operational effect produced by the controlled robot at its end, with and without the agent’s regulation of the trainee’s participation. Specifically, we evaluate the impact of the agent’s regulation on the error in the operational effect generated by the controlled robot.

It describes the results obtained from the proposed approach, indicating that the deviation of the operation is reduced when using the agent’s regulation compared to the case without the agent (Fig. 18). Additionally, the safety requirements are met by the agent in both cases. The conclusion highlights the effectiveness of the proposed approach in improving trainee participation and ensuring task safety.