2.1. Slip detection and prevention

Slip detection research can be broadly classified into three main categories: gross slip, involving the complete displacement of the object surface; incipient slip, where partial slippage occurs; and slip prediction, which utilizes tactile features to anticipate slip events [Reference James and Lepora7]. Slip prevention strategies encompass both model-based techniques employing concepts such as friction cones and beam bundle models and model-free approaches including supervised and deep learning methodologies [Reference Romeo and Zollo4, Reference James and Lepora7]. Detection methodologies encompass friction-based methods utilizing multi-axial force sensing, analysis of tactile sensor signals, and machine learning-based classifiers [Reference Romeo and Zollo4]. An array of sensors, including pressure-resistance, optical, piezoelectric, and thermal sensors, is employed for slip-detection purposes [Reference Yang and Wu8]. Advanced signal processing techniques, such as Fourier transforms and wavelet decomposition, play a crucial role in enhancing slip detection capabilities [Reference Romeo, Lauretti, Gentile, Guglielmelli and Zollo5]. However, these methodologies often necessitate manual thresholding of tactile sensing signals, thereby constraining the automation of slip detection for objects with unknown properties [Reference Romeo and Zollo4].

Slip prevention strategies are broadly classified into two categories: reactive and proactive methods [Reference Nazari and Mandil9]. Reactive approaches involve responding to detected slippage signals, while proactive methods anticipate and provide warnings of impending slips before they occur. The integration of slip signals into control algorithms for bionic hands poses notable challenges [Reference Romeo, Lauretti, Gentile, Guglielmelli and Zollo5]. Advanced control algorithms such as PID, sliding mode control, fuzzy control, and model predictive control are employed by researchers to prevent slippage [Reference Yang and Wu8]. Typically, slip prevention entails the use of closed-loop controllers for either force or position control, as shown in Figure 1. Position-based controllers are less favored due to the variability in object stiffness [Reference Yang and Wu8]. Force control, complemented by an inner position loop, is commonly adopted to address objects with varying stiffness and to effectively handle unexpected disturbances for slip prevention [Reference Siciliano, Sciavicco, Villani and Oriolo10, Reference Carbone, Iannone and Ceccarelli11] (shown in Figure 2). Adaptive sliding mode controllers have demonstrated effectiveness in ensuring grasp stability [Reference Engeberg and Meek12], although their integration into bionic hands necessitates the automation and elimination of thresholding signals.

2.2. Deformation detection and control

Preventing deformation in robotic systems necessitates effective stiffness detection or deformation measurement, constituting a challenging and ongoing research endeavor [Reference Zhu, Cherubini, Dune, Navarro-Alarcon, Alambeigi, Berenson, Ficuciello, Harada, Kober and Xiang3]. Current methodologies for stiffness detection and control encompass several approaches. Intrinsic vibration frequency-based signal processing: This method analyzes the vibrational response to ascertain object stiffness via frequency domain decomposition, offering precise measurements albeit typically utilized for offline analysis. Time-domain analysis methods: Monitoring parameters such as equivalent force, deflection, and velocity in real-time facilitates the deduction of object stiffness based on these characteristics. Integration of measuring devices: This approach entails the incorporation of specialized measuring apparatus at the robot gripper’s terminus, correlating material stiffness with gripping forces post-contact. However, it may not be suitable for prosthetic hands due to size and weight constraints. Hooke’s law: Calculating the stiffness coefficient $(K = F/d)$ based on contact force $(F)$ and deformation $(d)$ provides stiffness detection, although instantaneous deformation calculation poses challenges for underactuated prosthetic hands [Reference Zhang, Xu, Xia and Deng13]. Some studies explore vision-based techniques for deformation detection [Reference Zhu, Cherubini, Dune, Navarro-Alarcon, Alambeigi, Berenson, Ficuciello, Harada, Kober and Xiang3, Reference Cretu, Payeur and Petriu14, Reference Makihara, Domae, Ramirez-Alpizar, Ueshiba and Harada15]. For instance, the Gelsight sensor measures elastomer deformation but may present cost and accessibility limitations [Reference Zhu, Cherubini, Dune, Navarro-Alarcon, Alambeigi, Berenson, Ficuciello, Harada, Kober and Xiang3]. Makihara et al. [Reference Makihara, Domae, Ramirez-Alpizar, Ueshiba and Harada15] employ pixel analysis to generate a stiffness map (’pix2stiffness’) for grasp pose detection to mitigate damage to deformable objects. However, stiffness map generation entails manual intervention, and no force control based on contact dynamics was considered to minimize deformation. Additionally, other frameworks predict object geometry and dynamics for deformable object manipulation, necessitating training and labeling with human intervention during design [Reference Shen, Jiang, Choy, Guibas, Savarese, Anandkumar and Zhu16].

Figure 1.

Slippage avoidance closed-loop control structure presented in literature [Reference Yang and Wu8].

Figure 2.

Slippage avoidance by force control with inner position loop.

Impedance control serves as a pivotal strategy for averting object deformation within robotic manipulation systems [Reference Ji, Zhang, Xu, Tang and Zhao17–Reference Jiang, Tian, Zhan, Xu and Zhang19]. By amalgamating real-time force sensing capabilities with adaptive control algorithms, robots can dynamically modulate compliance and stiffness to align with object characteristics, thereby enhancing precision and reliability in handling delicate objects. In a study by Hua Deng et al. [Reference Deng, Zhong, Li and Nie20], stiffness was regulated utilizing a polyvinylidene fluoride sensor, employing human-defined voltage thresholds for object categorization. Deformation control techniques leveraging Hooke’s law and impedance-based methods necessitate precise knowledge of stiffness and desired model references. Alternative methodologies encompass utilizing kinematics for stiffness detection in underactuated mechanisms [Reference Zhang, Xu, Xia and Deng13] and manipulating object weight through reorientation to regulate deformation [Reference Kaboli, Yao and Cheng21]. Bistable compliant underactuated grippers have demonstrated enhanced grasping capabilities for deformable objects [Reference Mouaze and Birglen22], whereas soft grippers offer adaptability although encountering limitations in variability and complexity [Reference Wang and Ahn23, Reference Milojević, Linß, Ćojbašić and Handroos24]. Addressing these constraints within prosthetic and robotic hands necessitates the integration of active control systems for adaptable grasping, a focal point of investigation in this paper.

2.3. Increasing the generalization capability

The generalization capability of $RL$ policies can be enhanced through various techniques, including domain randomization/domain randomized ( $DR$ ), adversarial rreinforcement learning (ARL), meta-learning, transfer learning, post-training augmentation, and knowledge distillation [Reference Cheng, Zhao, Wang, Block and Hovakimyan6, Reference Salvato, Fenu, Medvet and Pellegrino25, Reference Güitta-López, Boal and lvaro J López-López26]. $DR$ acts as a bridge between simulation and reality, akin to robust control in control theory, by designing controllers resilient to parameter variations and noise [Reference Salvato, Fenu, Medvet and Pellegrino25]. Even when trained sub-optimally in a simulator, $DR$ exhibits effectiveness in real-world scenarios due to its convergence properties [Reference Chen, Hu, Jin, Li and Wang27]. ARL enhances robustness and transferability by training controllers across diverse environment models, leveraging adversarial sub-agents to generate challenging models that minimize cumulative rewards [Reference Salvato, Fenu, Medvet and Pellegrino25]. Robust Adversarial RL frames the problem as a two-player zero-sum game, where a disturbing agent aims to create the worst disturbance, countered by a control agent striving for optimal control input [Reference Pinto, Davidson, Sukthankar and Gupta28, Reference Morimoto and Doya29]. However, both $DR$ and ARL may lead to fixed policies prone to overfitting [Reference Rice, Wong and Kolter30]. Meta-learning, or Meta-RL, focuses on building models capable of adapting and improving performance across new tasks without extensive retraining [Reference Nagabandi, Clavera, Liu, Fearing, Abbeel, Levine and Finn31]. It facilitates rapid adaptation of pre-trained policies to dynamic variations, thereby enhancing policy generalization. Nonetheless, learning optimal policies for all possible scenarios may unnecessarily increase complexity, particularly for simpler tasks [Reference Güitta-López, Boal and lvaro J López-López26]. Transfer learning encompasses techniques like zero-shot learning, few-shot learning, and domain adaptation, yet may result in learning deterministic policies unsuitable for simulation-to-reality transfer [Reference Güitta-López, Boal and lvaro J López-López26]. Meeting strict time constraints is crucial, particularly when implementing controllers on high-frequency physical devices. Deep neural network ( $DNN$ ) policies, especially ensembles, pose challenges in this context. Knowledge distillation, transferring expertise from a large, complex network to a smaller, more efficient one, reduces evaluation time. This technique distills a RL agent’s policy, trained in a large network, into a smaller network operating at an expert level [Reference Rusu, Colmenarejo, Gulcehre, Desjardins, Kirkpatrick, Pascanu, Mnih, Kavukcuoglu and Hadsell32]. Policy distillation has demonstrated efficiency surpassing domain randomization methods [Reference Kadokawa, Zhu, Tsurumine and Matsubara33, Reference Niu, Yuan, Ma, Xu, Liu, Chen, Tong and Lin34]. However, all the above methods tend to learn fixed policies, which may not be suitable for real-time adaptation during the presence of disturbances.

In the post-training augmentation-based strategy aimed at improving generalization capabilities, an augmented robust controller is integrated hierarchically with the $RL$ policy to counteract potential disturbances. In ref. [Reference Cheng, Zhao, Wang, Block and Hovakimyan6], the utilization of $\mathcal{L}_1$ adaptive controllers on the application to a pendubot and quadrotor systems is exemplified, showcasing their efficacy in attenuating the impact of matched uncertainties. Furthermore, Jeong Woo Kim et al. [Reference Kim, Shim and Yang35] introduced a disturbance-based observer to augment an $RL$ policy, addressing mismatches between simulated and real-world environments. Similarly, Anubhav Guha and Anuradha Annaswamy [Reference Guha and Annaswamy36] employed a model reference adaptive control system to estimate and rectify parametric uncertainties, following a comparable approach. The salient advantage of post-training augmentation, vis-à-vis other policy generalization techniques discussed previously, resides in its capacity to dynamically adapt to real-time disturbances. Unlike fixed policies, this approach accommodates variable disturbances not encountered during simulator training. Consequently, the controller can markedly enhance the performance of the learned policy in real-world settings, even amidst the presence of unforeseen and untrained disturbances.

4.1. Bionic reflex grasping policy

Algorithm 1:

Bionic Reflex Control

The entire grasping task is treated as a Model-Free Reinforcement Learning problem, enabling policy learning through direct interaction with the environment and mapping from states to actions. The deformation of the grasping state is represented as a continuous state variable, alongside other observational states such as joint angles, joint velocities, fingertip forces, slip states, wavelet coefficient energy, deformation states, and joint torques, all of which are continuous. Moreover, the action space consists of continuous joint torques. Hence, the Actor-Critic $RL$ algorithm is selected for its suitability in handling continuous state and action spaces. The pseudo-code for the $RL$ -based bionic reflex controller is outlined in Algorithm1. This algorithm aims to determine optimal joint torques ( $\tau$ ) necessary for lifting a grasped object without encountering slippage or deformation within a PyBullet [Reference Coumans and Bai38] grasping environment (as detailed in the subsequent section). The algorithm operates through a sequence of episodes, with each episode comprising steps aimed at executing successful grasping and lifting actions. Initially, the algorithm initializes state observations ( $s$ ) from the environment. During each step of the episode, actions are generated utilizing the Soft Actor-Critic algorithm [Reference Haarnoja, Zhou, Hartikainen, Tucker, Ha, Tan, Kumar, Zhu, Gupta and Abeel39] to manipulate the robot’s joints and grasp the object. Subsequently, the algorithm lifts the object while continually monitoring for signs of slippage or deformation. If the object drops or slips, the episode concludes, and the simulation is reset. In cases where slippage or deformation occurs during lifting, the joint torques ( $\tau$ ) are adjusted accordingly to either increase or decrease the applied force. Initially, the algorithm detects slips, correcting joint torques using $\delta \tau$ . Following this, if deformation occurs, the joint torques are further adjusted by a value $\lambda$ . If the grasp is successful without slippage or deformation, the episode concludes, and the simulation is reset. This iterative process continues until termination of the episodes, ensuring comprehensive exploration and refinement of the grasping strategy across multiple episodes.

4.2. RL training for the nominal policy

The policy governing the grasping and lifting actions of the robotic hand manipulating an object is trained within the PyBullet Simulator environment. This training utilizes a standard soft-actor critic rreinforcement learning lgorithm from the Stable-Baselines3 framework [Reference Raffin, Hill, Gleave, Kanervisto, Ernestus and Dormann40]. The objective is to derive optimal joint torques that enable the generation of requisite fingertip forces, thereby preventing object slippage and deformation during both grasping and lifting maneuvers.

4.2.1. MDP for the RL training

1. 1. States: Joint Angles, Joint Velocities, Fingertip Forces, Slip States, Wavelet Coefficients, Joint Torques, Deformation States

2. 2. Actions: Joint Torques

3. 3. Rewards:

   (1) \begin{equation} \sum \limits _{i = 1}^5{\left ({\frac{1}{{{\ln} ({x_i} + 1.1051)}}} \right )} + \sum \limits _{i = 1}^5{\left ({{\delta _i}.10} \right )} - \sum \limits _{i = 1}^5{\left ({{\psi _i}.10} \right )} - C \times \Delta - d^2 \end{equation}
   The objective is to meticulously guide the hand to grasp the object, ensuring slip-free lifting while mitigating any potential damage caused by deformation. The reward function comprises several terms tailored to achieve this goal. The first term employs an inverse logarithmic relation to compute the reward for each step. Specifically, $x_i$ denotes the distance between the fingertip and the object. As the finger approaches the object, $x_i$ diminishes towards zero, maximizing the reward. To further distinguish between close and actual contact, a discrete term $\delta$ is introduced, incrementing a + 10 reward upon contact of each finger ( $\delta$ equals 1 if contact is detected, and 0 otherwise). Subsequently, the reward function penalizes slippage during lifting. For each finger, a penalty of 10 units is deducted from the total reward whenever slipping is detected, indicated by the boolean variable, $\psi$ (with $\psi$ equaling 1 upon slip detection and 0 otherwise). However, a limitation of the previous terms lies in their potential to promote excessively tight grasps that may damage the object. To mitigate this, a penalty for deformation is incorporated. This penalty is computed based on the volume gradient ( $\Delta$ ), representing the change in volume, and is scaled accordingly ( $C = 50/(initial \hspace{0.5em} volume)$ ). This value is subtracted from the reward function, penalizing the agent for excessive deformation resulting from the grasp. Finally, the wavelet coefficient energy term serves as a metric for quantifying the degree of slippage [Reference Deng, Zhang and Duan41], providing an additional measure to regulate slip prevention.

4.2.1.1. Slip detection

Slip detection is facilitated through Haar wavelet decomposition of the force sensor signal, chosen for its capability to identify transient changes resembling slip signals owing to its non-differentiable, discontinuous, and asymmetric characteristics [Reference Romeo and Zollo4, Reference Yang, Hu, Cao and Sun42, Reference Romeo, Rongala, Mazzoni, Camboni, Carrozza, Guglielmelli, Zollo and Oddo43]. To get over the drawback of thresholding, we draw inspiration from the work in [Reference Yang, Hu, Cao and Sun42], which utilized the analysis of the property of the trend of pairwise details in a discrete wavelet transform (DWT)-based methodology to determine the occurrence of slip. Specifically, two subsequent DWT components had the same absolute value but a different sign due to the characteristics of the DWT that is used. Because the sign of paired components shifts from negative to positive in the load phase, as opposed to the slip phase, hence it is possible to discriminate between the two stages [Reference Yang, Hu, Cao and Sun42]. After thorough experimentation, we opted for the 5th-level decomposition, considering factors such as signal representation and slip detection requirements. This level strikes a balance between capturing fine-grained signal details and mitigating excessive noise or artifacts. Figure 4 illustrates the Haar wavelet decomposition-based slippage detection methodology employed in this study. Our methodology is different from the one mentioned in [Reference Yang, Hu, Cao and Sun42], because following signal decomposition, we analyze the gradient trend of the inverse Haar using the approximation and detailed coefficients of the 5th-level Haar transformed signal. A negative gradient signifies slip occurrence, while a positive gradient indicates its absence. The RL algorithm uses this trend as a slip state observation after detecting it with the slip detector logic. Subsequently, the RL agent’s policy is to acquire optimum torques for joint application based on the maximization of the accumulated reward function.

Figure 4.

Force sensor signal while grasping and lifting an object. Fifth-level Haar decomposition of raw force sensor signal is used to detect slip. The positive gradient is a reflection of the load being applied. The opposing variation trend is a representation of slip.

4.2.1.2. Deformation detection

The process begins with the creation of the deformable object in SolidWorks, resulting in a .stl (Standard Triangle Language/Standard Tessellation Language) file. Subsequently, tetrahedral meshing is executed using fTetWild, yielding a .msh file [Reference Hu, Schneider, Wang, Zorin and Panozzo44]. The deformable object is then generated in Gmsh as a .vtk file and simulated utilizing the built-in Finite Element Method (FEM) in PyBullet. FEM represents a robust tool for modeling deformable objects, leveraging the discretization of objects into small elements to derive deformation through the solution of partial differential equations. This method enables accurate representation of deformable object dynamics, particularly with fine tessellation, albeit at a computational cost. FEM efficiently approximates the genuine physical behavior of deformable objects [Reference Arriola-Rios, Guler, Ficuciello, Kragic, Siciliano and Wyatt45]. The resulting deformable object, as illustrated in Figure 5, is characterized by parameters detailed in Table I, which are utilized for our physics-based simulation.

4.3. Deformation calculation

In PyBullet, surface mesh data representation entails vertex and triangle lists, as depicted in Figure 5. The function “getMeshData” facilitates the retrieval of mesh information, specifically the vertex indices of triangular meshes constituting the 3D object. Additionally, the reference frame dynamically transitions from the ground to the grasped object (shifted origin) during grasping and lifting operations. This adjustment ensures the accommodation of translations or rotations experienced by the object during manipulation. To estimate volume changes for deformable objects devoid of explicit mathematical formulations, a general point (shifted origin) is defined to construct a tetrahedron, as depicted in Figure 5. Subsequently, the signed volume is computed using the following formula, as given in ref. [Reference Zhang and Chen46]:

(2) \begin{equation} Signed \hspace{0.5em} Volume = \textbf{AO}. (\textbf{AB} \times \textbf{AC}) / 6 \end{equation}

where points A, B, and C are concurrently selected from the mesh, and O is a reference mesh point taken arbitrarily. The term “signed volume” pertains to the orientation or direction employed for volume computation. Specifically, the surface normal of triangle ABC determines the sign and weight of each tetrahedron formed by the vertices, and the aggregation of these solids yields the object’s total volume. Subsequently, this volume metric serves as the basis for calculating deformation, denoting the change in volume observed at each time step. Our methodology affords a dynamic and comprehensive evaluation of deformation, facilitating real-time monitoring essential for reward calculation.

Table I.

Simulation parameters for deformable object.

Figure 5.

Triangular mesh description of the deformable cylinder. Point O represents reference origin. The tetrahedral mesh, ABCO is shown in the figure as well.

4.4. Randomization of the physics parameters

As previously highlighted, DR emerges as a viable strategy for effecting domain adaptation through parameter randomization. In our present experiment, we implement this technique by randomizing the physics parameters within PyBullet. Each episode entails sampling random trajectories from a predetermined range of randomized parameters, thereby diversifying the training data for our RL model. These parameters are subject to randomization around a predefined set of nominal values, encompassing a specified range of limits. This process facilitates robust training by exposing the RL model to a variety of simulated environments characterized by diverse physical properties.

4.5. Adaptive sliding mode control based augmentation for RL policy robustification

4.5.1. $AFNITSM$ controller for the robotic gripper

The dynamics of the anthropomorphic hand is given as:

(3) \begin{equation} B(\theta )\ddot \theta + C(\theta, \dot \theta )\dot \theta + g(\theta ) = \tau +{\tau _{ext}} \end{equation}

where $\theta, \dot \theta, \ddot \theta \in \mathbb{R}{^n}$ stand for the vectors of position, velocity, and acceleration of the joints, $B(\theta )=B_0(\theta )+\Delta B(\theta )$ represents the inertia matrix, $C(\theta, \dot \theta )=C_0(\theta, \dot \theta )+\Delta C(\theta, \dot \theta )$ represents the centripetal Coriolis matrix, $g(\theta )=g_0(\theta )+\Delta g(\theta )$ represents the gravitational matrix, $\tau$ represents the joint torques, $\tau _{ext}$ represents the external disturbance on the joint torque input. $B_0(\theta ), C_0(\theta, \dot \theta )$ , and $g_0(\theta )$ are the nominal terms of the dynamic model, and $\Delta B_0(\theta ), \Delta C_0(\theta, \dot \theta )$ , and $\Delta g_0(\theta )$ are the uncertainties of the dynamic model.

The dynamical equation in Eq. (3) can be written as:

(4) \begin{align} B_0(\theta )\ddot \theta + C_0(\theta, \dot \theta )\dot \theta + g_0(\theta ) = \tau +{\tau _{d}} \end{align}

where $\tau _d = \tau _{ext} - \Delta B(\theta )\ddot \theta - \Delta C(\theta, \dot \theta )\dot \theta - \Delta g(\theta )$ represents the lumped disturbances of the robotic gripper.

We define the desired position vector by $\theta _d$ , and the tracking error is then defined as $ e_1=\theta -\theta _d$ .

To design a robust chattering-free controller for a robotic manipulator with dynamics model (3), which is capable of tracking the desired trajectory accurately, some assumptions are presented as follows:

Assumption 1. $\tau _{d}$ is bounded, which satisfies the following function:

(5) \begin{equation} \left \|{{\tau _d}} \right \| \lt{a_0} +{a_1}\left \| q \right \| +{a_2}{\left \|{\dot q} \right \|^2} \end{equation}

where $a_0$ , $a_1$ , and $a_2$ are unknown positive constants. $\left \| . \right \|$ represents the 2-norm.

Assumption 2. $\left \|{{{\dot \tau }_d}} \right \|$ is bounded,

The derivatives of the tracking error are:

(6) \begin{equation} \begin{aligned}[t] \dot e &= \dot \theta - \dot \theta _d \\ \ddot e &= \ddot \theta - \ddot \theta _d \\ \dddot e &= \dddot \theta - \dddot \theta _d \end{aligned} \end{equation}

Using Eq. 4 , we have:

(7) \begin{equation} \ddot e ={B_0}^{ - 1}(\theta )[\tau - f(\theta, \dot \theta ) +{\tau _d}] - \ddot \theta \end{equation}

where $f(\theta, \dot \theta )=C(\theta, \dot \theta )\dot \theta + G(\theta )$ . Differentiation of the above equation gives:

(8) \begin{equation} \begin{aligned}[t] \dddot e &={\dot B_0}^{ - 1}(\theta )[\tau - f(\theta, \dot \theta ) +{\tau _d}] +{B_0}^{ - 1}(\theta )[\dot \tau - f(\theta, \dot \theta ) +{\dot \tau _d}(\theta, \dot \theta, \ddot \theta )] -{\dddot \theta _d} \\ &={\dot B_0}^{ - 1}(\theta )[\tau - f(\theta, \dot \theta )] +{B_0}^{ - 1}(\theta )[\dot \tau - f(\theta, \dot \theta )] -{\dddot \theta _d} + F(\theta, \dot \theta, \ddot \theta ) \end{aligned} \end{equation}

where $ F(\theta, \dot \theta, \ddot \theta ) = \dot B_0^{ - 1}(\theta ){\tau _d} + B_0^{ - 1}{\dot \tau _d}(\theta, \dot \theta, \ddot \theta )$

Using Assumptions1–2, we obtain $\left \| F(\theta, \dot \theta, \ddot \theta \right \| \lt b_0 + b_1 \left \| \theta \right \| + b_2 \left \| \dot \theta \right \|^2$ , where $b_0, b_1$ and $b_2$ are unknown positive constants.

4.5.2. Second-order adaptive integral terminal sliding mode controller design:

The integral sliding mode control ( $ISMC$ ) surface can be given as [Reference Utkin and Shi48]:

(9) \begin{equation} s = \dot e +{c_1}e +{c_2}\int e d\tau - \dot e(0) -{c_1}e(0) \end{equation}

where $c_1 = diag(c_{11},c_{12},\ldots, c_{1n})$ and $c_2=diag(c_{21},c_{22},\ldots, c_{2n})$ are positive-definite matrices. The first and second-time derivatives give:

(10) \begin{equation} \dot s = \ddot e + c_1 \dot e + c_2 e, \end{equation}

(11) \begin{equation} \ddot s = \dddot e + c_1 \ddot e + c_2 \dot e, \end{equation}

To expedite finite-time convergence and mitigate singularity issues, we employ a fast non-singular integral terminal sliding mode ( $FNITSM$ ). This approach ensures rapid convergence of the system state $s$ to equilibrium within a finite-time span, both in regions far from and close to equilibrium, while circumventing singularity problems. The design of the second-order fast nonsingular integral terminal sliding mode controller ( $SOFNITSMC$ ) surface is formulated as follows [Reference Hao, Hu and Liu37, Reference Li, Ma, Zheng and Geng49, Reference Alattas, Mobayen, Sami, Jihad, Afef, Wudhichai and Mai50]:

(12) \begin{equation} \sigma = \dot s + \int \limits _0^t{[{\beta _1}{\lambda _1}({\gamma _1},{\rho _1},s,{\varepsilon _1}) +{\beta _2}{\lambda _2}({\gamma _2},{\rho _2},\dot s,{\varepsilon _2})]d\tau } \end{equation}

where,

(13) \begin{equation}{\lambda _1}({\gamma _1},{\rho _1},s,{\varepsilon _1}) = \left \{\begin{array}{l@{\quad}l}{\mathop{\textrm{sgn}}}{s^{{\gamma _1}}}, & \left | s \right | \le{\varepsilon _1}\\[4pt]{\varepsilon _1}^{{\gamma _1} -{\rho _1}}{\mathop{\textrm{sgn}}}{s^{{\gamma _2}}}, & \left | s \right | \gt{\varepsilon _1} \end{array}\right. \end{equation}

(14) \begin{equation}{\lambda _2}({\gamma _2},{\rho _2},\dot s,{\varepsilon _2}) = \left \{\begin{array}{l@{\quad}l}{\mathop{\textrm{sgn}}}{\dot s^{{\gamma _2}}}, & \left | \dot s \right | \le{\varepsilon _2}\\[5pt]{\varepsilon _2}^{{\gamma _2} -{\rho _2}}{\mathop{\textrm{sgn}}}{\dot s^{{\gamma _2}}}, & \left | \dot s \right | \gt{\varepsilon _2} \end{array} \right . \end{equation}

$\beta _1=diag(\beta _{11},\beta _{12},\ldots, \beta _{1n}$ and $\beta _2=diag(\beta _{21},\beta _{22},\ldots, \beta _{2n}$ are positive-definite matrices, and $\gamma _i, \varepsilon _i,$ and $\rho _i$ are constants that satisfy $0\lt \gamma _2\lt 1, \gamma _1=\gamma _2/(2-\gamma _2), \rho _i \ge 1$ and $\varepsilon _i \gt 0 \quad (i=1,2)$ . The parameters $\gamma _1$ and $\gamma _2$ are critical in shaping the nonlinearity, ensuring finite-time convergence, and improving the smoothness and robustness of the sliding mode control [Reference Hao, Hu and Liu37].

The time derivative of (12) is:

(15) \begin{equation} \dot \sigma = \ddot s +{\beta _1}{\lambda _1}({\gamma _1},{\rho _1},s,{\varepsilon _1}) +{\beta _2}{\lambda _2}({\gamma _2},{\rho _2},\dot s,{\varepsilon _2}) \end{equation}

The control law of a sliding mode control (SMC) consists of an equivalent control law $\tau _{eq}$ and a switching control law $\tau _{sw}$ . The SOITSSMC is chosen as:

(16) \begin{equation} \tau ={\tau _{eq}} +{\tau _{sw}} = \int \limits _0^t{({{\dot \tau }_{eq}} +{{\dot \tau }_{sw}})d\tau } \end{equation}

The $\tau$ is designed to guarantee the $\sigma$ converges to zero. We can get $\dot \tau _{eq}$ by making $F(x,\dot x,\ddot x) = 0$ , and then we design $\dot \tau _{sw}$ i.e., the discontinuous control action, to deal with the disturbances.

When $F(x,\dot x,\ddot x) = 0$ , and let $\sigma =0$ , this enables us to obtain $\dot \tau _{eq}$ and $\dot \tau _{sw}$ as follows:

(17) \begin{equation} \begin{split}{\dot \tau _{eq}} & = -{B_0}(x){\dot B_0}^{ - 1}[\tau - f(x,\dot x)] +{B_0}(x)({\ddot x_d} -{c_1}\ddot e -{c_2}\dot e) -{B_0}[{\beta _1}{\lambda _1}({\gamma _1},{\rho _1},s,{\varepsilon _1}) \\ & +{\beta _2}{\lambda _2}({\gamma _2},{\rho _2},s,{\varepsilon _2})] + \dot f(x,\dot x), \end{split} \end{equation}

(18) \begin{equation} {\dot \tau _{sw}} = -{B_0}(x)[k\sigma + (b_0 + b_1 \left \| x \right \| + b_2 \left \| \dot x \right \|^2)] \end{equation}

where $k=diag(k_1,k_2,\ldots, k_n)$

4.5.2.1. Adaptive law design:

In practice, we cannot obtain the values of $b_0, b_1,$ and $b_2$ in Eq. 18. Therefore, we will use the adaptive parameter tuning scheme to estimate them:

(19) \begin{equation}{\dot \tau _{asw}} = -{B_0}(x)[k\sigma + (\hat b_0 + \hat b_1 \left \| x \right \| + \hat b_2 \left \| \dot x \right \|^2)] \end{equation}

where $\hat b_0, \hat b_1$ and $\hat b_2$ are the respective estimates. The adaptive laws for $\hat b_i (i=1,2,3)$ are as follows:

(20) \begin{equation} \begin{aligned}[t] \dot{\hat{b_0}} &= \left \| \sigma \right \| \\ \dot{\hat{b_1}} &= \left \| \sigma \right \| \left \| x \right \| \\ \dot{\hat{b_2}} &= \left \| \sigma \right \| \left \| \dot x \right \|^2 \\ \end{aligned} \end{equation}

We define the adaptation error as $\tilde b_i = b_i - \hat b_i (i=0,1,2)$ . The second-order adaptive fast nonsingular integral terminal sliding mode controller (SOAFITSMC) is then designed as follows:

(21) \begin{equation} \tau ={\tau _{eq}} +{\tau _{asw}} = \int \limits _0^t{({{\dot \tau }_{eq}} +{{\dot \tau }_{asw}})d\tau } \end{equation}

where,

(22) \begin{equation} \begin{split}{\dot \tau _{eq}} & = -{B_0}(x){\dot B_0}^{ - 1}[\boldsymbol{\tau } - f(x,\dot x)] +{B_0}(x)({\ddot x_d} -{c_1}\ddot e -{c_2}\dot e) -{B_0}[{\beta _1}{\lambda _1}({\gamma _1},{\rho _1},s,{\varepsilon _1}) \\ & +{\beta _2}{\lambda _2}({\gamma _2},{\rho _2},s,{\varepsilon _2})] + \dot f(x,\dot x), \end{split} \end{equation}

(23) \begin{equation} {\dot \tau _{asw}} = -{B_0}(x)[k\sigma + (\hat b_0 + \hat b_1 \left \| x \right \| + \hat b_2 \left \| \dot x \right \|^2)] \end{equation}

4.5.3. Stability analysis

In this section, we present the stability analysis of the $SOAFNITSMC$ . The Lyapunov function candidate is considered as follows:

(24) \begin{equation} \begin{aligned} V = \frac{1}{2}\left({\sigma ^T}\sigma +{\mu _0}\tilde b_0^2 +{\mu _1}\tilde b_1^2 +{\mu _2}\tilde b_2^2\right) \end{aligned} \end{equation}

The time derivative of the above equation gives:

(25) \begin{equation} \begin{aligned}[t] \dot{V} &={\sigma ^T}\dot \sigma + \mu _0\tilde b_0 \dot{\hat{b_0}} + \mu _1\tilde b_1 \dot{\hat{b_1}} + \mu _2\tilde b_2 \dot{\hat{b_2}} \\ &={\sigma ^T}[\ddot s +{\beta _1}{\lambda _1}({\gamma _1},{\rho _1},s,{\varepsilon _1}) +{\beta _2}{\lambda _2}({\gamma _2},{\rho _2},\dot s,{\varepsilon _2})] + \mu _0\tilde b_0 \dot{\hat{b_0}} + \mu _1\tilde b_1 \dot{\hat{b_1}} + \mu _2\tilde b_2 \dot{\hat{b_2}} \end{aligned} \end{equation}

Substituting Eqs. 8 and 10 into Eq. 25

(26) \begin{align} \dot{V} & ={\sigma ^T}[{\dot B_0}^{ - 1}(x)[\tau - f(x,\dot x)] +{B_0}^{ - 1}(x)[\dot \tau - f(x,\dot x)] -{\dddot x_d} + F(x,\dot x,\ddot x) + c_1\ddot e\nonumber\\& \quad + c_2\dot e{\beta _1}{\lambda _1}({\gamma _1},{\rho _1},s,{\varepsilon _1}) +{\beta _2}{\lambda _2}({\gamma _2},{\rho _2},\dot s,{\varepsilon _2})] + \mu _0\tilde b_0 \dot{\hat{b_0}} + \mu _1\tilde b_1 \dot{\hat{b_1}} + \mu _2\tilde b_2 \dot{\hat{b_2}} \end{align}

Using Eqs. (22) and (23), we have:

\begin{align*} \dot{V} &={\sigma ^T}[{-}k\sigma - (\hat b_0 + \hat b_1 \| x \| + \hat b_2 \| \dot x \|)sign(\sigma ) + F(x,\dot x, \ddot x)] + \mu _0\tilde b_0 \dot{\hat{b_0}} + \mu _1\tilde b_1 \dot{\hat{b_1}} + \mu _2\tilde b_2 \dot{\hat{b_2}} \\[3pt] &\le \| F(x,\dot x, \ddot x) \| \| \sigma \| - \| \hat b_0 + \hat b_1 \| x \| + \hat b_2 \| \dot x \|^2 \| + \| b_0 + b_1 \| x \| + b_2 \| \dot x \|^2 \| \| \sigma \| \\[3pt] & - \| b_0 + b_1 \| x \| + b_2 \| \dot x \|^2 \| \| \sigma \| + \mu _0\tilde b_0 \dot{\hat{b_0}} + \mu _1\tilde b_1 \dot{\hat{b_1}} + \mu _2\tilde b_2 \dot{\hat{b_2}} \\[3pt] &\le - [ \|{({b_0} +{b_1} \| x \| +{b_2}{{ \|{\dot x} \|}^2}}) \| - \|{F(x,\dot x,\ddot x} \| \| \sigma \|] - \|{{{\tilde b}_0}} \|( \| \sigma \| -{\mu _0}\sigma ) - \|{{{\tilde b}_1}} \|( \| \sigma \| \| x \| \\[3pt] & - {\mu _1} \| \sigma \| \| x \|) - \|{{{\tilde b}_2}} \|( \| \sigma \|{ \|{\dot x} \|^2} - {\mu _2} \| \sigma \|{ \|{\dot x} \|^2}) \\[3pt] &\le - \varrho \sqrt 2 \frac{{ \| \sigma \|}}{{\sqrt 2 }} -{\xi _0}\sqrt{2{\mu _0}} \frac{{\|{{{\tilde b}_0}} \|}}{{\sqrt{2{\mu _0}} }} -{\xi _1}\sqrt{2{\mu _1}} \frac{{ \|{{{\tilde b}_1}} \|}}{{\sqrt{2{\mu _1}} }} -{\xi _2}\sqrt{2{\mu _2}} \frac{{\|{{{\tilde b}_2}}\|}}{{\sqrt{2{\mu _2}} }} \end{align*}

where

(27) \begin{equation} \begin{aligned} \varrho &= \left \|{({b_0} +{b_1}\left \| x \right \| +{b_2}{{\left \|{\dot x} \right \|}^2}} \right \| - \left \|{F(x,\dot x,\ddot x} \right \|, \\[2pt] \xi _0 &= \left \| \sigma \right \| -{\mu _0}\left \| \sigma \right \| = (1-\mu _0)\left \| \sigma \right \|, \\[2pt] \xi _1 &= (\left \| \sigma \right \|\left \| x \right \| -{\mu _1}\left \| \sigma \right \|\left \| x \right \|) = (1-\mu _1)\left \| \sigma \right \|\left \| x \right \|, \\[2pt] \xi _2 &= (\left \| \sigma \right \|{\left \|{\dot x} \right \|^2} -{\mu _2}\left \| \sigma \right \|{\left \|{\dot x} \right \|^2}) = (1-\mu _2)\left \| \sigma \right \|{\left \|{\dot x} \right \|^2} \end{aligned} \end{equation}

We then obtain:

(28) \begin{align} \dot{V} & \le - \min \left \{ \sqrt{2} \varrho, \sqrt{2 \mu _0^{-1}} \xi _0, \sqrt{2 \mu _1^{-1}} \xi _1, \sqrt{2 \mu _2^{-1}} \xi _2 \right \} \times \left ( \frac{\left \| \sigma \right \|}{\sqrt{2}} + \frac{\sqrt{\mu _0}}{\sqrt{2}} \| \tilde{b}_0 \| + \frac{\sqrt{\mu _1}}{\sqrt{2}} \| \tilde{b}_1 \| + \frac{\sqrt{\mu _2}}{\sqrt{2}}\| \tilde{b}_2\| \right ) \nonumber\\[5pt] & \le -\alpha V^{1/2} \end{align}

where

\begin{equation*} \alpha = \min \left\{ \sqrt 2 \varrho, \sqrt {2{\mu _0}^{ - 1}} {\xi _0},\sqrt {2{\mu _1}^{ - 1}} {\xi _1},\sqrt {2{\mu _2}^{ - 1}} {\xi _2}\right\} \end{equation*}

and $\alpha \gt 0$ . The above inequality holds if $\mu _0 \lt 1$ , $\mu _1 \lt 1$ , and $\mu _2 \lt 1$ . The stability of the Lyapunov function in the context of SMC often relies on ensuring that certain parameters or terms within the Lyapunov candidate function remain bounded or decrease over time. For the Lyapunov function $V$ to be positive definite, all terms must be positive or zero. The given function $V$ is quadratic in nature and thus positive definite as long as $\sigma$ , ${\tilde b}_0$ , ${\tilde b}_1$ , and ${\tilde b}_2$ are finite and $\mu _0$ , $\mu _1$ , $\mu _2$ are positive. The parameters $\mu _0$ , $\mu _1$ , $\mu _2$ scale the estimation errors or uncertainties. If $\mu _i \geq 1$ , the corresponding term ${\mu _i}\tilde b_i^2$ could grow large, potentially causing $V$ to increase or not decrease sufficiently, leading to instability. By keeping $\mu _i \lt 1$ , the influence of the estimation errors on the Lyapunov function is bounded, ensuring that the function remains decreasing or non-increasing.

If $t_0 = 0$ , then $\sigma$ in Eq. 12 can converge to zero in a finite time ${t_1} ={V^{1 - 1/2}}(0)/\alpha (1 - 1/2) = 2{V^{1/2}}(0)/\alpha$ . Thus, the sliding variable $s$ will converge to zero in a finite time. The tracking error $e$ can asymptotically converge to zero. This completes the proof of stability.

The control architecture of the post-training augmented robust $RL$ policy is depicted in Figure 6. This architecture closely resembles an indirect force control setup with an inner position loop [Reference Siciliano, Sciavicco, Villani and Oriolo10, Reference Carbone, Iannone and Ceccarelli11], wherein the reference force is derived from the trained $RL$ policy. Following the application of the adaptive control input to the anthropomorphic hand, force error signals are generated by comparing the experimental force signals $F_e$ with the desired force signals $F_d$ . The desired force $F_d$ is linked to the generated pre-trained $RL$ policy (in terms of torques) through the Jacobian transpose. Subsequently, the error value $e(t)$ serves as the input for a proportional-derivative force control algorithm, comprising a proportional and a derivative regulator. The output $x_d$ of this algorithm is given by:

(29) \begin{equation} x_d = K_{p}e(t)+K_d\frac{de(t)}{dt} \end{equation}

where $K_p$ and $K_d$ are the proportional and derivative gains, respectively. The values of $K_p$ and $K_d$ can be identified by manual tuning through experimental tests to adjust the reaction time and limit the overshooting of the measured $F_e$ with respect to $F_d$ .

The output $x_d$ is added to the measured value of the finger position $\theta _e$ to give the position error signals, $e_p$ . This error signal serves as the input for the sliding surface as given in Eq. 6.

The nominal parameters for the simulation and the control parameters taken are given in Table II (details of the dynamical equation given in Appendix) and Table III.

Table II.

Physical parameters of the anthropomorphic hand used for simulation.

Figure 6.

Post-training augmented robust reinforcement learning control diagram for perturbed environment. This is the detailed diagram of the adaptive fast non-singular integral terminal sliding mode shown in Figure 3.

Table III.

Parameters of the proposed controller and their values.