2.1. E-skin design principle

To fully exploit the potential of e-skin in extracting motion information, a stretchable sensor that fits completely on the skin surface is designed. This sensor consists of 11-channel stretchable resistors attached to the major joints of the fingers and the skin surface, as shown in Fig. 1(a). The resistors 1–10 are distributed at the proximal and metacarpal joints of each finger, respectively, and resistor 11 is distributed at the purlicue. Carbon nanotubes are used as the fabrication material for stretchable resistors.

Figure 1.

(a) Schematic diagram of e-skin sensor. (b) Wearing diagram of e-skin sensor. (c) Overall manufacturing process of stretchable e-skin sensor. (d, e) Physical drawing of DB100 microelectronic printer.

2.2. E-skin fabrication scheme

For the e-skin sensor fabrication, a microelectronic printer DB100 from Shanghai Mifang Electronic Technology Co., Ltd is used to print the structure.

The design can be divided into the following steps:

• The Polydimethylsiloxane (PDMS) is coated on the Polyethylene (PET) membrane and dried in the drying box to form the PDMS substrate.

• The fabricated PDMS substrate is neatly placed on the operation platform of DB100 printing electronic multifunction printer, as shown in Fig. 1(d), (e).

• The printing parameters are properly set using the DB100 supporting software to print and draw silver wires on the PDMS substrate.

• Finally, carbon nanotubes with a concentration of 5% on the stretchable silica gel are uniformly spread on the PDMS substrate by template printing, and 11-channel stretchable e-skin is formed.

The overall fabrication process is shown in Fig. 1(c), and the final fabricated e-skin sensor is shown in Fig. 1(b).

2.3. Performance test of e-skin

After the manufacturing of e-skin, it is necessary to test the stretching and electrical properties of e-skin sensor. Here, we choose to test the stretching resistance of a single channel (the characteristics of other channels are similar). The stretching property of the sensor is tested by Instron 5565 (Instron 5565, Instron (Shanghai) LTD., USA), and the stretching ratio is set to 150%. In addition, the electrical performance of the sensor is detected and output by Keithley (DMM7510, Tektronix). The final cyclic stretching result is shown in Fig. 2. The sensor unit can maintain a relatively stable state under a large number of repeated stretching conditions.

Figure 2.

Graph of the electrical properties of the e-skin under cyclic stretching conditions.

3.1. Chip solution selection

The FPC mainly involves the microcontroller unit (MCU), the data wireless transmission module, the power management module, the analog-to-digital conversion (ADC) module, etc., where MCU is the core part. Considering low cost and power consumption, a STM32G431CBT6 chip with LQFP-48 package as the MCU is chosen. This chip has 170 MHz mainstream ARM Cortex-m4 MCU with DSP and FPU, and 128 KB flash memory. At the same time, the built-in ADC modules of STM32 are used to measure the resistance. In addition, it also has abundant peripheral resources such as UART, SPI, Timer, etc., which can fully meet the system requirements.

In order to ensure the reliability and convenience of data transmission, W600-A800, a low-power WiFi chip, is chosen as the data transmission module of our circuit to realize the communication between e-skin sensor and data receiving terminal.

After the selection of MCU and WiFi module, an appropriate power management chip is needed to ensure the normal work of STM32 and WiFi. Here, a widely used rechargeable 3.7 V lithium battery is chosen as the power source. Since the STM32 and WiFi chips need a stable 3.3 V voltage to work properly, a 3.7 V-to-3.3 V chip RT9013-33 is selected to generate 3.3 V voltage. For the convenience of calculation, the TL431 chip is used to output a stable 3 V voltage as the reference voltage to the ADC modules.

3.2. Circuit schematic design

The developed 11-channel resistors of e-skin sensor are measured based on two built-in ADCs in STM32. The overall circuit connection and measurement principle are shown in Fig. 3(a). Here, the resistors are connected in a voltage-dividing manner, and a reference resistor is connected in series with the resistor to be measured for each channel. Since the built-in ADC is 12-bit, its maximum reading value is 4095, corresponding to a 3 V voltage. Thus, the resistance value of each channel can be obtained:

(1) \begin{equation}{R_c} ={{{R_f}} \Big/{\left ({\frac{{4096}}{{{V_{\text{adc}}}}} - 1} \right )}} \end{equation}

where $ R_c$ represents the resistance of each stretchable resistor, $ R_f$ represents the reference resistance welded on the FPC, and $V_{\textrm{adc}}$ represents the reading value of the built-in ADC.

Figure 3.

(a) Circuit connection and signal transmission diagram. (b) Overview of the designed FPC.

The circuit schematic diagram of the e-skin monitoring unit is then designed based on above analysis. In addition to the basic chip interface, the SWD interface for program debugging and the sensor interface for connecting the e-skin are reserved. Finally, the physical diagram of the circuit after the welding test is shown in Fig. 3(b).

3.3. Acquisition circuit programming

In order to realize the real-time measurement of e-skin sensor by the designed FPC, the STM32CubeMx software is used to develop specific programs. The configuration of STM32CubeMx software mainly includes the following steps: First, the MCU corresponding to the STM32 is selected and its clock is configured. The clock frequency of all peripheral devices uses the default 170 MHz. The initialization parameters of peripheral modules, such as UART, ADC, Timer, are then configured. Among them, the baud rate of the UART is configured as 115200 for connecting to the WiFi chip, and the two ADCs are configured as multi-channel scan query mode.

To achieve efficient transmission and obtain enough observation data, the timing period of the timer is configured as 10ms; that is, the ADC collects a group of data every 10 ms (the acquisition frequency is 100 Hz). In addition, due to the load of network, 10 groups of data are packaged and sent to receiving terminal every 100 ms. This transmission can avoid packet loss and retransmission caused by frequent data transmission, which makes the transmission process reliable and stable. Finally, the initialization project code based on the keil integrated development environment is generated. The overall processing flow is shown in Fig. 4.

Figure 4.

Program flow chart of STM32.

4.1. Qt-based GUI design

This GUI consists of three parts: the numerical display, the curve scrolling, and the function buttons, as shown in Fig. 5. The numerical display in the top green box displays the received resistance value in real time, the curve scrolling in the middle red box scrolls and displays the data, and the button under the right bottom includes the control buttons for saving data and enabling WiFi communication.

Figure 5.

GUI with data display and save function developed based on Qt framework.

The system requires some configuration, such as IP address, port number, and other network information. Considering that the period of sending data of the FPC is 100ms, and there are 10 groups of measurement values in each period, we take the first group of measurement for display and save all 10 groups of data to the local. In this way, the real-time performance of data transmission is guaranteed.

4.2. Grasp test of different objects

Different gestures of the subjects will lead to the diversity of signals, so that the performance of e-skin system can be verified according to the signal discrimination of different gestures. For verification, we conduct a test in which the subject grasps objects of various sizes and shapes such as hamburger, carton, apple, banana, and water cup.

The obtained grasping test results are shown in Fig. 6. Specifically, for objects of different shapes, such as apple, banana, and water cup, the obtained e-skin response data have obvious differences across multiple channels. In addition, for objects with similar shapes such as apple and hamburger, the obtained grasping response curves are also distinct due to their different sizes. This not only verifies the good performance of e-skin sensor itself but also proves the stability and reliability of the signal acquisition and transmission process.

Figure 6.

Resistance response ratio of each channel of the e-skin when grasping different objects.

5.1. Signals preprocessing

A sliding window method is used to segment the original signals for constructing training samples and testing samples suitable for LSTM network. For a continuous period of e-skin data, it is intercepted from $T_1$ ms to $T_2$ ms. At this time, the amount of data obtained on each channel of e-skin is $N$ , and the following relationship is satisfied:

(2) \begin{equation} N = \left ({{T_2} -{T_1}} \right ) \cdot F/1000 \end{equation}

where $f$ represents the sampling frequency of e-skin by FPC, and its unit is Hz. Assuming that the number of channels is $C$ , the format of obtained original data meets $x_{\textrm{sample}} \in{R^{N \times C}}$ .

Figure 7.

Schematic diagram of sliding window.

After the original data is obtained, sliding window is used to segment the data to construct multiple sub-samples. The specific segmentation method is shown in Fig. 7. Assuming that the length of the sliding window is $ W$ , the sliding step is $ \lambda$ , and the number of sub-samples after segmentation is $ L$ , the relationship is:

(3) \begin{equation} LW - \left ({L - 1} \right )\!\lambda \le N \end{equation}

where the number of sub-samples of e-skin data can be obtained as $ L = \left [{\frac{{N - \lambda }}{{W - \lambda }}} \right ]$ , where $ \left [ \cdot \right ]$ represents the largest integer not greater than the number in it. The format of the resulting sub-images is ${x_{\text{sub-sample}}} \in{R^{W \times C}}$ .

5.2. LSTM neural network algorithm

LSTM network is a special recurrent neural network (RNN). Ordinary RNN cannot deal with the long-term dependency problem of time sequence signals in practice, while LSTM can deal with the long-term dependency problem due to its special structure. In addition, LSTM networks are currently widely used in speech recognition, machine translation, text generation and other fields. Therefore, this network algorithm is chosen to recognize our e-skin signals.

The basic principle of LSTM neural network is shown in Fig. 8. Its internal mechanism is to regulate the information flow through forget gate, input gate, and output gate. Each LSTM cell has its own unit state ${c_t}$ , and LSTM uses forget gate and input gate to control the content of the unit state ${c_t}$ at the current moment.

Figure 8.

Schematic diagram of LSTM network.

The forget gate determines how much of the unit state ${c_{t-1}}$ at the previous moment is retained to the current moment ${c_t}$ . The output of the forget gate is:

(4) \begin{equation}{f_t} = \sigma \!\left ({{W_f} \cdot \left [{{h_{t - 1}},{x_t}} \right ] +{b_f}} \right ) \end{equation}

where ${\sigma }$ represents sigmoid activation function, ${W_f}$ is the weight matrix of the forget gate, ${h_{t-1}}$ represents the output of the previous moment, ${x_t}$ represents the input of the current moment, and ${b_f}$ is the bias term of the forget gate.

The input gate determines how much of the input of the network at the current time is retained to the current state ${c_t}$ , and the output of the input gate is:

(5) \begin{equation}{i_t} = \sigma \!\left ({{W_i} \cdot \left [{{h_{t - 1}},{x_t}} \right ] +{b_i}} \right ) \end{equation}

where ${W_i}$ is the weight matrix of the input gate and ${b_i}$ is the bias term of the input gate. In addition, the cell state $\tilde c_t$ describes the current input:

(6) \begin{equation}{\tilde c_t} = \tanh \!\left ({{W_c} \cdot \left [{{h_{t - 1}},{x_t}} \right ] +{b_c}} \right ) \end{equation}

where ${W_c}$ is the weight matrix of the current input, and ${b_c}$ is the bias term of the current input.

The forget gate, the input gate and the state of the current input ${\tilde c_t}$ jointly determine the unit state at the current moment ${c_t}$ ,

(7) \begin{equation}{c_t} ={f_t} \circ{c_{t - 1}} +{i_t} \circ{\tilde c_t} \end{equation}

where $ \circ$ represents vector cross product operation. Through this combination, LSTM adds the current memory to the long-term memory, forming a new memory state ${c_t}$ .

Then, the output gate controls the updated memory state, and the output of output gate is:

(8) \begin{equation}{o_t} = \sigma \!\left ({{W_o} \cdot \left [{{h_{t - 1}},{x_t}} \right ] +{b_o}} \right ) \end{equation}

where $ W_o$ is the weight matrix of the output gate and $ b_o$ is the bias term of the output gate. The output of the LSTM unit $ h_t$ is finally determined by the output gate $ o_t$ and the new memory $ c_t$ :

(9) \begin{equation}{h_t} ={o_t} \circ \tanh \!\left ({{c_t}} \right ) \end{equation}

5.3. Experimental scheme

To verify the potential of the developed e-skin, we design multiple gesture recognition scenes. The gestures dataset in the experiment are divided into static gestures (SG) and dynamic gestures (DG). The SG include four categories, namely, the flex and extend of the index finger and the thumb, and the flex and extend of the remaining three fingers, as shown in Fig. 9: SG1, SG2, SG3, and SG4. The DG include five categories, which are defined as the gestures generated by the continuous movement of the hand. The transformation of gesture state is shown in Fig. 9: DG1, DG2, DG3, DG4 and DG5.

Figure 9.

LSTM network structure for five dynamic gestures and four static gestures recognition.

In order to realize the recognition of SG and DG at the same time, we specially design a LSTM network adapted to the e-skin signal to decode different gestures. The structure of the network is shown in Fig. 9. The time step $ t$ is set to 20, that is, the input data dimension of the network is $ 20 \times 11$ , which represents the amount of data of 100 ms. In addition, a 2-layer LSTM network is designed, and a fully connected layer and a softmax layer are set after the last time step. The construction of this network is based on the open source keras deep learning framework.

A participant, whose back of hand is the same size as the designed e-skin, is involved in the experiment. Then, the participant wears our e-skin to repeat different movements. Finally, based on the designed LSTM network, training and testing are carried out on the constructed samples, and 91.22% recognition accuracy is obtained. The confusion matrix of all movements recognition is shown in Fig. 10.

Figure 10.

Confusion matrix diagram for dynamic gesture recognition.

6.1. Robotic hand platform

We have independently developed a multi-fingered hand platform. The specific internal structure of the robotic hand is shown in Fig. 11(a). This robotic hand has 2 degrees of freedom. The index finger and thumb can be flexed and extended at the same time, as can the remaining three fingers. They are driven by 5 W and 2 W motor, respectively. In addition, the force-bearing parts of the robotic hand are made of aluminum, and the shell is made of 3D-printed resin material. Such a design not only enables strong mechanical properties but also reduces its weight.

Figure 11.

(a) Internal structure of the robotic hand. (b) Implementation diagram of robotic hand teleoperation.

6.2. FSM based robotic teleoperation

To completely match the previous gesture recognition results, we consider the four postures POS1, POS2, POS3, and POS4 of the robotic hand with the SG SG1, SG2, SG3, and SG4. DG are used as a priori for static gesture transformations to prevent misoperations caused by misrecognition. The mapping state transition diagram of the robotic hand posture and gesture dataset is shown in Fig. 12. The robotic hand adopts PD-based position control.

Figure 12.

State transition diagram of gestures and robotic hand.

The specific implementation of the robotic teleoperation is shown in Fig. 11(b), which is mainly divided into the following steps:

• The WiFi communication between the e-skin sensor terminal and Qt-based HMI software terminal is established to ensure that HMI software terminal can collect signals in real time.

• The socket communication between the HMI software and the signal processing terminal in python is established. Then, the pre-trained LSTM classification model is loaded in advance, and the online recognition and analysis of gesture signals are performed.

• Finally, the socket communication between the HMI software and the MFC-based control terminal is established, and the recognition results are transmitted to the robot to realize the state switching and follow-up of the robot.

We select SG1 as the initial gesture state and then test the state following performance of the e-skin wearer in the order of SG1-SG2-SG4-SG3-SG1-SG4-SG1. After several tests, the robotic hand can effectively follow the gestures to switch states, and the average switching delay is less than 1s. In addition, the false recognition rate is low, and barely false actions of the robotic hand are identified.

When the robotic fingers are in the open or closed state, the relative positions of the driving motors are defined as 0 and 1, respectively. After the experimental tests, the position displacements of the driving motors during the state switching are shown in Fig. 13. Figure 13 shows that the developed robotic hand has good position control and can move the corresponding position rapidly.

Figure 13.

The position displacements of the driving motors in different states.