2.1 Humanoid robot

Figure 1 shows the 45.5-cm-tall humanoid robot DARwIn-OP, which is an abbreviation for Dynamic Anthropomorphic Robot with Intelligence-Open Platform. DARwIn-OP, developed by ROBOTIS, operates with a 1.6 GHz Intel Atom Z530 (32-bit) onboard 4 GB flash solid-state drive and comprises 20 Dynamixel MX-28 actuators. The actuators offer 6-DOF leg movement, dual 3-DOF arm movement, and dual 2-DOF neck movement. The robot is equipped with a Universal Serial Bus (USB 2.0) high-definition Logitech C905 camera and has a three-axis gyroscope and a three-axis accelerometer as sensors to complete the task in the HuroCup competition. The motor ID and the related angles are defined in Table 1, where R indicates the right side and L represents the left side. Four force-sensing register (FSR) modules (Ha et al., Reference Ha, Tamura and Asama2013) are placed in each foot for ground reaction force measurements. These FSRs are used to find the zero moment point (ZMP), which indicates the most stable region of the robot.

Figure 1

Schematic of the DARwIn-OP servomotor numerically assigned positions

Table 1

DWRwIn-OP servomotor ID position and angle.

2.2 Problem description

For the HuroCup competition, the robot needs to step up to irregularly shaped differently colored boards according to the rules of FIRA’s lift and carry event. The challenge in this event is that the height of the board for each step is approximately 1.8–2.0 cm, as shown in Figure 2. To perform the step-up motion successfully, the robot must automatically recognize the edge of the color boards.

Figure 2

Photograph of the robot performing the lift and carry event

3.1 Human motion imitation

To accomplish the lift and carry task, the robot is designed based on imitations of human motion. Human motion is captured by a Kinect v2 sensor, and the corresponding skeletal joints are recorded and screened by a personal computer, as shown in Figure 3. To make the original and imitated motion consistently, the number of human joints, captured by the Kinect as shown in Figure 3(a) and (b), is modified according to the servomotor ID of DARwIn-OP.

Figure 3

Twenty human skeletal joints captured by the Kinect. (a) Skeletal joints. (b) Kinect image

Next, the Kinect records the human motion of stepping up in three different views so that the motion can be analyzed and applied to train the robot. These three views of human motion—side, front, and top—are shown in Figures 4–6, respectively. The joints of human motion are not easily recognized; hence, the different views provide more recognizable motion for further analysis. The motion of stepping up is divided into five states, and the depth images for each state from the Kinect are shown in Figure 4. In Figure 4(a), the human is standing up in a ready position. The human now raises its left foot to step up to a stair (Figure 4(b)). Then, the human adjusts its center of mass by lowering its body and maintains balance by moving its arms, as shown in Figure 4(c). Next, the human keeps its left foot on the stair and raises its right foot to the stair, as seen in Figure 4(d). Finally, the human finishes the step-up task (Figure 4(e)).

Figure 4

Five states of the human step-up motion from the side view

Figure 5

Five states of the human step-up motion from the front view

Figure 6

Five states of the human step-up motion from the top view

3.2 Data analysis and fuzzy algorithm

Simple geometrical operations are used to calculate the angles between vectors, as shown in Equations (1) and (2), where V _C3C1 represents the vector of the left elbow to the left and V _C3C5 represents the vector of left elbow to left wrist as shown in Figure 7, and then the joint angle for left elbow can be calculated by vectors V _C3C1 and V _C3C5

(1) \begin{equation}\overrightarrow {{\textbf{V}_{{C_3}{C_1}}}} \times \overrightarrow {{\textbf{V}_{{C_3}{C_5}}}} = \left| {\overrightarrow {{\textbf{V}_{{C_3}{C_1}}}} } \right|\left| {\overrightarrow {{\textbf{V}_{{C_3}{C_5}}}} } \right|\cos ({\theta _{LE}})\end{equation}

(2) \begin{equation}{\theta _{LE}} = {\cos ^{ - 1}}\left( {\frac{{\overrightarrow {{\textbf{V}_{{C_3}{C_1}}}} \times \overrightarrow {{\textbf{V}_{{C_3}{C_5}}}} }}{{\left| {\overrightarrow {{\textbf{V}_{{C_3}{C_1}}}} } \right|\left| {\overrightarrow {{\textbf{V}_{{C_3}{C_5}}}} } \right|}}} \right)\end{equation}

Figure 7

Schematic showing the angle calculation

However, there are several differences between a human body and a humanoid robot. For example, the difference in the DOF between humans and humanoid robots was discussed by Zheng et al. (Reference Zheng, Fu, Yang and Lv2012). Hence, direct movement mapping between a human arm and a humanoid robot arm is not feasible owing to differences in kinematics and dynamics parameters. Instead, the humanoid robot has to imitate human motion. The angles of the human step-up motion for five stages are listed in Table 2. The ID number is the same as that listed in Table 2. Because R_Hip_Yaw (ID 7), L_Hip_Yaw (ID 8), R_Ankle_Roll (ID 17), and L_Ankle_Roll (ID 18) are not available from the depth image, and the head (ID 19 and ID 20) does not move during the step-up process, servomotor IDs 7, 8, 17, 18, 19, and 20 are excluded in Table 2. In Table 2, IDs 1, 2, 11, and 12 are marked in blue. Because IDs 1 and 2 control robot arm movement and IDs 11 and 12 control robot body tilt, these four IDs significantly vary during the step-up process. The remaining IDs have less variation during the step-up process. These four motors were selected to be involved in the designed fuzzy logic rule because they control the balance of the robot and thus are the most significant factors influencing its step-up motion.

Table 2

Angles of human step-up motion for the five stages.

The fuzzy logic system is designed as follows (Betin et al., Reference Betin, Pinchon and Capolino2000; Zhao and Bose, Reference Zhao and Bose2002). The parameter degree obtained from the human is called ϕ _i , and i = 1, 2, 11, and 12 are the inputs called X _i in the fuzzy system. Each parameter X _i will be fuzzed into value within the range [0, 1], and the membership functions are very low (VL), low (L), medium (M), high (H), and very high (VH). The membership functions evaluate the ability x belonging to the set F denoted by μ _F (x), in which x is the value of the estimated parameter and F is a set of real values within the range [0, 1], as defined in Figure 8. Even though different motors have different values for VL, L, M, H, and VH, these values are used to classify the fuzzy range of inputs. Fuzzy sets: F∈{VL, L, M, H, VH}, μF is the degree of the membership function for input fuzzy set F, {0 20 70 90 120 180}, {0 20 50 80 100 180}, {−30 −10 20 60 90 180}, and {0 30 50 60 90 180} are the values of input fuzzy set F for IDs 1, 2, 11, and 12, respectively.

Figure 8

Membership functions for each input X _i

Next, the single-input and single-output fuzzy inference systems are adapted to define the fuzzy rule if… then… as follows:

Similarly, membership function evaluates the ability Y _i belonging to the set A denoted μ _A (y), in which y is the value of an estimated parameter and A is a set of real values within the range [0, 1] as defined in Figure 9. Finally, based on the fuzzy methodology, the center of gravity method is applied in this study to obtain the related output Y _i which is the angle of motor ID 1, ID 2, ID 11, and ID 12 for the robot as shown in Equation (3).

(3) \begin{equation}{Y_i}^* = \frac{{\int\limits_{{y_i} \in Y} {{\mu _A}({y_i}) \cdot {y_i}{\kern 1pt} \,d{y_i}} }}{{\int\limits_{{y_i} \in Y} {{\mu _A}({y_i})\;d{y_i}} }}\end{equation}

Figure 9

Membership functions for each output Y _i

After the fuzzy algorithm calculation, the relationship between the angle variation in the human and robot step-up motion for IDs 1, 2, 11, and 12 is plotted (Figure 10).

Figure 10

Designed ID angle variation for the human and robot step-up motion

3.3 Robot motion adjustment and balance

Inside the sole area of the support foot, the FSR maps the current pressure point at which the whole body exerts its weight. The position values acquired from the FSR mounted in the DARwIn-OP humanoid are represented with integral quantities ranging from 0 to 254, as shown in Figure 11. For the double-stand phase of the humanoid robot, the stable region of the ZMP (the red rectangular region in Figure 11) is tested (Chien et al., Reference Chien, Sung and Kim2015) and is located at the center of both feet.

Figure 11

Definition of FSR values and the stable region for the DARwIn-OP humanoid

The related motors are adjusted to find a stable position for the humanoid robot (Figure 12). First, the related parameters of the robot are imported to the algorithm to obtain the ZMP from the FSRs of both feet. Next, a decision-making process is followed to adjust the related parameters of the motors to ensure that the ZMP from the FSR is in the stable region. If the ZMP is too far to the left, motor IDs 4, 6, 10, and 14 on the left side of the robot are adjusted. Similarly, if the ZMP is too far to the right, motor IDs 3, 5, 9, and 13 on the right side of the robot are adjusted. If the ZMP of the robot is too forward, related motor IDs 1, 2, 11 and 12 are adjusted to make the robot move its center of mass to the back and vice versa.

Figure 12

Flowchart showing how to adjust the robot position to be stable according to the ZMP

The ZMP is determined by the values of FSR. Then, the ZMP is adjusted based on the rule described below. If ZMP is too left, the motor IDs 4, 6, 10, and 14 are modified with a value Δ $\theta$ _i , i = 4, 6, 10 and 14 as shown in Table 3. The listed values were obtained by experiments. The method is also applied when ZMP is at right, forward, and backward, respectively. If the motor needs to rotate clockwise, the angle increases with a small amount; otherwise, the angle decreases. The adjustment for ZMP is summarized in Table 3.

Table 3

ZMP adjustment for Figure 12

Moreover, motor IDs 15, 16, 17, and 18 may affect the balance of the robot as well. The motors IDs 15 and 16 control the soles of the feet to left or right, while IDs 17 and 18 control them forward or backward. Therefore, the angles of those motors are within −15° to 15° to keep the soles of the feet parallel to the ground in X and Y planes so that the robot can step on the stairs with its entire feet and prevent itself from tripping.

Besides, a humanoid robot cannot move from one step to another step directly because suddenly changing position leads to balance loss and falling. Therefore, for a humanoid robot, a transition state (Chiang et al., Reference Chiang, Kuo, Lin and Chen2017), as illustrated in Figure 13, is required for balance and motor protection. The robot is designed so that it moves from a standing straight position (Figure 13(a)) to raise the right foot up (Figure 13(d)). In the process, two more steps, shown in Figure 13(b) and (c), are added to the transition state to maintain balance. These steps are added in the step-up motion of the robot for each of the five stages.

Figure 13

Adding transition steps in the robot’s motion to maintain balance

To handle a wide range of transitions, we also consider the case where two points (such as in Figure 13(a) and (d)) in the state space cannot be joined in one step. Specifically, given two states, S _i and S _j , we design transition states or two intermediate states – S _a ^i→j (Figure 13(b)) and S _b ^i→i (Figure 13(c)) (Da et al., Reference Da, Hartley and Grizzle2017), as shown in Equation (4) – and the corresponding control parameters so that the robot transitions are implemented. The transition state is obtained by linear interpolation, as shown in Equations (5) and (6).

(4) \begin{equation}{S_i} \to S_a^{i \to j} \to S_b^{i \to j} \to {S_j}\end{equation}

(5) \begin{equation}S_a^{i \to j} = \alpha \;{S_i}(\Theta ) + (1 - \alpha )\,\;{S_j}(\Theta )\end{equation}

(6) \begin{equation}S_b^{i \to j} = \beta \;{S_i}(\Theta ) + (1 - \beta )\,\;{S_j}(\Theta )\end{equation}

where $\alpha \in (0,1)$ , $\beta \in (0,1)$ , and $\Theta$ are all motors parameters.

3.4 Robot vision system

The robot is supposed to autonomously walk to and step up onto each color board in the lift and carry event. Thus, the vision system is used to recognize the color of the board and find a suitable step-up position. This position is the site where the robot’s balance is most likely to be maintained, with the size of the robot feet (10.4 cm in depth and 16.5 cm in width) taken into consideration. Thus, the relationship between the pixel size in the image and the real size of the color board, as shown in Figure 14 from the view of the robot, must be obtained. Next, the depth vector D, defined in Equation (7), is a 1 × 32 dimension vector and is obtained from the first color board location from the bottom row for each column, as shown in Figure 15(a).

(7) \begin{equation}\textbf{D} = \left[ {{d_1},{d_2}, \cdots ,{d_{32}}} \right]\end{equation}

(8) \begin{equation}\text{If } d_i<T,\ \text{for}\ i=1, \ldots, 32, \text{then step up} \end{equation}

Figure 14

The relationship between pixels versus dimension

Figure 15

The robot tries to find the edge of the color board

Here, d _i is the distance from the first color board grid location to the robot for the ith column.

Then, every d _i is calculated. If all the di are less than the threshold value T (in this experiment, T = 7 pixels) as described in Equation (8), then the robot has hit the edge of the color board, as shown in Figure 15(b).