A) Human target motion model

Two different human motion models have commonly been used in the literature: mathematical parametric model [Reference Van Dorp and Groen10] and empirical non-parametric model [Reference Vignaud, Ghaleb, Le Kernec and Nicolas2]. Although parametric models are designed for some specific motions (i.e. walking and running), empirical non-parametric models, which are based on human motion capture data, are usually more realistic and accurate. In CMU database [11], a motion capture system consisting of 12 infrared cameras and 41 markers attached to a human body, was used to record motions. Datasets of various motion types (e.g. walking, running, and jumping) are provided. For this study, we select five most crucial markers (i.e. torso, left hand, right hand, left foot, and right foot), and an equal RCS is assumed for the reflection from all the markers for simplicity. Then we construct human range profiles by coherently summing the echoes from different parts of the human body. According to the human kinematic studies [Reference Chen5], one complete human gait of our model consists of four main phases (see Fig. 1): (1) double support (both feet are on the ground), (2) right stance (only the right foot is on the ground), (3) double support, and (4) left stance (only the left foot is on the ground).

Fig. 1.

One complete human gait.

B) Radar signals of the human target

Currently there are three tools to analyze radar targets, namely HRRP, micro-Doppler images, and RD images. Range profile is used as the most common tool due to its simplicity. We simulated range profiles for a walking human model (Fig. 1) using Gaussian pulses with an operational frequency from 0.8 to 1.6 GHz (−10 dB), which provides a theoretical down-range resolution of 0.18 m. The range profiles simulated show the slow-time evolution of five different scatterers (see Fig. 2). The largest fluctuations in the range profiles result from the backscattering of the feet, and the reflections of the torso and hands are heavily overlapped. Although a certain level of target signatures is preserved in range profiles, the range profiles do not show the Doppler information.

Fig. 2.

Simulated human target range profiles.

The micro-Doppler of a walking human can be derived by applying time-frequency transform to the range profiles. Different transforms (e.g. short-time Fourier transform and Wigner–Ville distribution) can be chosen considering the trade-off between Doppler resolution and cross-term interference. The micro-Doppler image corresponding to the range profiles in Fig. 2, computed by short-time Fourier transform, is shown in Fig. 3. Unlike the range profiles, micro-Doppler preserves the target Doppler signature, while it neglects the range information.

Fig. 3.

Simulated human target micro-Doppler image.

Similar to the micro-Doppler image, RD image is also derived from range profiles. The basic idea of RD processing is to apply Fourier transform to analyze the samples from a fixed range bin over a few successive range responses. The RD image is constructed by the spectra in all the range bins. Although the micro-Doppler image describes the target Doppler signature over a certain period, the RD image only shows the RD signature at one specific instant. The main stages of one human gait, corresponding to the phases in Fig. 1, are illustrated in four RD images (Fig. 4). A cyclic movement is constructed by the RD responses of the arms and the feet around the torso contour. Note that the arm responses are difficult to separate from the torso response, since they overlap with each other.

Fig. 4.

The range-Doppler images of the simulated human gait (the person is walking toward the radar with a constant velocity).

The range profiles and the micro-Doppler image both illustrate the target movement as a function of the slow-time, whereas the RD image describes the RD pattern at one particular instant. Therefore, we proposed RDVS [Reference He, Le Chevalier and Yarovoy9] previously to describe the slow-time evolution of target RD signatures. As shown in Fig. 5, RDVS is built by successive RD frames. One pulse repetition interval (PRI) can be used as a typical interval between two frames. Only five of all the RDVS frames are shown here, and the human RD contours between consecutive frames are also demonstrated.

Fig. 5.

Simulated range Doppler video sequence of the human target.

Range profiles, micro-Doppler image, and RDVS represent the radar time series based on range, Doppler, and RD signatures, respectively. As we shall see, they all exhibit unique self-similarity characteristics, due to the cyclic human movement. In the next section, the self-similarity characteristic of these three different radar signals will be analyzed.

A) Self-similarity matrix (SSM)

Recurrence is quite common in dynamic systems, and multi-dimensional time series is usually used to represent such systems. Eckmann et al. first proposed the recurrence plot (RP) [Reference Eckmann, Kamphorst and Ruelle12] to perform time series analysis of dynamic systems. As a variant of RP, SSM [Reference Cutler and Davis6] can be considered as a generalization of the correlation matrix, measuring the “similarity” – rather than just the correlation – between successive samples of the multidimensional time series. SSM is also an effective tool to analyze system behaviors, since the textures in SSM (e.g. the peaks and valleys, if we illustrate SSM as an image) are distinctive for different time series.

In the field of video processing, SSM has already been successfully applied for analyzing human motions in video sequences, such as multi-view human action recognition [Reference Junejo, Dexter, Laptev and Perez8], human gait recognition [Reference BenAbdelkader, Cutler and Davis7, Reference Cutler and Davis13], and video matching [Reference Shechtman and Irani14]. For the application of camera-based human action recognition, Junejo et al. [Reference Junejo, Dexter, Laptev and Perez8] claimed that an important structural stability of SSM can be found for a moving person observed with different cameras. This indicates that, while cameras observe the person from various ranges and perspectives, a similar slow-time feature (i.e. SSM) can still be captured.

Since range profiles, micro-Doppler image, and RDVS can all be seen as multi-dimensional time series, SSM should also be useful to analyze them. Although radar signals differ significantly from camera-based optical images, we will demonstrate that the SSM theory can be usefully extended to radar signals.

The range profiles, micro-Doppler image, and RDVS are represented by multi-dimensional time series S = {F ₁, F ₂,…,F _N}. The dimension of S depends on the specific type of the radar signal. Taking the range profiles for example, they are the time series consisting of N one-dimensional vectors (pulse responses). Similarly, micro-Doppler time series are composed of N one-dimensional vectors (Doppler responses), and RDVS time series have N two-dimensional vectors (RD images). Then, the SSM of S is defined as a square symmetric matrix:

(1)$$\left[{d_{ij} } \right]_{i\comma j=1\comma 2\comma ...\comma N}=\left({\matrix{ {d_{11} } & {d_{12} } & \ldots & {d_{1N} } \cr {d_{21} } & {d_{22} } & \ldots & {d_{2N} } \cr \ldots & \ldots & \ldots & \ldots \cr {d_{N1} } & {d_{N2} } & \ldots & {d_{NN} } \cr } } \right).$$

Here d _ij denotes the similarity coefficient, which measures the degree of similarity between F _i and F _j.

The exact form of the SSM depends on the similarity measure used for calculating d _ij. Different similarity measures have already been proposed [Reference Van der Weken, Nachtegael and Kerre15]. The selection of a specific measure depends on particular applications. Mutual information (MI) [Reference Pluim, Maintz and Viergever16], a well-known statistical measure for comparing vectors, is selected to compute the SSM for radar signals. Unlike other classical measures (e.g. Euclidean distance, normalized cross-correlation) that compare intensities directly, the MI approach, which is based on Information Theory, computes the statistical dependence between two vectors using histograms. Accordingly, MI-based SSM is independent of exact target profiles (range, Doppler, or RD) and thus is less sensitive to the range alignment process – an essential step to compute SSM in the next section.

Since MI describes the shared information between two vectors statistically, the MI-based SSM does provide a simple way to quantitatively exhibit time series patterns. The MI between vector X and Y is defined by:

(2)$$I\lpar X\semicolon \; Y\rpar =H\lpar X\rpar +H\lpar Y\rpar - H\lpar X\comma \; Y\rpar \comma \;$$

where H(X) denotes the Shannon entropy of vector X, and H(X,Y) indicates the joint entropy of X and Y. Shannon entropy [Reference Shannon17] quantifies the expected information contained in a vector. It has been widely used in radar-related applications. In [Reference Li18], entropy-based approach is applied to remove noise in micro-Doppler images. The threshold used in denoise procedure is selected by maximizing the sum of the object and background entropies. Entropy is also used to assess the additional information introduced by new features in human target classification [Reference Molchanov, Astola, Egiazarian and Totsky19]. In this study, MI is defined via the entropy concept. The entropy of vector X is defined by:

(3)$$H\lpar X\rpar =- \sum\limits_{m=1}^M {P_m \log _2 P_m }\comma \;$$

where M is the number of histogram bins used to estimate the empirical probability density. The histogram meets the following condition:

(4)$$N_p=\sum\limits_{k=1}^M {N_k }\comma \;$$

where N _p denotes the size of the vector X, and N _k indicates the number of the samples whose intensities fall into the kth bin. The empirical probability density in the mth bin is given by:

(5)$$P_m=\displaystyle{{N_m } \over {N_p }}\quad \lpar m=1\comma \; 2\comma \; \ldots\comma \; M\rpar.$$

Similarly, the joint entropy H(X,Y) can also be defined as:

(6)$$H\lpar X\comma \; Y\rpar =- \sum\limits_{m=1}^M {\sum\limits_{n=1}^M {P_{mn} \log _2 P_{mn} } }\comma \;$$

where P _mn denotes the joint empirical probability density of vector X and Y. More precisely, P _mn indicates the probability for a given sample, which falls into the mth bin in histogram i−1, to be located in the nth bin in histogram i (it is a kind of transition probability). It is given by:

(7)$$P_{mn}=\displaystyle{{N_q } \over {N_p }}\comma \;$$

where N _q is the size of class set Q _mn. The set Q _mn is defined as:

(8)$$Q_{mn}=\left\{{l\vert g\lpar i\! -\! 1\comma \; l\rpar =m\comma \; g\lpar i\comma \; l\rpar =n} \right\}\comma \;$$

where g(i,l) is the operator to get the histogram value of vector i at sample l(1 ≤ l ≤ N _p), and 1 ≤ m, n, ≤ M. Note that here we assume X and Y have the same size N _p.

Then, MI can be derived by equations (2), (3), and (6) as:

(9)$$I\lpar X\semicolon \; Y\rpar =\sum\limits_{m=1}^M {\sum\limits_{n=1}^M {P_{mn} \log _2 \displaystyle{{P_{mn} } \over {P_m P_n }}} }.$$

It can be analyzed as a similarity measure between the actual empirical joint probability, P _mn, and the empirical joint probability in the case of independent vectors, P _mP _n. Finally, the entries in SSM can be obtained by:

(10)$$d_{ij}=I\lpar F_i\semicolon \; F_j \rpar.$$

B) Extracting SSM of radar signals

In general, extracting the SSM of the radar signals consists of the following steps. First, we track the targets in range or RD domain. Then the range profiles (or RDVS) are aligned using target trajectory information. The aligned range-profiles and RDVS can be seen in Figs 6 and 7, respectively. Note that the micro-Doppler image does not require alignment, since it naturally sums the information from all the range cells. Although this study did use the position of the torso marker as the centroid in alignment for simplicity, more details about tracking and alignment for real data can be found in our previous work [Reference He, Savelyev and Yarovoy20]. It is also worth pointing out that accurate tracking is not necessarily required for computing SSM, since MI depends on histograms, rather than the target profiles. At last, MI-based SSM can be computed for different radar signals.

Fig. 6.

Alignment of the range profiles.

Fig. 7.

Alignment of the range-Doppler video sequence.

The SSMs of different radar signals are shown in Fig. 8. We may note the following unique properties of SSM: (1) The diagonal shows the maximum amplitude of each row, because the MI between one image and itself is always maximum; (2) The repeated structures demonstrate the periodic characteristic of the human motion, such as the bright lines parallel to the main diagonal in (a) and (c), and the bright “square” in (b). Although the textures of (a), (b), and (c) all show some distinctive target signatures, the SSM of RDVS in (c) obviously demonstrates the target periodic characteristic in a more direct and simple way. One possible explanation could be that RDVS is more informative than range profiles and micro-Doppler image, as both range and Doppler information of the target are used. The parallel lines in (c) are experimentally found to be corresponding to the cyclic motion of the human, and in fact the distance between two parallel lines indicates the period of a half cycle of the human motion (i.e. one complete motion of one leg).

Fig. 8.

Self-similarity matrix of radar signals.

C) Angle-invariant characteristic of SSM

Since SSM mainly exhibits the self-repeating information of the target motion, and MI is independent of target range/Doppler profiles, SSM is considered to be invariant to radar observation angles. Strictly speaking, SSM is not fully angle-invariant around 90° and 270°, since targets can often not be detected in these angles (detection is the pre-condition for analyzing SSM). The SSMs (Figs 9(b)–9(e)) were simulated from four different radar locations, and the simulation scenario is shown in Fig. 9(a). Results show that the HRRP-based SSMs presents weaker periodic structures, and the micro-Doppler-based SSMs are not robust at some angles (i.e. 135°, 225°). On the contrary, the RDVS-based SSMs demonstrate similar periodic textures in all aspect angles. This concludes that RDVS is able to produce robust angle-invariant SSMs. These important characteristics of SSM indicate its potential in the applications of target association, classification and recognition in multistatic radars.

Fig. 9.

(a) A person is walking from P1 to P2. Radars are deployed at four locations; (b)–(e) Self-similarity matrix of different radar signals from four radar locations.

D) Component analysis of SSM

Since the RDVS-based SSM is seen to be more robust, it has been used for further analysis. The SSMs of different body parts are shown in Fig. 10. Although the SSM of the torso and hands (Fig. 10(a)) show only weak similarity along the slow-time axis, the SSM of the torso and feet (Fig. 10(b)) show clear bright lines that are parallel to the main diagonal. This indicates that the most obvious contribution for SSM is from the combination of torso and feet. One possible reason for this phenomenon could be that the responses of feet and torso are more separable in one RD image, which results in larger contour changes in the adjacent frames of the RDVS. It can also be inferred from the results that the hands movement seems less periodic than the feet movement, and thus spreads over the SSM image.

Fig. 10.

Self-similarity matrix of different body parts.

A) Measurement set-up

The measurement was conducted in a room at Delft University of technology to provide a typical indoor environment. An m-sequence pseudo-noise radar equipped with one transmitting horn antenna and two receiving horn antennas was used for the data acquisition (Fig. 13(a)). The operational frequency band (at −10 dB level) of the radar is 0.8–1.6 GHz, which gives a down-range resolution of 0.187 m. The sampling clock frequency is 4.5 GHz and the length of the sampling window in propagation time is 113 ns, which covers an unambiguous range of 16.95 m. In the receiver, the received signal is correlated with a replica recorded from the transmitter. Thus a long m-sequence signal is compressed to a short impulse and the signal-to-noise-ratio is increased significantly via pulse compression. The PRI is 0.06 s, which leads to a maximum unambiguous Doppler frequency of 8.33 Hz for the RD processing. The measurement was carried out with one human target moving in the room from P1 to P2 (Fig. 13(b)) with a constant velocity.

Fig. 13.

Measurement setup and scenario.

B) Range-doppler video sequences

The RDVS was generated by RD processing in each receiver. Each frame was computed with a CPI duration of 1 s (16 successive pulses), and the time between two successive frames is one PRI around 0.06 s. The range profiles used for generating RDVS is shown in Figs 14(a) and 14(b), and the RD frames in both channels at one specific slow-time instant are illustrated in Figs 14(c) and 14(d). For simplicity, the maximum amplitude of each RD image was selected as the centroid of the human target for target alignment.

Fig. 14.

Human backscattering in multistatic UWB radar.

C) Self-similarity matrix

The SSMs obtained in different receivers are shown in Figs 15(a) and 15(b). It can be noted that they are very similar to the SSMs obtained in the previous simulation (see Figs 8 and 9). The periodic characteristic of the human motion is demonstrated clearly by the repeated lines, and the time between two parallel lines indicates the period of a half cycle of the human motion. The pixelization of these two images is due to the lower frame repetition frequency (16.7 Hz) of the RDVS, whereas 120 Hz was used in previous simulations. The cadence frequencies of the two SSMs are shown in Fig. 15(c). Similar cadence frequencies (around 1 Hz) were obtained in both receivers. This proves again the angle-invariant characteristic of the SSM. Finally, it is interesting to point out that SSM preserves rich target information, and cadence frequency is just one of the examples. Further investigation can be interesting to analyze SSM and extract more target signatures from it.

Fig. 15.

Gait analysis for the measured human target data.