Contour reconstruction from the measured reflectivity

Contour reconstruction of the human body via a noisy collection of 3D radar reflectivity is challenging. An automatic, real-time reconstruction algorithm for extracting the main contours is introduced through thresholding 2D cross-section slices of normalized reflectivity data into a 2D binary image. Pixels on the thick body-surface contour are assigned to unity, and the rest are assigned to zero. The width of contours is determined by the finite bandwidth of the radar. A Canny edge detector filter [Reference Gonzalez and Woods33], via the MATLAB built-in function edge, applies a Gaussian filter, with the standard deviation of $\sqrt 2$ and kernel size of 11×11, to decrease the image noise level, find the edges of the contour reflectivity, and remove the spurious output. The filter uses two thresholds to detect both strong and weak edges; thus, it is more resistant to the noise. The threshold values, typically from 0.06 for the lower threshold to 0.35 for the higher threshold, are different for topologically different body regions, but the same with or without the attached object. The goal is to accurately extract the nominal part of the body with no object attached, and use it to reconstruct the nominal body contour with affixed objects or shadowed by other body parts. The Canny edge detector filter parameters are selected empirically by applying the algorithm to a training dataset of reflectivity images, with and without the attached objects. The optimized parameters are further verified for a second testing dataset. Each dataset consists of reflectivity images from a variety of volunteers of different weights and genders wearing different clothing types and different numbers and types of concealed objects. The actual 3D images are generated by the HD-AIT screening system; a mm-wave laboratory prototype consists of a wide-bandwidth, circularly-polarized, multi-bistatic array of antennas working at 10–40 GHz with fine depth resolution and minimal attenuation through clothing [Reference Sheen, Fernandes, Tedeschi, McMakin, Jones, Lechelt and Severtsen22, 32]. Opening and closing morphological operations are then used to remove small, isolated, or disconnected collections of pixels, and to unify the remaining pixels in the binary image [Reference Gonzalez and Woods33]. More explicitly, the MATLAB built-in function imclose is used to expand parallel contours into one fat contour. Next, a second morphological operation is performed, via the MATLAB function bwmorph, to shrink each solid object with no hole into a minimally connected contour, and to reduce each object with a hole to a continuous ring midway between its outer boundary and the hole inside [34]. MATLAB functions bwareaopen and imopen are also used to remove features smaller than certain number of pixels, while the main axial contours are sharpened slice-by-slice. This further reduces the noise level of the radar image. The extracted contours can clearly indicate the position of foreign anomalies.

Figure 1 shows a projected image of the back view of a subject's legs captured by the HD-AIT system. There is a cylindrical 20 × 3 × 3 cm silicone rubber slab attached to the lower part of the left thigh. A 2D axial cross-section of the normalized reflectivity, with centroid object height z = 0.62 m, is shown in Fig. 2(a) for both left and right lower thighs, when the person faces to the direction shown by the arrow. Figure 2(b) depicts the effect of using the Canny edge detector filter, while Fig. 2(c) illustrates the result of applying morphological operations. Finally, Fig. 2(d) shows the clean, thin contour of the lower thighs. The depression in the left thigh is due to the affixed foreign object. The algorithm reduces the raw reflectivity data volume, lowers the noise level, includes object thickness information, and is performed fully automatically and in real-time.

Fig. 1.

Maximum intensity back view of an HD-AIT image of the legs. A cylindrical silicone rubber slab is attached to the lower part of the left thigh.

Fig. 2.

Axial cross-sections of the lower part of the right thigh without the object (first row) and left thigh with the attached object (second row) of Fig. 1 both at z = 0.62 m, when the subject faces to the left in the direction indicated by the arrow: (a) normalized reflectivity, (b) Canny edge detector filter output, (c) morphological processed output, and (d) the reconstructed contour. The depression in the second row of images is due to the affixed cylindrical silicone rubber slab.

Fourier approximation to determine the nominal contours of the body

The estimation of the human body surface contour is necessary for the characterization of the concealed affixed objects [Reference Gonzalez-Valdes, Alvarez, Martinez-Lorenzo, Las-Heras and Rappaport19, Reference Asri and Rappaport30]. We start from the observation that the radar reflectivity images of most parts of the human body are smoothly varying with respect to the circumferential angle φ. However, a foreign affixed object tends to appear in the reflectivity image as a discrete depression or protrusion with rapid radial variation with respect to the circumferential angle. This implies that a few-term Fourier series cannot fit the rapidly varying lumpiness due to the affixed object, but is enough to estimate the smoothly varying nominal body contour even in the presence of a foreign object attached to the body, or when sections of the body have not been fully captured by the multi-bistatic antennas of the screening system.

The Fourier approximation of a 2D nominal body contour is computed as follows: (1) Compute the centroid of the reconstructed contour and make it the center of the Cartesian coordinate system; (2) Transform the contour points from Cartesian to polar representation; (3) Calculate a few-term Fourier series with the amplitude coefficients a _n and b _n, n = 1, 2, …, N, through solving the following linear system of equations:

(1)$$\left[{\matrix{ {\matrix{ 1 & {{\rm cos}( {\varphi_1} ) } \cr 1 & {{\rm cos}( {\varphi_2} ) } \cr } \;\;\;\;\;\matrix{ \ldots & {{\rm sin}( {N\varphi_1} ) } \cr \ldots & {{\rm sin}( {N\varphi_2} ) } \cr } } \cr {\matrix{ \vdots & \vdots \cr 1 & {{\rm cos}( {\varphi_M} ) } \cr } \;\;\;\;\;\matrix{ \vdots & \vdots \cr \ldots & {{\rm sin}( {N\varphi_M} ) } \cr } } \cr } } \right]\left[{\matrix{ {\matrix{ {a_0} \cr {a_1} \cr } } \cr \vdots \cr {b_N} \cr } } \right] = \left[{\matrix{ {\matrix{ {\rho_1} \cr {\rho_2} \cr } } \cr \vdots \cr {\rho_M} \cr } } \right]$$

to fit best the extracted contour with M points (φ _m, ρ _m), m = 1, 2, …, M, captured by the screening system; (4) Define an evenly-spaced array of φ _i spanning from −π to +π, and compute the 2D contour approximation

(2)$$\rho ( \varphi ) = a_0 + \mathop \sum \limits_{n = 1}^N \{ {a_n\cos ( {n\varphi } ) + b_n{\rm sin}( {n\varphi } ) } \} $$

for every φ _i even if the amplitude coefficients are not obtained from the entire 2π circumference, that is, sections of the reconstructed contour are absent from the image or within the object area; (5) Transform the approximated φ − ρ points back to x − y representation.

It is important that only the smooth variations of the extracted surface, with respect to the circumferential angle, be modeled as the nominal body contour. Therefore, the fewest number of terms is utilized for the Fourier modal series to model the nominal body surface. Our results indicate that five-term Fourier series (N = 2) is adequate to approximate human torso but is not enough for more challenging parts of the body. However, seven-term Fourier series (N = 3) is sufficient to approximate the nominal cross-sectional contours of more challenging regions of the body like knees and calves. As an example, a right knee cross-section at z = 0.47 m is shown in Fig. 3. While the five-term Fourier series cannot fit the predominantly triangular cross-section, the seven-term series approximates the entire contour well. The results can be compared quantitatively through computing the minimum distance of each contour point (blue) from the Fourier series approximation (red). While the root mean square error for the five-term series approximation is 0.49 cm, it is 0.15 cm when seven-term series is used. It is noted that when sections of the nominal body contour are not fully captured by the imaging system or are behind the attached foreign objects, the amplitude coefficients are found based on the remaining captured points alone. Any attached object, with faster angular variation relative to the low-order nominal body contour, will be classified as an anomaly (depression or protrusion in the images).

Fig. 3.

(a) Five-term and (b) seven-term red Fourier series approximation to blue right knee contour points at z = 0.47 m.

To show the effectiveness of the developed algorithm, the obtained contour of the lower part of the right thigh cross-section, as shown in the first row of Fig. 2, is examined. Although our contour extraction method correctly reconstructs the complete closed contour, we test the robustness of our Fourier series angular decomposition with artificially removing 1/6 of the contour, as shown in Fig. 4. The seven-term Fourier series fits the remaining blue contour points and estimates the missing green contour data. Figure 4(left) shows the calculated fit in φ − ρ representation, and Fig. 4(right) illustrates the transformed fit in x − y representation. Because of the slow variation of radius as a function of circumferential angle, seven-term modal series is sufficient to accurately approximate the artificially removed portion of the contour. Our results show that the developed algorithm reconstructs the nominal body contour correctly if at least 2/3 of the consecutive data points on the nominal part of the main contour are extracted. For fewer consecutive data points, owing to a relatively large attached object size or an unexpected shadowing, the Fourier series result may diverge, which indicates the algorithm being unable to clear the anomaly. It will be up to the airport security officers to decide what to do with that information.

Fig. 4.

Seven-term Fourier series fit to blue right thigh contour data of the first row of Fig. 2 in φ − ρ representation (left) and x − y representation (right). The result shows reasonable estimation of the full contour; even when 1/6 of the contour data (green) are artificially removed, in addition to a portion of missing contour data.

Object characterization using Fourier modal approximation

The advantage of our developed Fourier angular series reconstruction for the characterization of attached objects is demonstrated via applying it to the actual images of the HD-AIT screening system for different types of objects. As the first example, we consider the extracted contour of the lower part of the left thigh cross-section, as shown in the second row of Fig. 2. The depression in the image is due to the presence of the affixed silicone rubber slab, with the estimated relative permittivity of ɛ _r = 3 [Reference Von Hippel35]. The missing part is due to the shadowing by the right leg. Assuming the object region is given, based on the projected image of Fig. 1 by the automatic target recognition (ATR) (a software that all TSA scanners are currently equipped with) [36], the seven-term Fourier series approximation is employed to find the best fit by using all the remaining blue contour points; excluding the green depression points, as shown in Fig. 5. The seven-term Fourier series for a body cross-section is computed fully automatically and in real-time. Figure 5(left) shows the calculated fit in φ − ρ representation, and Fig. 5(right) illustrates the transformed fit in x − y representation. Next, the maximum displacement due to the depression (green) normal to the estimated nominal body contour via Fourier series fit (red) is calculated. The dielectric constant of the attached object is approximated as [Reference Gonzalez-Valdes, Alvarez, Martinez-Lorenzo, Las-Heras and Rappaport19]

(3)$$\sqrt {\varepsilon _r} \cong \left({1 + \displaystyle{{{d}^{\prime}} \over d}} \right)/\sqrt p , \;$$

where d′ is the displacement of the depressed contour from the nominal position (3.2 cm), d is the thickness of the slab (3.0 cm), and p is the correction factor for cylindrical cross-sections (1.1). The estimated dielectric constant will be 3.9, which is in good agreement with the 3.0 nominal value.

Fig. 5.

Seven-term Fourier series fit to blue left thigh contour data of the second row of Fig. 2 in φ − ρ representation (left) and x − y representation (right). The result shows reasonable estimation of the full contour; even in the presence of the affixed silicone rubber slab (green), in addition to a portion of missing contour data. The reconstructed contour (red curve) approximates the nominal body surface for object characterization.

The next example is a 20 × 3.5 × 3.5 cm cylindrical bag filled with liquid petroleum jelly attached to the upper left thigh between the legs at z = 0.67 m. The normalized reflectivity of a 2D cross-section is shown in Fig. 6(a). The main contour is extracted using the Canny edge detector filter and the morphological process, as depicted in Fig. 6(b). It is noted that the contour extraction alone cannot estimate the full contour, due to the low-intensity reflectivity. Figure 6(c) illustrates the derived Fourier series reconstruction result. It finds the best seven-term fit through using all the blue contour points, excluding the green depression region. With the displacement of the depressed contour from the nominal position (d′ = 1.9 cm) and the thickness of the slab (d = 3.5 cm), the estimated dielectric constant will be 2.2, using Eq. (3). This is in perfect agreement with the 2.16 nominal value of petroleum jelly's relative permittivity [Reference Von Hippel35].

Fig. 6.

(a) Normalized reflectivity of the 2D cross-section of a cylindrical bag filled with liquid petroleum jelly attached to the upper left thigh between the legs at z = 0.67 m, (b) the reconstructed contour using the Canny edge detector filter and morphological process, and (c) seven-term Fourier series fit to blue contour points. While the upper part of the image shows low-intensity reflectivity due to the presence of the foreign object, the reconstructed contour (red curve) approximates the nominal body surface for the characterization of the attached object.

Our Fourier series reconstruction of nominal body surface can be used not only to determine the electric permittivity of weak dielectric objects (seen as depressions in radar images) but also the thickness of highly conductive materials (seen as protrusions). As an example, an irregular metallic object of interest with the total size of 18 × 12 × 3 cm attached to the front side of the subject's left calf at z = 0.34 m is shown in Fig. 7. The normalized reflectivity of a 2D cross-section is shown in Fig. 7(a). The reflectivity contour, extracted via using the Canny edge detector filter and the morphological process, is shown in Fig. 7(b). The result of seven-term Fourier series fit to blue contour points is depicted in Fig. 7(c). It is noted that about a quarter of the circumference is the object region (green points). Moreover, the upper part of the extracted contour is missing. The developed angular decomposition method is able to approximate the nominal body contour for the entire circumference. Comparing the maximum protrusion distance normal to the reconstructed nominal body contour, the estimated maximum thickness will be 2.7 cm; in excellent agreement with the 3.0 cm nominal value.

Fig. 7.

(a) Normalized reflectivity of the 2D cross-section of an irregular metallic object of interest attached to the front side of the left calf at z = 0.34 m, (b) its reconstructed contour using the Canny edge detector filter and morphological process, and (c) seven-term Fourier series fit to blue contour points. About a quarter of the circumference is the object region (green points), while the upper part of the extracted contour is also missing. The reconstructed contour (red curve) approximates the nominal body surface for the entire circumference, and can be used to estimate the thickness of the attached object.