Introduction
The possibility of monitoring variability in the density of canopies is of great interest for accurately quantifying local biomass, particularly in the case of vineyards, where knowledge of variability can be not only a useful mean to evaluate the health condition of the vines and of the grapes, but also an important input to allow variable management practices (Monsó et al., Reference Monsó, Arnó and Martínez-Casasnovas2013; Mathews and Jensen, Reference Mathews and Jensen2013) or medium and long term efficiency simulations (Pezzuolo et al., Reference Pezzuolo, Basso, Marinello and Sartori2014).
By way of example, pruning (Liu et al., Reference Liu, Kantor, De la Torre and Zheng2012) or variable rate application (Rinaldi et al., Reference Rinaldi, Llorens and Gil2013) can benefit from real time three dimensional reconstruction of canopy. In the first case automatic trimming or pruning can be based on actual positioning and direction of branches; in the second case the sprayer could modify instantaneously the working parameters based on local leaves density.
Many attempts have been made in the recent past in order to allow three-dimensional reconstruction of plants and specifically of vineyard canopies, also benefitting from the constant miniaturization of sensors and increase in data processing speed (Marinello et al., Reference Marinello, Schiavuta, Cavalli, Pezzuolo, Carmignato and Savio2014; Pajares et al., Reference Pajares, Peruzzi and Gonzalez-de-Santos2013). Methods can be mainly referred to three methods: ultrasonic sensors, LIDAR and structure from motion (Su and Zhang, Reference Su and Zhang2010).
Even if ultrasonic technology is a possible solution which can provide a convoluted profile of canopy, its performance is unsatisfactory in the case of very thin objects (as in the case of branches with no leaves) or in the case of very irregular surfaces (as in the case of branches sticking out from the overall canopy).It is also very sensitive to dust and humidity; additionally interference can be produced by implementation of multiple sensors. On the other hand it is cheap and is already at its industrial level: indeed it is actually implemented in some recent sprayer machines, to regulate flow of different nozzles.
A different approach is that based on LIDAR technology. It relies on implementation of a light source, using the delay for a laser signal to return, to estimate relative distance. Researchers have demonstrated its effectiveness in several fields, including vineyard application (Rinaldi et al., Reference Rinaldi, Llorens and Gil2013; Del-Moral-Martínez et al., Reference Del-Moral-Martínez, Rosell-Polo, Company, Sanz, Escolà, Masip, Martínez-Casasnovas and Arnó2016); however it is mainly two dimensional and relatively expensive.
A third approach is that based on structure from motion. This is a technique based on principles of photogrammetry, as it takes advantage of a set of images captured from adjacent position to allow 3D reconstruction. Such technique is low cost and examples in vineyard application are already available (Mathews and Jensen, Reference Mathews and Jensen2013); however reasonably high resolution imaging has necessarily to be supported by robust algorithms and powerful processors.
To avoid too expensive technique, the present work implements a depth sensing camera and specifically a Microsoft Kinect™ RGB-depth camera. In the last decade such solution has had a large diffusion mainly because of its introduction as a video game interface.
Such solution is not new in the agricultural field: relevant applications have been already demonstrated in the case of soil characterization (Marinello et al., Reference Marinello, Pezzuolo, Gasparini, Arvidsson and Sartori2015a) as well as for ergonomic analysis of agricultural machines (Marinello et al., Reference Marinello, Pezzuolo, Simonetti, Grigolato, Boscaro, Mologni, Gasparini, Cavalli and Sartori2015b). In this work, Kinect was statically positioned into a vineyard and successfully implemented to allow collection of three-dimensional information from vines canopies. A schematic representation of a possible layout is reported in Figure 1, where the Kinect sensor is mounted in front of the tractor and a feedback is generated to allow regulation of working parameters.
Schematic representation of a three-dimensional sensor unit, positioned in front of the tractor for a feed-back system, allowing variable rate distribution with the implement.
Materials and methods
Depth-camera sensing technology
The present work was implemented with a Kinect (version 1), a non-contact instrument based on structured light depth-camera sensing technology (RGB-D). Such instrument uses an infra-red projector which shines light onto the scene: light reflected from the scene is collected by an infrared depth sensor and is used to produce a three-dimensional reconstruction of the scene (Marinello et al., Reference Marinello, Pezzuolo, Gasparini, Arvidsson and Sartori2015a).
Despite being a low cost instrument (<100 €) developed as an interface for video-games, the Kinect provides interesting performances, as demonstrated in previous works (Marinello et al., Reference Marinello, Pezzuolo, Gasparini, Arvidsson and Sartori2015a).
To help understanding, x-y-z coordinate axes can be defined, with the z axis perpendicular to the sensor and x-y plane parallel to the canopy (Figures 2 and 3). Main performance indicators in terms of lateral resolution (i.e. the minimum resolvable separation between two pixels in the x-y plane), vertical resolution (i.e. the minimum resolvable depth in z direction) and maximum detectable slope (i.e. the maximum resolvable angle of a plane tilted relatively to x-y plane) are reported in Table 1. During experiments, the sensor was statically positioned at a 1.6 m height and at a 2 m distance from the center of the vine rows (Figure 2): at such distance lateral and vertical resolutions can be kept far below 10 mm and background noise estimated as root mean square signal (RMS) on a nominal flat surface below 2 mm.
The Kinect sensor (left) positioned for three dimensional reconstruction of vine canopy (right): the grayscale renders the achieved depth image, where darker or lighter regions are associated respectively to depressed or elevated portions on z axis.
Geometrical approximation for estimation of volume from hand measurements (left) and three dimensional rendering of a vineyard row portion elaborated from a Kinect digital measurement and implemented for volume estimation (right).
Main Kinect (version 1) performances as a function of average distance of the sensor from the target (RMS=root mean square), according to previous published work (Marinello et al., Reference Marinello, Pezzuolo, Gasparini and Sartori2013)
Experimental site and measurements
The present study was carried out in a private farm in north-eastern Italy in a typical Po Valley vineyard (45.283119N, 11.836040 E). Two grapevine varieties were available (Glera and Merlot) with a total extension of 13000 m2, both planted in 2010 and managed through a Sylvoz pruning system. Grape branches typically lay at a 1.2–1.3 m height. Vine-stocks were planted with a 2.8 m inter-row and 1.3 m intra-row spacing (corresponding to 2750 plants per hectare). The vineyard is north-south oriented, thus receiving an optimal sun radiation during daylight.
During the measurement campaign, eight different vines from the two varieties (4 Glera and 4 Merlot) were monitored, including:
-
– three-dimensional reconstruction of vines canopy through implementation of the Kinect sensor;
-
– counting of the number of leaves per vine (hand measurement)
-
– estimation of the area of the leaves through implementation of a digital camera and a reference grid (hand measurement);
-
– measurement of canopy height (hand measurement);
-
– measurement of canopy width at three different levels (hand measurement).
In order not to alter the health condition of the plants, no leaves were removed from the vines undergoing 3D Kinect reconstruction.
Measurements were repeated on 18 different dates, from 4 April 2016 to 27 June 2016, when the main vegetation evolution can be observed.
A total of 1728 widths and 576 heights hand measurement on vines, 1440 measured leaves and 432 3D Kinect reconstructions were carried out.
Reference parameters
In order to evaluate the applicability of the Kinect sensor for the characterization of canopy, selected Merlot and Glera vines were characterized in terms of average volume.
Volume was estimated by means of both hand measurements and Kinect.
In the first case the height and the width of the canopy were collected with a scale. Widths at three different heights were measured, namely w 1 , w 2 and w 3 as depicted in Figure 3, considering as a reference surface the virtually flat plane passing through the trellising supporting the plants. The three widths were measured on parallel positions at a relative distance of about 33 cm, in order to have 4 partial volume values over 1 m length (L). Canopy volume from hand measurements V hm was approximated by means of equation (1):
$$V_{{hm}} =L \cdot {{\bar{w}_{1} \cdot \bar{h}_{1} {\plus}\bar{w}_{1} \cdot \bar{h}_{2} {\plus}\bar{w}_{2} \cdot \bar{h}_{2} {\plus}\bar{w}_{2} \cdot \bar{h}_{3} {\plus}\bar{w}_{3} \cdot \bar{h}_{3} {\plus}\bar{w}_{3} \cdot \bar{h}_{4} } \over 2}$$
Signs over w and h symbols represent the operation ‘mean value’ between data collected on different positions.
In the second case, the same canopy portion was captured by means of the three dimensional sensor positioned at 2 m from the plants rows (Figure 2), allowing estimation of canopy volume V k , according to (2):
$$V_{k} =\mathop{\int} {z\,dxdy} $$
where z represents the function describing the vineyard convoluted surface averaged on three measurements; again the zero level is defined in correspondence of the trellising supporting the plants.
Results and discussion
Field experiments were started in the first week of April, when the first leaves appeared in Merlot and Glera plants, but actual three-dimensional measurements were started only on 18 April, when leaves had a sufficient size to be correctly detected by the Kinect sensor. In the following weeks both the varieties exhibited a rapid growth, but with a higher canopy vigor in the case of Glera vines. Analyses were concluded on 27 June: such 3 months period not only allows thorough monitoring of the evolution of the canopy, but is also the most interesting from the phytosanitary point of view. Volume measurements from Kinect sensor V k were correlated to hand volume measurements V hm . Results are reported in Figure 4a. It can be noticed how Kinect estimation is slightly higher with respect to hand measurements, by 17.5% in the case of Merlot and by 7.8% in the case of Glera. Such difference can be explained considering that digital reconstruction relies on a higher number of points (pixels), and thus can provide a more comprehensive reconstruction of vines, including isolated or protruding branches. A good correlation can be recognized in both cases, with a coefficient of determination R2=0.922 in the case of Merlot and R2=0.943 in the case of Glera. The variability and relative deviations, tend to increase as the vigor of the canopy increases, however with an acceptable combined standard error for both cultivars ε=0.038 m3/m.
(a) Correlation between hand measured volume V hm and Kinect estimation (V k ). (b) Correlation between the number of leaves and Kinect volume estimation (V k ).
As already mentioned, the interest on the canopy volume is due to the fact that it can be an indirect indicator of plant vigor. Thus, the volume of the canopy V k was correlated to the number of leaves. Results, graphically reported in Figure 4b, give evidence of a certain correlation between the two variables, with coefficients of determination R2=0.743 and R2=0.758 in the cases of Merlot and Glera respectively, and a higher combined standard error for both cultivars ε=0.090 m3/m.
If by one side the number of leaves can help determining canopy growth, the management of the vineyard can be better supported by information on the overall leaves area. Indeed the leaf area index (LAI) is probably the most common parameter related to the foliage used in viticulture (Arnó et al., Reference Arnó, Escolà, Vallès, Llorens, Sanz, Masip, Palacín and Rosell-Polo2013).
Thus, the correlation between the whole data set and the reference leaf area indexes was also considered. In this case the correlation between the two variables is still high (R2=0.767). Such result is reasonable considering that the overall canopy volume is the result not only of plant development and of the number of leaves but also of leaves size. Indeed larger leaves occupy higher volumes and have a higher weight causing larger bending of the branches, eventually producing an apparent expansion of the plant. The results are in good agreement with those published in a recent work based on LIDAR technology (Arnó et al., Reference Arnó, Escolà, Vallès, Llorens, Sanz, Masip, Palacín and Rosell-Polo2013), where a linear regression analysis highlighted a similarly good correlation between LAI and canopy volume in the case of analysis of both the total width (R2=0.81) and half the width of the row (R2=0.71).
Figure 5 shows how the correlation between volume and leaf area is relatively higher in the first months; then variability tends to increase due to uncontrollable deviations affecting leaves size and density such as local meteorological phenomena, localized damages, disease. The coefficient of determination (R2=0.760) is slightly higher than that estimated after hand measurements (R2=0.690): this can be ascribable to the higher and more robust number of points considered in the case of digital canopy reconstruction. Furthermore, the relatively low standard error (ε=0.078 m²/m) supports the possibility of implementing Kinect instrument for fast on the go monitoring of plants leaf area index, with an acceptable degree of approximation.
Leaf area index reported as a function of the estimated Kinect volume (V k ).
Conclusions
In this paper a Kinect three-dimensional depth-camera is proposed as a solution for fast characterization of three-dimensional structure of grapevine canopy. Experiments have shown the very high correlation between digital analysis and hand measurements results for two different grapevines varieties: Glera and Merlot. In both cases, the volume can be related to the number of leaves and to their area, with a coefficient of determination R2=0.76 in the latter case. The sensor cannot be applied in the early stages when leaves are too small compared to resolution, negatively affecting the three-dimensional reconstruction process. Also, at present no algorithms have been implemented to allow recognition of diseased leaves or plants. Additionally, attention has to be paid to the sun light conditions, in order to avoid excessive radiation which could blind the 3D detector. Apart from these limitations, the Kinect sensor can be a useful solution for on the go monitoring of leaf area variability.
Aknowledgements
The study is supported by a grant from the University of Padova (code CPDA143174), Italy. We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Tesla K40 GPU used within the present research.
