Common ragweed is an aggressive invasive plant species capable of causing allergic and asthmatic symptoms in humans. As a field weed, it decreases the productivity of crops such as maize (Zea mays L.), sunflowers (Helianthus annuus L.), and beans (Phaseolus vulgaris L.) (Verschwele Reference Verschwele2014). The plant originated in North America, and has since spread to many regions of the world. According to Makra et al. (Reference Makra, Matyasovszky, Hufnagel and Tusnady2015), common ragweed is now found as a neophyte in Europe: in the Pannonian Plain, in southeastern France, in the Po Valley in Italy, in southern and eastern Ukraine, and in southern Russia. In recent years, common ragweed has reached Germany from southeastern Europe. In particular, the eastern parts of the state of Brandenburg have large established populations of common ragweed. As a result, allergic diseases have increased in eastern Germany (Lüdke Reference Lüdke2009). Common ragweed continues to invade the northern regions of Central and Eastern Europe and the British Isles, supported by climate change (Hammaoui-Laguel et al. Reference Hammaoui-Laguel, Vautard, Liu, Solmon, Viovy, Khvorostyanov, Essl, Chuine, Colette, Semenov, Schaffhauser, Storkey, Thibaudon and Epstein2015; Storkey et al. Reference Storkey, Stratonovitch, Chapman, Vidotto and Semenov2014).
Due to its high environmental tolerance, common ragweed grows well under a wide range of soil conditions (Fumanal et al. Reference Fumanal, Girod, Fried, Bretagnolle and Chauvel2008), including both dry and waterlogged conditions (Leiblein Reference Leiblein2008). Its main habitats are agricultural lands, wastelands, roadsides, and riverbanks. Due to the negative effects of ragweed on humans and agriculture, it is important to control its propagation using a monitoring system. The identification of plant species traditionally is performed by means of visual definition of taxonomic characteristics using botanical field guides. This method is in many cases quite complicated and requires a specialist with expert knowledge of botany. The development of an automated system for identifying plants could facilitate or even obviate this task. Progress in the field of information technology, and the widespread use of portable computers and smart devices, would allow such a system to be available to non-experts through new technologies such as crowdsourcing.
However, the accurate identification of common ragweed can become difficult because common ragweed is usually associated with a large number of other plant species, including common lambsquarters (Chenopodium album L.), perennial ryegrass (Lolium perenne L.), buckhorn plantain (Plantago lanceolata L.), field bindweed (Convolvulus arvensis L.), mugwort, prostrate knotweed (Polygonum aviculare L.), ladysthumb (Polygonum persicaria L.), garden sorrel [Rumex acetosa L. ssp. thyrsiflorus (Fingerh.) Hayek], halberdleaf orach (Atriplex patula L.), horseweed [Conyza canadensis (L.) Cronq.], chicory (Cichorium intybus L.), prickly lettuce (Lactuca serriola L.) and Canada thistle [Cirsium arvense (L.) Scop.] (Brandes and Nitzsche Reference Brandes and Nitsche2006; Fumanal et al. Reference Fumanal, Girod, Fried, Bretagnolle and Chauvel2008; Jarić et al. Reference Jarić, Mitrović, Vrbničanin, Karadžić, Đurđević, Kostić, Mačukanović-Jocić, Gajić and Pavlović2011; Silc Reference Silc2009; Sirbu Reference Sirbu2008). Mugwort and common ragweed have very similar ontogenetic and morphological expression (Dammer et al. Reference Dammer, Intress, Beuche, Selbeck and Dworak2012; Kaden and Terentjeva Reference Kaden and Terentjeva1979). Both of the species are typical representatives of ruderal flora. They belong to the same family (Asteraceae), and have deep root systems, and thus can remain green for a long time under dry conditions and can blossom under Central European conditions in the late summer till early autumn.
Modern digital image classification techniques can be used for the differentiation of plant types from photographs. Generally, these classification techniques can be divided into two broad categories: pixel-based and object-based methods. Pixel-based methods are based on the assessment of spectral information that is contained in the pixels (i.e., grey value), whereas object-based methods are based on the analysis of shape, texture, and spectral characteristics of objects, which are represented in a set of pixels (form) in the digital image, such as specific leaf parts.
So far, only pixel-based methods have been investigated to identify common ragweed from digital imagery. Studies indicate that the identification of common ragweed via pixel-based methods from digital images strongly depends on the seasonal period and the species composition of the plant community. For example, Auda et al. (Reference Auda, Blasco, Gastellu-Etchegorry, Marty and Dechamp2002) found that during the period from July to August near Lyon, France, common ragweed had the highest photosynthetic activity out of all the species present, which manifested itself as an increased absorption within the red wavelengths (at approximately 660 nm). Vuković et al. (Reference Vuković, Mesić, Bajić, Krtalić, Gold, Kisić, Bašić, Zgorelec and Sajko2007) used the one-pass clustering method of classification with digital RGB images for the discrimination of common ragweed and soybean in central Croatia. Dammer et al. (Reference Dammer, Intress, Beuche, Selbeck and Dworak2012) found seasonal differences in the spectral reflection of common ragweed and mugwort stems based on a combination of wavelengths of 550 and 650 nm using hyperspectral imaging. Studies performed by Ngom and Gosselin (Reference Ngom and Gosselin2014) in Quebec, Canada showed a potential method for the identification of common ragweed based on satellite data (Worldview 2, Rapid Eye, and SPOT 4 HRVIR) using only the green spectral range (500 to 590 nm). They showed that the plant community and seasonal features had an important influence on classification accuracy. Jung et al. (Reference Jung, Kardevan and Reisinger2006) used hyperspectral imaging in the spectral range (350 to 2500 nm) for the discrimination of common ragweed in sunflower populations. Leiblein-Wild and Tackenberg (Reference Leiblein-Wild and Tackenberg2014) explained the phenotypic variability of common ragweed populations in 38 different regions of Western and Central Europe. The significant differences in the distribution of biomass, as well as the dynamics of growth and flowering, depending on the environmental conditions such as temperature and rainfall, were shown. Ustyuzhanin et al. (Reference Ustyuzhanin, Intreß, Schirrmann, Chochlov and Dammer2015) did experiments on a field in Brandenburg, Germany and showed that the separation of common ragweed from mugwort and rye by thresholding of digital RGB images is hampered by the high intraspecific variations of the spectral reflectance of common ragweed.
Recently, object-based methods have become more popular for the classification of plants from digital imagery. A number of studies have analyzed various features of leaves to categorize plants (by shape, texture, and color) and have used machine learning methods such as support vector machine (SVM) or artificial neural networks to learn a classifier for specific plant groups. The advantage of using leaf parameters for classification is that leaves are available during the entire growing season. By contrast, flowers can only be observed at specific growth stages. Yanikoglu et al. (Reference Yanikoglu, Aptoula and Tirkaz2014) evaluated the leaves of 126 plant species. The best results were achieved with a combination of different form features (58% to 84% classification accuracy). The texture features did not largely influence the results (61% to 81%). Interestingly, the use of color features led to a significant decrease of classification accuracy in this study (49% and 73% for the shape plus color combination, 52% and 75% for the shape plus texture plus color combination). Hernandez-Serna and Jimenez-Segura (Reference Hernandez-Serna and Jimenez-Segura2014) identified 32 species of plants with an accuracy of 93% (90:10 training to test ratio). Fakhri A Nasir et al. (Reference Fakhri, Nasir, Nordin, Rahman, Nashriyah and Rasid Mamat2014) discriminated 5 varieties of mistletoe fig (Ficus deltoidea Jack) using object-based methods with a maximum accuracy of 99%, and Aakif and Khan (Reference Aakif and Khan2015) identified 14 species of fruit trees with a maximum accuracy of 96%. Jin et al. (Reference Jin, Hou, Li and Zhou2015) proposed an original method of classifying plants based on leaf teeth features.
The bag of visual words (BoVW) is an object-based image classification method that has gained interest recently. It was originally developed for document categorization (bag of words) but has already showed its effectiveness for identifying objects from digital images in genetics (Ji et al. Reference Ji, Li, Zhou, Kumar and Ye2009), medicine (Bromuri et al. Reference Bromuri, Zufferey, Hennebert and Schumacher2014; Mukti et al. Reference Mukti, Eswaran and Hashim2015; Wang et al. Reference Wang, Liu, She, Nahavandi and Kouzani2013), remote sensing (Bahmanyar et al. Reference Bahmanyar, Cui and Datcu2015; Sun et al. Reference Sun, Sun, Wang, Li and Li2012; Vetrivel et al. Reference Vetrivel, Gerke, Kerle and Vosselman2016; Yuan and Hu 2015), cell biology (Larsen et al. Reference Larsen, Vestergaard and Larsen2014), palynology (Lozano-Vega et al. Reference Lozano-Vega, Benezeth, Marzani and Boochs2014), and face recognition (Faraki et al. Reference Faraki, Palhang and Sanderson2015; Li et al. Reference Li, Imai and Kaneko2011). Up till now, BoVW has not been applied to the classification of plant types.
Therefore, this study investigated the prospects of BoVW for object-based identification of weeds using the example of categorizing images of common ragweed and mugwort. Special attention was paid to the influence of leaf damage on the classification results. Here, we provide the first steps and general applicability of BoVW for detecting weeds under controlled conditions as the basis for a future online toolbar implementation or crowdsourcing smartphone application.
Materials and Methods
Sampling and Image Acquisition
Common ragweed and mugwort leaf samples were collected between October 6 and October 16, 2015, at various locations in Brandenburg, Germany, including the following areas: harvested field of maize (52.43°N, 13.02°E), building site (52.43°N, 13.05°E), wasteland (52.42°N, 13.01°E), livestock farm area (51.64°N, 14.25°E), and roadside (51.65°N, 14.17°E). At these locations, common ragweed and mugwort were primarily associated with common lambsquarters, halberdleaf orach, redroot pigweed (Amaranthus retroflexus L.), and green foxtail [Setaria viridis (L.) Beauv.].
Leaves were sampled with the whole petioles cut directly at the nodes at different locations on the stems. The chosen plants were in various stages of development (from juvenile to senescent). After sampling, leaves were kept in a fresh state until the image acquisition in the laboratory. For image acquisition, each leaf was placed separately, planar and adaxial on a horizontal table aligned along the same axis. The imaging was performed with a camera positioned at a zenith angle of 0° and a distance to the sample of 0.3 m. The camera used was a Canon EOS 60D (prime lens 25 mm). In total, RGB images of 394 individual leaves were taken for each of the two plant species (Figure 1).
Examples of images of common ragweed (top) and mugwort (bottom) leaves used as training data. Includes intact common ragweed leaves (a–d), a damaged common ragweed leaf (e), intact mugwort leaves (f–i), and a damaged mugwort leaf (j).
Image Classification
Preprocessing of images was kept to a minimum in order to retain image details and strive for more generalization of the image classifier. All images were cut with a few pixels’ space between leaf tips and image edge. The original aspect ratio of the leaves was preserved.
The discrimination between images containing common ragweed and images containing mugwort was done by the object-based method BoVW. Constructing an image classifier with the BoVW method involved several steps (Csurka et al. Reference Csurka, Dance, Fan, Willamowski and Bray2004; Ren Reference Ren2016; Vetrivel et al. Reference Vetrivel, Gerke, Kerle and Vosselman2016; Willamowski et al. Reference Willamowski, Arregui, Csurka, Dance and Fan2004). First, a set of training images containing rich information were used to detect and describe discriminative image features. In the second step, an image vocabulary was constructed by assigning feature descriptors to specific cluster centers (visual words) using a vector quantization algorithm. For each image, the image features were approximated to the nearest visual word and summarized in a feature histogram (the bag of visual words). Visual words are basic, distinctive visual elements describing a larger image set. The binned histograms were used as input feature vectors for multi-class supervised learning in order to train the image classifier.
The detection of interesting image features was done either with the speeded-up robust features detector (SURF) or, more blatantly, by defining a grid over the image with an 8 by 8 pixel resolution. The SURF detector applies a Hessian matrix on an integrated image and uses local maxima in the Hessian determinant space to find interesting point features in the image that are highly discriminant, such as corners, edges, junctions, or blobs (Oyallon and Rabin Reference Oyallon and Rabin2015).
The feature description then gathers information in the neighborhood surrounding each feature point. The SURF descriptor was used in this study. The SURF descriptor describes the distribution of the intensity values in the neighborhood in terms of gradients. These gradient vectors are computed using Haar wavelet transforms within subregions aligned along a regular and oriented square grid around the feature points. The goal is to obtain a geometric invariant description, which is also robust to noise, illumination, or contrast change, while it is at the same time computationally effective (Bay et al. Reference Bay, Ess, Tuytelaars and Van Gool2008; Oyallon and Rabin Reference Oyallon and Rabin2015).
In order to build the “vocabulary”, the feature description vectors were clustered with k-means. The objective function of the k-means algorithm is to minimize the sum of squared distances between the description vectors and arbitrary chosen cluster centers within the vector space. It is mandatory to select the number of clusters (k) prior to clustering. For this study, we set k to 500. Each cluster center represented a unique visual word. All visual words were indexed, and for each image, the extracted feature descriptions were related to its nearest visual word (cluster center). The local histograms of the feature description occurrences indexed to its visual word are called the BOVW.
To train the image classifier, the BoVW histograms were encoded as vectors and used as training feature vectors to teach a SVM that knows whether each vector is related to a common ragweed or a mugwort image. The SVM classifier has been used in many studies of object recognition with BoVW (Guo et al. Reference Guo, Fukatsu and Ninomiya2015). The SVM essentially finds an optimal hyperplane within an n-dimensional feature space using a kernel-based learning algorithm that optimally finds a border between the points distributed in the feature space. We implemented a two-class problem here (common ragweed or mugwort) with a SVM using a linear kernel and the iterative single data algorithm solver. All analysis was conducted with Matlab® 2015b using the Computer Vision ToolboxTM (Mathworks®, Natick, MA).
Training and Testing
The images were split into different sets of training and test data with the ratios 50%:50%, 60%:40%, 70%:30%, 80%:20%, and 90%:10%, to evaluate the accuracy of the classifier. Data were split equally within each category (common ragweed or mugwort). The training data were used to compute the BoVW histograms and to train the SVM classifier. In one run we excluded all the images containing damaged leaves, while in a second run we included them in order to check the robustness of the classifier concerning anomalies present in the leaf structures.
Results and Discussion
Figure 1 gives a representation of the diversity we observed among the leaf images of common ragweed and mugwort. In particular, it shows the variability of parameters such as leaf shape, degree of dissection, color, and leaf rachis curvature. Leaves of common ragweed were ovate or broadly ovate in shape and were pinnatisect or pinnately divided. Leaves of mugwort were ovate, broadly ovate, rounded to elliptic, pinnatisect, or pinnately lobed. The color of common ragweed leaves varied in this study from different shades of green to yellowish and reddish brown. The color of mugwort varied in different shades of green only.
In many images, individual leaf blades of common ragweed and mugwort overlapped with each other (Figure 1a,b,e,h,i). Another important feature among the images was the morphological damage observed in about 9% of the sampled leaves (Figure 1e,j).
In Figure 2, the feature points selected by SURF in images with intact and damaged leaves are illustrated. The circles around the keypoints denote the neighborhoods in which the SURF descriptors were calculated. As can be seen from this example, detected keypoints corresponded mostly to the edges of the leaves, especially to the tips.
Keypoints detected using the speeded-up robust features (SURF) method in a selection of images. The circle centers represent the keypoints, and within the limits of the circles the neighborhoods of the SURF descriptors are defined. From left to right: an intact common ragweed leaf, a damaged common ragweed leaf, an intact mugwort leaf, and a damaged mugwort leaf.
Figure 3 shows examples of the visual vocabularies of a mugwort and a common ragweed image. The horizontal axis of the BoVW histogram is a sequence of the cluster centers representing the different visual words found by k-means (k=500). The vertical axis corresponds to the frequency of each detected feature point related to its nearest visual word.
Examples of visual vocabularies of common ragweed (top) and mugwort (bottom) images used to create the feature vectors for classification in a support vector machine.
Results of this study showed that the BoVW technique allows discrimination of images of common ragweed and mugwort leaves with high accuracy. Table 1 provides the results of classification depending on the keypoint detection method and the size of the training data set. In this case, all images, including the images of damaged leaves, were included. Even when using only 50% of the images as training data, we still obtained full differentiation of common ragweed and mugwort plants with 100% classification accuracy. Using the grid sampling method resulted in slightly less recognition accuracy (98% to 99%) compared with the SURF detection method, but the difference was not statistically significant (P>0.1, McNemar’s test) (Dietterich Reference Dietterich1998).
Percentage of correctly recognized leaves using either the speeded-up robust features (SURF) or the grid sampling method of keypoint selection, and a variety of training data set sizes.
* No significant difference between the accuracy of the SURF and grid methods (McNemar’s test, null hypothesis was not rejected at P>0.1)
However, the keypoint selection method had a strong effect on the processing time of classification (Table 2). The total time required for learning the image classifier with SURF detection was 17 to 31 times shorter than the time required with grid sampling. Once a model has been found, the classification of a new image using SURF needs only a small fraction of the time compared to learning the model. The time for prediction is smaller because the image classifier does not need to build the vocabulary again (vector quantization or clustering is omitted). The cluster centers in the BoVW model contain the vocabulary of the images used in the models, i.e., the descriptions of unique image features. A higher number of clusters makes it easier to differentiate the images with the BoVW model. We analyzed the time for learning and predicting the BoVW model with varying numbers of clusters. Our results showed that the number of clusters is not important when predicting new images. However, there is a decrease in classification accuracy when using cluster numbers lower than 150 to 200. In contrast, changing the number of visual words, i.e., the number of clusters, did not change the prediction speed of classifying a new image drastically (Figure 4). However, for proper classification of a new image, it is still mandatory to extract the keypoint descriptors and compare them with the already-built vocabulary to generate the BoVW histogram, which is then used in the SVM model to decide which category it belongs in: common ragweed or mugwort. Therefore, there is a difference in prediction speed for grid sampling and SURF detection (Table 2; Figure 4). SURF is 6 to 7 times faster than grid sampling. Because accuracy is not significantly different with grid sampling, SURF detection should be generally preferred. With only 0.14 s image classification speed, about 4 frames per second would be reached in a future online implementation of the model.
Classification accuracy and time required for learning and classification processes (for one image) depending on number of clusters.
Total model learning time and time required to classify a new image, based on 788 images using a 3.33-GHz processor, using either the speeded-up robust features (SURF) or the grid sampling method of keypoint selection, and a variety of training data set sizes.
* Average time for classifying one new image, based on 394 new images
Table 3 shows the results of the image classification depending on the presence or absence of images with morphologically damaged leaves in the composition of the training and test data. For this comparison, we used the grid sampling approach. Omitting the images with damaged leaves from the training and test data had no significant impact on the recognition results. Even using only the intact leaves for training and the damaged leaves for testing yielded a classification accuracy of 97% and higher.
Percentage of correctly recognized leaves depending on leaf category used for training of the classifier and the training data set size.
With the advent of mobile phones and new online technologies such as crowdsourcing, there are new possibilities for combating invasive plant species. These technologies make it possible to automatically identify plants from digital images with high accuracy. In our study, we provide the first use of BoVW for detecting common ragweed or weeds in general. At this stage, it can be argued that the method works, if the leaves are photographed separately, planar and adaxial, on a homogeneous background with a fixed, nadir camera position. Additional image preparation is not required. We used a quite practical resolution setting: the images were shot at a resolution of 72 dpi. The length of the leaves on the images ranged from 500 to 1600 pixels. We also showed that our model was robust through the inclusion of damaged leaves and partially occluded hidden leaves.
Despite being disturbed by damaged leaves or overlapping leaf blades, the accuracy of the image classification between common ragweed and mugwort leaves was around 100%. In contrast, no prospects for discrimination of leaf parts using pixel-based algorithms based on hyperspectral images has been previously reported for these plant species (Dammer et al. Reference Dammer, Intress, Beuche, Selbeck and Dworak2012). Previous studies have been able to discriminate between plants by analyzing the stems using the wavelength ratio of 650:550 nm during fruit development. However, the practical use of this approach is severely limited. Our results showed that the object-based method BoVW is not influenced by the stage of plant development. Furthermore, SURF detection provided higher accuracy than grid sampling as a keypoint detection method because SURF is directed to image features that constrain differences in the gradient of the grey values through its use of the Hessian matrix window. This in turn leads to feature detection that is to some extent scale-invariant and strongly object-related.
As the next step, the effectiveness of this method should be tested using a larger number of weed species. Previous studies by different authors revealed that the more plant species are classified, the fewer plants will be properly detected. Thus, the method of Yanikoglu et al. (Reference Yanikoglu, Aptoula and Tirkaz2014), which evaluated the leaves of 126 plants species, allowed achieving 84% accuracy as the maximum result. Hernandez-Serna and Jimenez-Segura (Reference Hernandez-Serna and Jimenez-Segura2014) evaluated 32 plant species and achieved an accuracy of 93%. Aakif and Khan (Reference Aakif and Khan2015) classified 14 plant species with 96% accuracy, and Fakhri A Nasir et al. (Reference Fakhri, Nasir, Nordin, Rahman, Nashriyah and Rasid Mamat2014) could discriminate 5 plant varieties with 99% accuracy.
Finally, the study should be extended to employ the BoVW technique for the identification of weed species from digital images taken directly in the field. The long-term goal would be for the model to identify if there is common ragweed somewhere in a photograph taken in any environment with a smartphone. The next steps towards this aim are that the model should allow different backgrounds, such as soil or grassland; the plant leaves should not be fixed but still intact on the common ragweed plant without any preparations; and the camera perspective and lighting conditions should be unconstrained. Subsequently, the BoVW technique should be applied under fully uncontrolled, ambient conditions. Until now, the results of field studies were fairly limited, and there were no studies related to common ragweed identification. Swain et al. (Reference Swain, Nørremark, Jørgensen, Midtiby and Green2011) proposed a new method for weed identification based on the automated active shape matching system. Using this technique, the authors obtained 90% accuracy in the identification of nightshade (Solanaceae) plants. Hemming and Rath (Reference Hemming and Rath2001) evaluated eight different shape features and three color parameters to separate the leaves of cabbage (Brassica oleracea L.) and carrot (Daucus carota L. var. sativus Hoffm.) crops from weeds in field images. Their average identification accuracy was 88% for cabbage and 72% for carrot. Weis and Gerhards (Reference Weis and Gerhards2007) used different shape parameters to detect weeds in images taken in a barley (Hordeum vulgare L.) field, with a classification accuracy of 98%. McCarthy et al. (Reference McCarthy, Rees and Baillie2012) presented research on the discrimination of three weed species, wild oats (Avena fatua L.), fleabane, and sowthistle (both Asteraceae), based on leaf shape and color. The authors were able to achieve 100% separation only between wild oats and fleabane.
Acknowledgements
The authors wish to thank the staff of the Department of Engineering for Crop Production (ATB), the agricultural cooperative Drebkau e.G., and especially Mrs. Vogt, for their support in carrying out the experiments.
