Introduction
Motivated by environmental concern, especially relating to protection of Danish groundwater resources, political action plans aiming for reducing dependency and use of pesticides have been executed in Denmark since the mid 1980ies. These plans included a wide spectrum of activities, where design and implementation of ‘decision support systems’ (DSS) were included to support rational and efficient management of weeds, pests and diseases in conventional agriculture. While maintaining requirements for agronomical robustness, options for significant reduction in pesticide use were strived for.
Since 1991, commercialized DSS for Integrated Weed Management (IWM) have been available in Denmark. Two commercialized DSS are currently available: 1) ‘Crop Protection Online’ (CPO), which was released in 2001, and which is distributed by SEGES in Denmark and 2) ‘IPMwise’ (IPMW), which was released in 2016, and which is distributed by IPM Consult. While CPO may be perceived as a technical 3rd generation tool, IPMW may be perceived as technical 4th generation tool.
Both DSS originate from a basic idea of adjusting measures for IWM to conditions on farm and field levels. Without jeopardizing agronomic requirements for robustness in IWM, these DSS were designed to systematically exploit that: 1) weeds are inhomogeneous distributed in time and space, 2) some weeds can be ignored, while other weeds need efficient control, 3) some herbicides can control some weeds sufficiently by use of significantly reduced dose rates of a single herbicide, while other weeds require two or more herbicides in combination in relatively high dose rates.
After >25 years of development of methods, which have been integrated in these DSS, and validated in field experiments, these methods currently constitute a professionally recognized point of reference on IWM in Denmark among important stakeholders, which include farmers, crop advisors, agrochemical companies, schools, universities, regulatory bodies, NGOs, etc. (Jørgensen et al., Reference Jørgensen, Noe, Langvad, Jensen, Orum and Rydahl2007; Been et al., 2009).
These methods require use of field reports on actual weed infestations, i.e. combinations of weed species and classes of density growth stages. Upon entering, the DSS will: 1) quantify needs for control on a weed species level, 2) identify single herbicides, which can meet identified needs for control, 3) select and optimize (for cost or various index) suitable 2–4 way herbicide tank-mixtures and 4) present options for treatment (Rydahl, Reference Rydahl2004).
Due to lack of innovation in the agrochemical industry, still stricter regulation on pesticides and increasing problems with pesticide resistance, Directive 2009/128/EC, Annex III (8 general principles), regarding ‘Integrated Pest Management’ (IPM) is already a real necessity to continue conventional crop production in several combinations of countries and crops. After integration in IPMW of: 1) measures to prevent/delay additional development of herbicide resistance and 2) measures for non-chemical weed control, IPMW distinctively addresses 7 of these general IPM (IWM) principles.
Due to generic qualities in agro-biological modelling and IT basis, IPMW is currently being customized for professional use also in countries abroad, mainly Norway, Germany and Spain, and start-up activities are in progress in additional EU countries. In all countries involved so far, DSS has demonstrated sufficiently robust IWM, and potentials for reducing the use of herbicides in the range of 20–40% as compared to local ‘best practice’ programs Reduction (Rydahl, Reference Rydahl2004; Tørresen et al., Reference Tørresen, Netland and Rydahl2004; Sønderskov et al., Reference Sønderskov, Kudsk, Mathiassen, Bøjer and Rydahl2014; Sønderskov et al., Reference Sønderskov, Fritzsche, de Mol, Gerowitt, Goltermann, Kierzek, Krawczyk, Bøjer and Rydahl2015; Montull, Reference Montull2016). However, in all countries involved, a common recognized major bottleneck for a wider exploitation of reduction potentials has been identified in terms of reluctance among farmers to conduct manual field inspections (Jørgensen et al., Reference Jørgensen, Noe, Langvad, Jensen, Orum and Rydahl2007). As the total capacity for manual weed inspection is relatively low, most herbicide applications are still relying on recommendations from crop advisors on the regional level. In order to provide sufficient control in all fields, such ‘best practice’ recommendations need to be rather conservative, which explains the potential of the DSS.
The project ‘RoboWeedSupport’ (RWS) was designed to overcome farmer’s reluctance against manual weed inspections by introducing automatic detection of weeds on a species level from pictures.
Materials and methods
‘RoboWeedSupport’ (RWS) is also the name of an online tool (roboweedsupport.com), which has been designed for upload and analyses of digital pictures of weeds, taken in farmer’s fields by use of various techniques, e.g. handheld-, drone born or ATV born cameras (Laursen et al., Reference Laursen, Jorgensen, Dyrmann and Poulsen2017).
Pictures from one field were uploaded to RWS, where elements, which are detected as weeds, are automatically marked-up (Figure 1). These elements were subsequently classified (‘annotated’) manually by crop advisors, who determined weed species and accompanying classes of weed growth stages (Figure 2). Based on these, sum stats were made automatically.
RWS showing elements in a picture identified automatically as weeds.
Close up of a single plant.
Sum stats from analyses were transferred to CPO / IPMW / human advisor, as preferred by the farmer. Here, needs for control was quantified, and accompanying measures for control were suggested. Results were sent to the farmer, who uploaded the pictures. In addition, results from annotations were transferred to a convolutional neural network, which in this way was trained gradually to enable automatic detection of weeds in future pictures (Dyrmann & Jørgensen, Reference Dyrmann and Jørgensen2015; Dyrmann et al., Reference Dyrmann, Karstoft and Midtiby2016a; Dyrmann et al., Reference Dyrmann, Mortensen, Midtiby and Jørgensen2016b). In case some weeds belonging to a certain group of weeds, could not be determined on a species level but instead on group levels, e.g. ‘veronica species’, ‘crucifer species’ or ‘grassy species’, worst-case representatives from these groups according to the design of DSS, were used as input to the DSS. When submitting this paper, automatic integration between RWS and DSS was not yet fully operational. In calculations with IPMW, the following general settings were used: 1) the identified group names ‘fine grasses‘ and ‘coarse grasses’, respectively, were reported to the DSS as Apera spica-venti and Lolium perenne, 2) use of the following classes of weed density (pl./m2): Few pl./field,
$$\raise.5ex\hbox{$\scriptstyle 1$}\kern-.1em/ \kern-.15em\lower.25ex\hbox{$\scriptstyle 2$} $$
–1, 2–10, 11–50, 51–150, 151–300, 301–600, >600, 3) use of IPMW requirements for efficacy targets (Rydahl & Bojer, Reference Rydahl and Bøjer2016), 4) optimization and sorting of recommended options for treatment according to cost, 5) no known already herbicide resistant biotypes of weeds and 6) no selected measures to protect against herbicide resistance development.
Results
As >3,500 pictures were taken during the autumn of 2016 analyses have not yet been finalized, why only provisional results are presented here.
In one winter wheat field of 2.72 ha, an ATV mounted with high-speed camera and flash, was used to take pictures, using an intended grid size of 10 m. This resulted in 364 pictures covering at total photographed area of: 364×0.25 m2 =91 m2. The driving speed varied from a few km/h up to 50 km/h, and it took about 10 minutes to collect the pictures. It subsequently took about 5 minutes to automatically analyze these pictures by convolutional neural networks, which resulted into 5 min. x 60 sec./min / 364 pictures=0.82 sec./picture.
In Table 1, summary results of 20 weed species detected in this field are presented. These results were transferred to IPMW as illustrated in Figure 3. Only two of the reported species (Brassica napus L. and Apera spica-venti) occurred in classes of density, which released efficacy targets >0.
IPMwise showing the entered field report from Table 1.
Manually and automatically marked-up and detected weeds in 364 pictures of 0.25 m2 (91 m2 in total) which were taken in a 2.72 ha field of winter wheat in Denmark October 2016.
Notes:
1) To adjust for ‘Undetermined weeds’, calculated densities per m2 have been multiplied in the DSS by 2951 / 1329=2.2238.
2) The group ‘fine grasses’ has been classified for DSS as Apera spica-venti.
3) The group ‘coarse grasses’ has been classified for DSS as Lolium perenne.
In Figure 4, the three cheapest recommendations for control in this field are shown. The cheapest option was a two-way tank-mixture of significantly reduced dose rates of Express (500 g/kg tribenuron-methyl) and Lexus (500 g/kg flupyrsulfuron-methyl) plus a non-ionic surfactant at a total cost of 50.53 DKK/ha, equal to 6.75 Euro/ha and TFI was summarized to 0.33. The expected efficacy of this mixture is also presented, including also efficacy targets as determined by IPMW (IPM) or eventual overruling of these targets by the user. 38 alternative 1–3 way herbicide compositions were suggested by IPMW for this field, where the most expensive costed 114.41 Euro/ha.
IPMwise showing the top-3 cheapest options for control, where only 2 species, Brassica napus L. and Apera spica-venti occurred in densities requiring control.
In the RWS project as a whole, about 10,000 pictures taken by hand-held or drone carried cameras have been collected from a combination historical databases and new pictures from the project. In total, about 90,000 weed elements have been collected, of which about 25,000 elements have yet been annotated manually (Dyrmann & Jørgensen, Reference Dyrmann, Jørgensen and Midtiby2017). Currently, a total of >30 broadleaved weed species have now been annotated by >100 weed elements. This level of manual annotation is presently considered as a minimum level to enable automatic determination of weeds in future pictures (Dyrman & Jørgensen, Reference Dyrmann, Jørgensen and Midtiby2017)
Discussion
As indicated by results in Table 1, only about 1/3-1/2 of weed elements were determined manually/automatically. When weeds are overlapping with other plants, systematic under-estimations of weed density may be expected. However, according to design of DSS (CPO or IPMW) (Rydahl, Reference Rydahl2004; Rydahl and Bøjer, 2016), only some such imperfections are important, while others may be automatically corrected for or just ignored, without jeopardizing requirements for agronomic robustness, according to the DSS design.
Important is, however, not to completely overlook species, which cause serious economic losses, and/or which require special control measures. Less important is to provide precise estimates of density, as classes of density in context of the DSS (section Materials and methods) are relatively broad. Even if the density is classified one class wrong, only relatively small effects will occur in the output from the DSS. In Table 1, measured densities were adjusted with a factor to compensate for non-classified elements, assuming that these had the same distribution among species, as the classified elements. After that the weighted densities were converted to density classes used in the DSS.
In the project ‘RoboWeedMaPS’ (2017–2020), commercial product chains will be designed and made operational for real-time and site-specific: acquisition of pictures by high-speed cameras mounted on tractor with sprayer ->automatic determination of weeds, densities and weed growth stages ->analyses by DSS of needs for control and accompanying, optimized treatments ->input to injection sprayer. This concept has many implications for logistics, technical integrations, speed of processes, etc., which shall be solved during the project. In this context, the provisional result of <1 sec. for analyses of 1 picture seems prosperous.
Conclusion
In order to exploit well-known potentials of 20–40% reducing the input of herbicides in field crops, by exploiting also well-known interactions between crops, weeds, control measures and additional conditions on farm and field levels as integrated in DSS for IWM, the project RoboWeedSupport addresses the challenge of avoiding needs for manual field inspections for weeds, which is a dominating and well-known bottleneck for a wider exploitation of these potentials.
This is done by introducing analysis of digital pictures to determine weed infestations on a field level in terms of connected datasets on weed species and classes of weed density and weed growth stages. Results were used to: 1) send a report on weed infestation and suggestion for control by use of DSS (CPO / IPMW) or as recommended by a crop advisor to the farmer 2) train convolutional neural networks to gradually enable and improve automatic determination of weeds.
Provisional results from automatic analyses indicate that sufficiently robust and fast automatic detections can be made, which very soon can be made operational for use on a field level. During the next 4 years efforts will be made in the successor project ‘RoboWeedMaPS’ to enable monitoring, processing and application of measures for IWM in real-time and site-specifically for commercial use.
Acknowledgements
EU Framework 7 for sponsoring the project ‘PURE’ (2010–2014), in which the DSS proof-of-concept developed in ‘ENDURE’ was implemented online, customized and field validated for IWM in maize in Germany, Slovenia and Italy.
EU Era-net C-IPM for sponsoring the project ‘DSS-IWM’ (2016–2018), in which the DSS ‘IPMwise’ shall be customized and commercialized for use in maize and winter wheat in Denmark, Germany and Spain.
Danish Agrifish Agency for sponsoring the project ‘RoboWeedSupport’ (2014–2016), in which automatic determination of weeds from pictures was designed and enabled, where technical integration with DSS for IWM was implemented and which has generic qualities suitable for scaling up and out.
Danish Innovation Fund for sponsoring the project ‘RoboWeedMaPS’ (2017–2020), in which results from RoboWeedSupport will elaborated for site-specific IWM, in 2 timesteps (application map, treatment) and in 1 time-step (real-time and site-specific treatment).
