Introduction
Consumers are increasingly concerned with how agricultural practices impact the environmentReference Potter1. This concern has translated into the growth of a variety of programs including the United States Department of Agriculture (USDA's) National Organic Program, independently certified eco-label strategies, and integrated pest management (IPM) programs. Traditionally, IPM can be defined as a systems approach that employs biological, chemical and other agricultural practices to minimize environmental, health and economic risks from agricultural pestsReference Cuperus, Mulder, Royer, Rechcigl and Recheigl2,Reference Zalom and Krieger3. To reduce those types of risks, IPM practices commonly include such things as reducing pesticide use, applying reduced risk pesticides and releasing beneficial insects into a farming areaReference Cuperus, Mulder, Royer, Rechcigl and Recheigl2.
While existing as an agricultural strategy for many decades, IPM programs took on an increased level of significance in the past ten years as new programs were introduced emphasizing IPM strategies as a way to reduce the environmental impacts from pesticide use. One of these programs was the National IPM Initiative. The purpose of this initiative was to have 75% of agricultural producers implement IPM practices by the year 20004. However, the General Accounting Office discovered a number of problems with this initiative and IPM programs in general. Among these problems were a lack of measurable goals for IPM programs, an absence of a methodology to measure any progress occurring in IPM programs, and inefficient management of IPM programs by the USDA and the United States Environmental Protection Agency (USEPA)4. From a policy perspective, this report revealed a basic deficiency in IPM programs. Without measurable goals, IPM programs could not be evaluated properly and did not conform to the Government Performance and Results Act (GPRA). In addition, no data existed to indicate that IPM techniques were better for the environment than conventional agricultural techniques. Without this type of data, IPM stakeholders faced the prospect of reduced funding since policymakers would not be able to determine if resources spent on IPM programs actually achieved anything.
As a result of the GAO report, consensus has emerged on the need to evaluate the environmental benefits of IPM programs. Researchers traditionally evaluated IPM programs using ‘adoption surveys’ that measured how many growers in one specific area consistently used an IPM technique in their farming practices4. The assumption behind such surveys was that an increased level of IPM adoption equated to a decreased level of environmental risk from agricultural practices. However, IPM programs now have to quantitatively show program outcomes. One way to quantitatively show that IPM programs can reduce the environmental risk from pesticide applications is with pesticide risk indicators.
Pesticide risk indicators generate a potential environmental risk score based on certain data inputs like the amount of pesticide applied to a field, the level of toxicity of the pesticide to beneficial organisms, and the pesticide's ability to infiltrate groundwater. This potential risk score represents the amount of potential environmental impact that a pesticide application strategy causes in a particular area (since only environmental sampling can indicate if an actual environmental impact occurred, results from pesticide risk indicators are always phrased with the word potential impact). Researchers have not determined the validity or reliability of these indicators with real application data. Instead, researchers typically use estimates of pesticide application data to determine validity and reliabilityReference Reus, Leendertse, Bockstaller, Fomsgaard, Gutsche, Lewis, Nilsson, Pussemier, Trevisan, van der Werf, Alfarroba, Blumel, Isart, McGrath and Seppala5. As a result, this study attempts to establish the validity and reliability of eight pesticide risk indicators with actual application data.
Pesticide Risk Indicators
There are two basic types of pesticide risk indicators. Those that use a ranking approach to generate potential risk scores, and those that use some combination of ranks and predicted environmental concentrations (PECs) to generate potential risk scores. In the ranking approach, the indicator establishes unitless ranks that categorize data points on the pesticide's toxicity to humans and a variety of beneficial organisms. In addition, indicators can use the ranking methodology to categorize data points that reveal how long a pesticide stays in surface water, ground water, or soil. Depending on the indicator, different equations either divide or multiply these two ranks to estimate a potential risk score for a pesticide's active ingredient. Then, by multiplying or dividing the potential risk score by the application rate, these types of indicators can generate a potential risk score for each application of a pesticide's active ingredient.
Other pesticide risk indicators rely on a more quantifiable PEC methodology to assess potential risk. In this type of methodology, the indicator uses environmental engineering equations in order to calculate how much of the pesticide remains in the soil, groundwater, surface water and even air. The indicators that use a PEC methodology often also employ a ranking approach, either to categorize the amount of the pesticide's concentration or to categorize pesticide toxicity data. By multiplying or dividing PECs by ranked data, the indicators using a PEC methodology estimate a potential risk score for a pesticide application.
After reviewing the existing literature, the authors chose eight indicators to analyze because they either appeared in peer-reviewed journals or were actively used by governmental agencies and European researchers (Table 1). The design of these indicators, how they can be used, and the equations behind the indicators have all been specifically explained in previous studiesReference Reus, Leendertse, Bockstaller, Fomsgaard, Gutsche, Lewis, Nilsson, Pussemier, Trevisan, van der Werf, Alfarroba, Blumel, Isart, McGrath and Seppala5–Reference Benbrook, Sexon, Wyman, Stevenson, Lynch, Wallendal, Diercks, Van Haren and Granadino14. Therefore, in consideration of space, this research will only provide a brief overview of each indicator (however, a complete overview of each indicator's equations is available at http://www.aftresearch.org/ipm/risk/equations.pdf).
Pesticide risk indicators and their country of origin.
EIQ: environmental impact quotient (USA)
J. Kovach, C. Petzoldt, J. Degni, and J. TetteReference Kovach, Petzoldt, Degni and Tette6 developed the EIQ indicator. EIQ is a pesticide risk indicator that assesses the potential impact of pesticides on farmworkers, consumers and terrestrial organisms. It ranks a pesticide's toxicity data to generate potential risk scores for beneficial organisms and humans. It then multiplies these potential risk scores by the pesticide's application rate to determine a final potential risk score.
CHEMS 1: chemical hazard evaluation for management strategies (USA)
M. SwansonReference Swanson, Davis, Kincaid, Schultz, Bartmess, Jones and George7 led a group of researchers who designed the CHEMS 1. CHEMS 1 is a chemical risk indicator that considers the potential environmental impact of chemicals on air, soil, groundwater and surface water. CHEMS 1 uses a ranking approach to generate risk scores for chemical applications. For example, the indicator ranks chemical toxicity data to determine a chemical hazard value for human health and the environment. These hazard values are then multiplied by a ranked concentration factor that shows how much of the chemical already resides in the environment. To arrive at a ranked concentration factor, the indicator adds the amount of chemical applied and the amount of chemical already residing in the specific environmental area, and then ranks that resulting value. It is important to note that the design of CHEMS 1 favors industrial chemicals rather than agricultural pesticides. However, the authors decided to include it in the analysis since it uses a ranking methodology similar to other risk indicators designed exclusively for agriculture.
SYNOPS_2: synoptic evaluation model for plant protection agents (Germany)
The Federal Biological Research Centre for Agriculture and Forestry, Institute for Technology Assessment in Plant Protection, Kleinmachnow, Germany developed SYNOPSReference Gutsche, Rossberg, Reus, Leendertse, Bockstaller, Fomsgaard, Gutsche, Lewis, Nilsson, Pussemier, Trevisan, van der Werf, Alfarroba, Blümel, Isart, McGrath and Seppala8. The purpose of the indicator is to assess the environmental risk potential of a pesticide application strategy in a region and to compare pest management strategies with different pesticide options. The indicator calculates PECs in soil, groundwater and surface water for each pesticide. Then, the indicator divides those values by toxicity data for beneficial organisms and humans to produce a potential risk value.
PERI: pesticide environmental risk indicator (Sweden)
C. NilssonReference Nilsson, Reus, Leendertse, Bockstaller, Fomsgaard, Gutsche, Lewis, Nilsson, Pussemier, Trevisan, van der Werf, Alfarroba, Blümel, Isart, McGrath and Seppala9 developed the PERI indicator as part of a system of indicators that Swedish farmers could use to evaluate potential environmental risk as part of an ISO 14001 certification process. To evaluate potential environmental risk, PERI uses a ranking methodology that assesses pesticide properties and toxicity values on a 1–5 scale. Once calculated, the indicator multiplies the potential environmental risk score by the pesticide application rate to arrive at a final estimate of potential environmental risk for groundwater, surface water and air.
SyPEP: system for predicting the environmental impact of pesticides (Belgium)
L. PussemierReference Pussemier, Reus, Leendertse, Bockstaller, Fomsgaard, Gutsche, Lewis, Nilsson, Pussemier, Trevisan, van der Werf, Alfarroba, Blümel, Isart, McGrath and Seppala10 developed the SyPEP model to help farmers, extension services and regulating agencies by providing information on the environmental impact of pesticides. The indicator calculates a long-term PEC for groundwater, a short-term PEC for groundwater, and a PEC for surface water. It then divides toxicity information by the PEC in each environmental compartment. The resulting value in each of the three compartments is then ranked on a 0–5 scale to arrive at a SyPEP score.
EYP: the environmental yardstick for pesticides (The Netherlands)
J. Reus and P. LeendertseReference Reus, Leendertse, Reus, Leendertse, Bockstaller, Fomsgaard, Gutsche, Lewis, Nilsson, Pussemier, Trevisan, van der Werf, Alfarroba, Blümel, Isart, McGrath and Seppala11 developed EYP for use by Dutch farmers and governmental officials. The EYP indicator calculates a PEC of a pesticide for groundwater, surface water and soil. The indicator then multiplies the PEC by the pesticide's toxicity data in order to produce Environmental Impact Points that reflect potential risk to beneficial organisms and humans.
EPRIP: environmental potential risk indicator for pesticides (Italy)
M. Trevisan, G. Errera, E. Capri, L. Padovani, and A. Del ReReference Trevisan, Errera, Capri, Padovani, Del Re, Reus, Leendertse, Bockstaller, Fomsgaard, Gutsche, Lewis, Nilsson, Pussemier, Trevisan, van der Werf, Alfarroba, Blümel, Isart, McGrath and Seppala12 developed EPRIP for Italian agriculture. EPRIP calculates and compares PECs across the environmental compartments of air, soil, groundwater and surface water. To arrive at a PEC, the indicator uses a variety of equations that consider a pesticide's exposure potential and site-specific application data. Then, EPRIP divides the PEC by the pesticide's toxicity data in order to generate an EPRIP potential risk score for beneficial organisms and humans.
MATF: multi-attribute toxicity factor model (USA)
C. BenbrookReference Benbrook, Sexon, Wyman, Stevenson, Lynch, Wallendal, Diercks, Van Haren and Granadino14 led a research team that designed the MATF indicator to calculate the toxicity of pesticides for the ‘Healthy Grown’ Wisconsin Potato IPM Labeling Project. The MATF indicator ranks toxicity data in order to generate toxicity factor scores for beneficial organisms and humans. It then multiplies these scores by the pesticide's application rate in order to produce toxicity units for each application, with more toxicity units indicating more potential risk. It is important to note that researchers designed this indicator specifically for Wisconsin potatoes. However, the present authors included it in this analysis because its methodology is similar to the other indicators analyzed.
Method of Analysis
Building upon the results of other research, stressing the need to develop systems that can evaluate the impacts of pesticide applicationsReference Sorenson15,Reference Lynch, Greene, Kramer-LeBlanc, Lynch, Greene and Kramer-LeBlanc16, this study analyzed eight pesticide risk indicators with actual pesticide application data from four IPM farms in Southwest Florida that grew either tomatoes or peppers (Table 2). These farms were small with field acreage ranging in size from 8 ha (20 acres) for farms A, B and C, and 34 ha (85 acres) for farm D. All of the farms had sandy soil, resided near water bodies and routinely implemented advanced IPM tactics as part of their growing strategy. By analyzing these indicators with actual pesticide application data from these farms, this analysis reached a basic conclusion on the validity and reliability of these indicators.
Crop scenarios for each farm participating in the study.
As shown in Table 3, each pesticide risk indicator calculated a potential risk score based on certain input parameters. For example, all of the indicators required pesticide application data, which was obtained from four vegetable growers over two growing seasons. For these years, the growers grew either tomatoes or peppers, or left their fields fallow. In addition to pesticide application data, some of the indicators required soil data. To obtain these data, researchers used soil probes to collect the top 20 cm of topsoil from representative sites on each farm. In total, 12 soil samples were taken: five from Farm A, one from Farm B, one from Farm C, and five from Farm D. More samples were taken on Farms A and D because fields in those areas had slightly more variability in soil characteristics. In addition, more samples were taken on Farm D because of its larger size. When multiple samples were taken on a farm, researchers divided the farm into four identical rectangular quadrants and then randomly chose five locations for sample sites with at least one sample originating from each quadrant. All of these samples were then analyzed separately to determine organic content levels, pH levels, and sand/silt/clay levels. These results were then averaged for Farms A and D so that each farm had one set of soil data. As shown in Table 3, some of the indicators used this type of information to calculate the ability of a pesticide's active ingredient to leach into groundwater.
Required data inputs for each pesticide indicator.
Additional information collected included the acreage size of each field, meteorological data, a field's distance to a body of water, and toxicity data for beneficial organisms. Toxicity data came from the US Environmental Protection Agency's Ecotox Database (http://www.epa.gov/ecotox/) and The Pesticide Manual Reference Tomlin17. Once collected, all of these data points allowed each pesticide risk indicator to generate a potential risk score for each farm over 2 years.
If the indicators possess validity, then high-risk pesticide applications should result in higher potential risk scores. These types of high-risk pesticide applications can include the use of high-risk active ingredients such as methyl bromide. But they can also include large applications of newer, safer pesticides since all pesticides possess some amount of toxicity. This type of basic validity, where a measure produces accurate assessments, is known as measurement validityReference Hatry, Newcomer, Wholey, Hatry and Newcomer18.
Additionally, if the indicators possess reliability, then they should generate consistent potential risk scores across different farms and through different years. For example, two different indicators should similarly assess methyl bromide applications even as other site-specific variables change across different farms and through different years. This is known as measurement reliability, since it examines whether a measurement tool such as a pesticide risk indicator produces similar results with repeated usesReference Hatry, Newcomer, Wholey, Hatry and Newcomer18.
Assessing this type of validity and reliability is not easy with pesticide risk indicators. As detailed in the preceding section, each indicator has a different methodology with different sets of data inputs. This makes a direct comparison of each indicator's potential risk scores meaningless. Researchers on European pesticide risk indicators associated with the Concerted Action on Pesticide Environmental Risk Indicators project (more commonly known as the CAPER project) solved this problem by ranking pesticide application rates and the potential risk scores of each indicator from highest to lowest, and then analyzing those ranks with Spearman's Rho correlations for ranked dataReference Reus, Leendertse, Bockstaller, Fomsgaard, Gutsche, Lewis, Nilsson, Pussemier, Trevisan, van der Werf, Alfarroba, Blumel, Isart, McGrath and Seppala5.
In a Spearman's Rho correlation analysis, interval level data cases are ranked on an ordinal scale from highest to lowest with tied data cases resulting in averaged ranks. This results in ranked data for each variable of interest. After ranking each variable's data cases, the ranked results between each variable of interest can be analyzed with Spearman's Rho correlations to determine if there is a statistically significant correlation between the ranks of one variable and the ranks of another variable. This type of analysis helped CAPER researchers determine if different pesticide risk indicators from Europe assessed the same application strategy similarly. Building upon the results of the CAPER project, such an analysis will also be performed in this project to help assess measurement validity and reliability for both European and American pesticide risk indicators.
When the application rate of the pesticide's active ingredient and the potential risk score from each indicator are ranked in such a manner, a determination of validity can occur since the ranked potential risk scores should correlate with the ranked application rates (i.e. higher potential risk scores should correlate with higher risk application rates). This type of Spearman's Rho correlation analysis can also help determine reliability since statistically significant Spearman's Rho correlations between one indicator's ranked potential risk scores and another indicator's ranked potential risk scores should remain as pesticide application strategies change from farm to farm and from year to year. For example, if the EIQ and MATF indicators reliably generate potential risk scores that correlate with one another, then that correlation should remain across different farms and through different years. This helps to determine if the indicators possess measurement reliability.
Results
Establishing validity
We performed this methodology on each indicator, for each farm, for 2 years. Basic conclusions on validity became apparent without even using the Spearman's Rho analysis. As shown in Tables 4 and 5, most indicators gave higher potential risk scores to higher risk pesticide applications that included active ingredients in restricted use pesticides, organophosphates and carbamates. The highest potential risk scores generally went to methyl bromide applications. This indicates at least a rudimentary level of measurement validity: the pesticide risk indicators gave higher potential risk scores to higher risk pesticide applications.
Total ranked scores for farms in year 1 (with tied ranks averaged according to Spearman's Rho analytic protocolReference Chen and Popovich19).
Total ranked scores for farms in year 2 (with tied ranks averaged according to Spearman's Rho analytic protocolReference Chen and Popovich19).
However, the best evidence for validity is from the Spearman's Rho correlation analysis with ranked application rates and ranked potential risk scores. If these indicators measure the potential risk of pesticides correctly, then larger application rates should result in larger potential risk scores. That would result in statistically significant correlations between the ranked potential risk score of each indicator and the application rate. As detailed in Tables 6 and 7, most of the indicators had a statistically significant correlation with the application rate on at least one farm. However, only EIQ, MATF and SYNOPS had statistically significant correlations on each farm throughout the 2 years of application data. This reveals that these indicators possess the most measurement validity since they had constant, statistically significant correlations with the application rate across different farms, application strategies and different years.
Spearman's Rho correlations for Farms A–D, year 1.
* Correlation is significant at the 0.05 level (two-tailed).
** Correlation is significant at the 0.01 level (two-tailed).
Spearman's Rho correlations for Farms B–D, year 2.
* Correlation is significant at the 0.05 level (two-tailed).
** Correlation is significant at the 0.01 level (two-tailed).
Establishing reliability
Tables 6 and 7 also reveal conclusions on measurement reliability. These tables show that significant correlations between many pesticide risk indicators are not consistent from year to year or even from farm to farm. This indicates that certain pesticide risk indicators do not generate reliable potential risk scores. For reliability, we would expect to find that significant correlations between pesticide risk indicators would remain from farm to farm and from year to year, even though pesticide application strategies changed. Instead, we find that significant correlations between many of the indicators disappear and reappear when the analysis shifts to new farms or new years that have different pesticide application strategies.
These inconsistencies make a final interpretation on reliability difficult. Perhaps the best interpretation of the data is that the rankings of potential pesticide risk from the indicators SYNOPS, EIQ and MATF generally correlate with each other at statistically significant levels. In contrast, the other indicators do not have consistent correlations. As a result, this study's results indicate that these three pesticide risk indicators have the most measurement reliability. That is, the ranked values of potential risk scores correlate fairly consistently with these three indicators from different farms over different years.
Discussion
Using pesticide risk indicators to evaluate IPM programs has certain advantages. Perhaps most importantly, using these indicators to evaluate IPM programs allows policymakers and stakeholders to determine if IPM programs actually reduce the environmental and health risks from pesticide use. However, there are also disadvantages to using these indicators. Currently, there is disagreement in the toxicology literature regarding the most effective method to measure environmental and health impacts from agricultural chemical applications. The literature recognizes that agricultural operations can result in negative environmental impacts and pose some risk to human healthReference Ruhl20. Yet, no agreed upon methodology exists to measure these impacts. For example, methods for measuring the environmental effects of pesticide use include hazard indicators that assess environmental effects by analyzing chemical property data associated with a particular pesticide and methods that attempt to predict the actual environmental concentration of the pesticide in the environmentReference Falconer21. Consequently, choosing one environmental assessment method over another becomes a complex process that often depends on which environmental threat the evaluators, stakeholders or policymakers wish to assess.
When choosing pesticide risk indicators for IPM evaluation, evaluators, stakeholders and policymakers should analyze how the pesticide risk indicator actually generates potential risk scores. In this analysis, the most consistent pesticide risk indictors (SYNOPS, EIQ and MATF) all differ in terms of methodology. When compared to the MATF and EIQ indicators, the SYNOPS indicator is more complex. It considers daily weather information and specific soil information while the MATF and EIQ indicators do not. The SYNOPS indicator also predicts environmental concentrations of the pesticide used in soil, air, surface water and groundwater. In contrast, indicators like EIQ and MATF categorize data points with a ranking methodology to arrive at a potential risk score. Therefore, if an evaluator needs an actual prediction of environmental concentration (PEC) for part of an IPM evaluation, then indicators like SYNOPS have to be used.
These types of issues comprise the main concerns over using any pesticide risk indicators in IPM policy. Evaluators can use any pesticide risk indicator as part of an overall approach to evaluate an IPM growing strategy. However, based on this analysis, we recommend that evaluators use the EIQ, MATF or SYNOPS indicators. Additionally, we recommend that evaluators or researchers validate the results of these indicators with actual environmental samples. Only then, can we be sure that pesticide risk indicators generate accurate potential risk scores for pesticide applications.
Conclusion
In many cases, the pesticide risk indicators analyzed are either used by governmental agencies or by farming groups to assess environmental impact from pesticide applicationsReference Reus, Leendertse, Bockstaller, Fomsgaard, Gutsche, Lewis, Nilsson, Pussemier, Trevisan, van der Werf, Alfarroba, Blumel, Isart, McGrath and Seppala5,Reference Benbrook, Sexon, Wyman, Stevenson, Lynch, Wallendal, Diercks, Van Haren and Granadino14. However, in order to achieve a fully integrated approach to assessing IPM programs, a variety of other components have to be assessed other than pesticide use. Such things as nutrient management as well as the economic costs of switching to reduced risk must be included when policymakers or agricultural professionals attempt to measure the impact of IPM programs. As a result, the importance of these indicators is in their ability to contribute to a comprehensive risk assessment for IPM practices.
Researchers need to place more emphasis on interdisciplinary assessments of the economic, environmental and human health impacts of agricultural programs such as IPM. In general, IPM researchers have not included such an assessment in their work. In addition, since the passing of GPRA, there is a growing demand by the public as well as government to be socially and economically accountable for all publicly funded research. Under increased funding constraints, IPM programs must show that IPM techniques result in less environmental risk. Pesticide risk indicators can help that occur. But this research shows that these indicators, as currently devised, need more scrutiny. Future analyses should examine how specific data inputs influence a pesticide risk indicator's potential risk score. In addition, researchers must determine if potential risk scores heavily influenced by application rates actually give usable information on how the pesticide application scheme is affecting the environment, or whether such scores are just an extension of the qualitative scheme that equated reduced application rates of pesticide with reduced environmental risk. Lastly, future research has to examine the economic effects of reducing environmental risk. Using pesticide risk indicators to measure environmental risk means nothing if the grower has no incentive to reduce risk in the first place.
Acknowledgements
Support for this study was provided by the US Environmental Protection Agency and American Farmland Trust.
