We appreciate the effort and thought that Stewart-Oaten (Reference Stewart-Oaten2008) has put into his comment on our paper in Environmental Conservation (Bulleri et al. Reference Bulleri, Underwood and Benedetti-Cecchi2007). Clearly, the philosophical and methodological approaches used in ecological impact assessments (EIAs) warrant much further attention and discussion by ecologists. We would like, however, to clarify that the goals of our comment were far from summarizing available procedures and, even more, from expressing an opinion about which among these should be retained or rejected. This is quite well reflected by the contents of our paper (Bulleri et al. Reference Bulleri, Underwood and Benedetti-Cecchi2007). Our ‘main purpose’ (to quote Stewart-Oaten Reference Stewart-Oaten2008) was to point out that the degree of naturalness of a site is irrelevant to whether or not some proposed disturbance causes an impact and that all disturbances potentially cause negative impacts.
Nonetheless, we welcome this opportunity for offering our contrasting view on some of the points made by Stewart-Oaten (Reference Stewart-Oaten2008) that would supposedly invalidate the design-based approach (Underwood Reference Underwood1991, Reference Underwood1992, Reference Underwood1994) as a tool for EIAs. We start by reiterating some of the points we made in our previous paper on the issue of assessing and interpreting ecological impacts in human dominated environments.
There, we simply reported that two major EIA approaches have, to date, received a great deal of attention. In this context, we used the term ‘alternative’, which Stewart-Oaten (Reference Stewart-Oaten2008) considered to be misleading. The term was used solely to indicate intrinsic philosophical differences between the two and did not imply that they are, in their present form, compatible. Reference sites (or controls) can, in fact, be investigated in both model- and design-based analyses, but with very different purposes (Underwood Reference Underwood1991, Reference Underwood1992, Reference Underwood1994; Stewart-Oaten & Bence Reference Stewart-Oaten and Bence2001). Irrespective of whether reference sites are or are not included in a study and of the specific objectives and criteria by which these are eventually chosen, it is only in terms of relative changes in response variables (be they the number of species, their abundances or any other measure of choice) that an impact could ever be quantified. As such, the background degree of ‘naturalness’ does not play a role at this stage of the EIA. Comparing an impacted site with a natural condition becomes important at a later stage, when a decision is made as to whether the estimated change can or cannot be considered acceptable (Bulleri et al. Reference Bulleri, Underwood and Benedetti-Cecchi2007).
Stewart-Oaten's (2008) main argument against design-based analyses of impact assessments is that the disturbed site and the reference sites are not and cannot be randomly chosen from a population of sites (Underwood Reference Underwood1992, Reference Underwood1994). He stated that this is inevitable because the site that is going to be affected by a given human intervention is not chosen at random, but is chosen to fulfil the requirements of the planned human intervention. Of course we agree that disturbed site(s) are not distributed randomly in nature. The candidate sites might be anywhere and not random in space, but a representative (random) sample of them is still required. Nonetheless, it is possible and appropriate to argue that the variables measured in disturbed and reference sites can be random samples from some population of sites (i.e. the means of the variables across the sites have an underlying random distribution), provided that the criteria used by developers for identifying the features of the ‘impacted’ site are also used to define the reference sites. We assume that, whatever are the features of a site determining its selection by an oil company or a local council to build a drilling platform or to place a sewage outfall, these can be used to define a set of possible pieces of habitat to sample as references. As discussed in detail elsewhere, including the decision to choose a single reference (or control) location in before-after/control-impact (BACI) designs (Green Reference Green1979; Stewart-Oaten et al. Reference Stewart-Oaten, Murdoch and Parker1986), the criteria used to determine features of the site to be disturbed can be used to define a set or population of such sites. So, if an outfall requires a certain depth of water, speed of current, distance from an urban area or other criteria, other places with this depth, speed and distance should not be dismissed as being appropriate reference locations. In this respect, reference sites do not have to be identical to the potentially impacted one, they have to be a sample representing the distribution of sites from which the disturbed one has been chosen. Of course, not all possible variables have to be measured, only those considered relevant. If restricting the number of variables taken into account to define a population of sites is a flaw (we believe it is not), it would also affect model-based assessments. If any variable not measured as part of defining the pre-disturbance conditions were to change during the period while the development is being built, it could affect the variables being measured, but such effects could not possibly be predicted. In the model-based methodology, it could not be considered at all. In the design-based approach, its effects on the reference and disturbed areas would be measured independently, so, in principle, it would be used in interpretations.
Choosing sites (usually on the basis of their physical features) does not mean choosing the populations or assemblages of organisms to be sampled. We agree with Stewart-Oaten that each bay along a stretch of coast will have unique features (for example size, substratum, depth and exposure), but this does not imply that the assemblages or population of organisms to be sampled in them do not have mean values of relevant variables that belong to a ‘population’ of values that include those at the site to be disturbed. To argue that the relevant ecologically variables at the site chosen to be disturbed are inevitably intrinsically different (come from a different distribution) from other sites requires that these variables must be used by developers in choosing their site. It would also require carefully designed preliminary surveys in order to identify a site that biologically behaves differently from all the other candidates that could be included in the potential set.
When before data are available, it is possible to estimate whether the site that will be affected by the planned intervention and those identified as controls should not be considered to belong to the same ‘population’. Denying this possibility is equivalent to denying that it is possible to ascertain whether assemblages or populations of organisms at a number of sites along a coast do or do not belong to some ‘population’.
Stewart-Oaten's argument could and should be equally applied to choosing sites in an experiment, where he sees no problem in randomizing. The only difference between such experiments and many environmental assessments is that the former are usually smaller than disturbed locations in environmental monitoring. This makes no difference to the argument that everywhere (at some specified scale) is unique, which is the core of his argument. We know of no reason to do with scale per se that makes choosing sites suddenly correct at a smaller scale, but not possible at a larger scale. There is plenty of evidence to suggest that there will always be variation in processes affecting assemblages or populations of organisms at sites along a coast (Fraschetti et al. Reference Fraschetti, Terlizzi and Benedetti-Cecchi2005).
Here, we wish to rebut the criticism made by Stewart-Oaten (Reference Stewart-Oaten2008) of design-based impact assessments, without producing a counter-attack to the model-based approach, even though the latter has been stated by the same author to be the only valid tool for assessing ecological impacts (Stewart-Oaten & Bence Reference Stewart-Oaten and Bence2001; Stewart-Oaten Reference Stewart-Oaten2008). Patterns of abundance and distribution of populations and assemblages of organisms vary greatly in space and through time, generating spatial-temporal interactions. It is difficult to conceive of an impact being anything different from a particular form of spatial–temporal interaction (from before to after the disturbance, things must change in the disturbed site in some way different from natural changes over the same period in sites not subject to that disturbance). For this reason, we believe that several locations must be sampled as the references, so that the amounts of such interaction can be measured to determine whether the proportion associated with the disturbed location is in any way unusual. A recent attempt to implement the design-based approach with a model-based one (using geographical coordinates as covariates to reduce possible spatial effects introduced by the use of a set of reference sites) has generated promising results (Benedetti-Cecchi & Osio Reference Benedetti-Cecchi and Osio2007), suggesting that further research is needed to improve confidence in assessing ecological impacts.
We insist on the importance of everyone thinking for themselves about the pros and cons of design- versus model-based analysis. We invite practitioners to make up their own minds whether or not they believe that induction is better than anything else. The ‘counterfactual’ advocated by Stewart-Oaten (Reference Stewart-Oaten2008) could only be used to demonstrate the presence of an impact if it were believed (inductively) that no similar change would have occurred if the disturbance had not happened. Design-based analyses are also inductivist, but at least attempt to estimate the magnitude of interactions through time amongst various reference sites. In the absence of any estimate of how much change actually happened in reference sites (i.e. not due to the planned disturbance), any departure from what was predicted by the model-based approach would have to be interpreted as an impact. Readers should also weigh up whether or not they have sufficient data from before a disturbance to be in any way confident about making predictions from one site.
Finally, we agree with Stewart-Oaten in arguing that people doing analyses (be they about impacts or not) should also be aware of the great need to explain what they are doing and why, and take care to ensure that readers will be aware of potential shortcomings and pitfalls for interpretation.
You have
Access
We appreciate the effort and thought that Stewart-Oaten (Reference Stewart-Oaten2008) has put into his comment on our paper in Environmental Conservation (Bulleri et al. Reference Bulleri, Underwood and Benedetti-Cecchi2007). Clearly, the philosophical and methodological approaches used in ecological impact assessments (EIAs) warrant much further attention and discussion by ecologists. We would like, however, to clarify that the goals of our comment were far from summarizing available procedures and, even more, from expressing an opinion about which among these should be retained or rejected. This is quite well reflected by the contents of our paper (Bulleri et al. Reference Bulleri, Underwood and Benedetti-Cecchi2007). Our ‘main purpose’ (to quote Stewart-Oaten Reference Stewart-Oaten2008) was to point out that the degree of naturalness of a site is irrelevant to whether or not some proposed disturbance causes an impact and that all disturbances potentially cause negative impacts.
Nonetheless, we welcome this opportunity for offering our contrasting view on some of the points made by Stewart-Oaten (Reference Stewart-Oaten2008) that would supposedly invalidate the design-based approach (Underwood Reference Underwood1991, Reference Underwood1992, Reference Underwood1994) as a tool for EIAs. We start by reiterating some of the points we made in our previous paper on the issue of assessing and interpreting ecological impacts in human dominated environments.
There, we simply reported that two major EIA approaches have, to date, received a great deal of attention. In this context, we used the term ‘alternative’, which Stewart-Oaten (Reference Stewart-Oaten2008) considered to be misleading. The term was used solely to indicate intrinsic philosophical differences between the two and did not imply that they are, in their present form, compatible. Reference sites (or controls) can, in fact, be investigated in both model- and design-based analyses, but with very different purposes (Underwood Reference Underwood1991, Reference Underwood1992, Reference Underwood1994; Stewart-Oaten & Bence Reference Stewart-Oaten and Bence2001). Irrespective of whether reference sites are or are not included in a study and of the specific objectives and criteria by which these are eventually chosen, it is only in terms of relative changes in response variables (be they the number of species, their abundances or any other measure of choice) that an impact could ever be quantified. As such, the background degree of ‘naturalness’ does not play a role at this stage of the EIA. Comparing an impacted site with a natural condition becomes important at a later stage, when a decision is made as to whether the estimated change can or cannot be considered acceptable (Bulleri et al. Reference Bulleri, Underwood and Benedetti-Cecchi2007).
Stewart-Oaten's (2008) main argument against design-based analyses of impact assessments is that the disturbed site and the reference sites are not and cannot be randomly chosen from a population of sites (Underwood Reference Underwood1992, Reference Underwood1994). He stated that this is inevitable because the site that is going to be affected by a given human intervention is not chosen at random, but is chosen to fulfil the requirements of the planned human intervention. Of course we agree that disturbed site(s) are not distributed randomly in nature. The candidate sites might be anywhere and not random in space, but a representative (random) sample of them is still required. Nonetheless, it is possible and appropriate to argue that the variables measured in disturbed and reference sites can be random samples from some population of sites (i.e. the means of the variables across the sites have an underlying random distribution), provided that the criteria used by developers for identifying the features of the ‘impacted’ site are also used to define the reference sites. We assume that, whatever are the features of a site determining its selection by an oil company or a local council to build a drilling platform or to place a sewage outfall, these can be used to define a set of possible pieces of habitat to sample as references. As discussed in detail elsewhere, including the decision to choose a single reference (or control) location in before-after/control-impact (BACI) designs (Green Reference Green1979; Stewart-Oaten et al. Reference Stewart-Oaten, Murdoch and Parker1986), the criteria used to determine features of the site to be disturbed can be used to define a set or population of such sites. So, if an outfall requires a certain depth of water, speed of current, distance from an urban area or other criteria, other places with this depth, speed and distance should not be dismissed as being appropriate reference locations. In this respect, reference sites do not have to be identical to the potentially impacted one, they have to be a sample representing the distribution of sites from which the disturbed one has been chosen. Of course, not all possible variables have to be measured, only those considered relevant. If restricting the number of variables taken into account to define a population of sites is a flaw (we believe it is not), it would also affect model-based assessments. If any variable not measured as part of defining the pre-disturbance conditions were to change during the period while the development is being built, it could affect the variables being measured, but such effects could not possibly be predicted. In the model-based methodology, it could not be considered at all. In the design-based approach, its effects on the reference and disturbed areas would be measured independently, so, in principle, it would be used in interpretations.
Choosing sites (usually on the basis of their physical features) does not mean choosing the populations or assemblages of organisms to be sampled. We agree with Stewart-Oaten that each bay along a stretch of coast will have unique features (for example size, substratum, depth and exposure), but this does not imply that the assemblages or population of organisms to be sampled in them do not have mean values of relevant variables that belong to a ‘population’ of values that include those at the site to be disturbed. To argue that the relevant ecologically variables at the site chosen to be disturbed are inevitably intrinsically different (come from a different distribution) from other sites requires that these variables must be used by developers in choosing their site. It would also require carefully designed preliminary surveys in order to identify a site that biologically behaves differently from all the other candidates that could be included in the potential set.
When before data are available, it is possible to estimate whether the site that will be affected by the planned intervention and those identified as controls should not be considered to belong to the same ‘population’. Denying this possibility is equivalent to denying that it is possible to ascertain whether assemblages or populations of organisms at a number of sites along a coast do or do not belong to some ‘population’.
Stewart-Oaten's argument could and should be equally applied to choosing sites in an experiment, where he sees no problem in randomizing. The only difference between such experiments and many environmental assessments is that the former are usually smaller than disturbed locations in environmental monitoring. This makes no difference to the argument that everywhere (at some specified scale) is unique, which is the core of his argument. We know of no reason to do with scale per se that makes choosing sites suddenly correct at a smaller scale, but not possible at a larger scale. There is plenty of evidence to suggest that there will always be variation in processes affecting assemblages or populations of organisms at sites along a coast (Fraschetti et al. Reference Fraschetti, Terlizzi and Benedetti-Cecchi2005).
Here, we wish to rebut the criticism made by Stewart-Oaten (Reference Stewart-Oaten2008) of design-based impact assessments, without producing a counter-attack to the model-based approach, even though the latter has been stated by the same author to be the only valid tool for assessing ecological impacts (Stewart-Oaten & Bence Reference Stewart-Oaten and Bence2001; Stewart-Oaten Reference Stewart-Oaten2008). Patterns of abundance and distribution of populations and assemblages of organisms vary greatly in space and through time, generating spatial-temporal interactions. It is difficult to conceive of an impact being anything different from a particular form of spatial–temporal interaction (from before to after the disturbance, things must change in the disturbed site in some way different from natural changes over the same period in sites not subject to that disturbance). For this reason, we believe that several locations must be sampled as the references, so that the amounts of such interaction can be measured to determine whether the proportion associated with the disturbed location is in any way unusual. A recent attempt to implement the design-based approach with a model-based one (using geographical coordinates as covariates to reduce possible spatial effects introduced by the use of a set of reference sites) has generated promising results (Benedetti-Cecchi & Osio Reference Benedetti-Cecchi and Osio2007), suggesting that further research is needed to improve confidence in assessing ecological impacts.
We insist on the importance of everyone thinking for themselves about the pros and cons of design- versus model-based analysis. We invite practitioners to make up their own minds whether or not they believe that induction is better than anything else. The ‘counterfactual’ advocated by Stewart-Oaten (Reference Stewart-Oaten2008) could only be used to demonstrate the presence of an impact if it were believed (inductively) that no similar change would have occurred if the disturbance had not happened. Design-based analyses are also inductivist, but at least attempt to estimate the magnitude of interactions through time amongst various reference sites. In the absence of any estimate of how much change actually happened in reference sites (i.e. not due to the planned disturbance), any departure from what was predicted by the model-based approach would have to be interpreted as an impact. Readers should also weigh up whether or not they have sufficient data from before a disturbance to be in any way confident about making predictions from one site.
Finally, we agree with Stewart-Oaten in arguing that people doing analyses (be they about impacts or not) should also be aware of the great need to explain what they are doing and why, and take care to ensure that readers will be aware of potential shortcomings and pitfalls for interpretation.