Formulating individual hypotheses

The logic of hypothesis testing places the target assertion into the alternative hypothesis. Everything else within the parameter space of interest goes into the null. The sponsor of an efficacy study supports its product's regulatory approval by rejecting the null of zero-difference. The sponsor of a safety study should firmly defend its product's authorization by rejecting the null of non-equivalence, rather than blindly adopting the hypotheses in the efficacy study and wishing for non-significance. Defining safety in terms of equivalence is not without precedence. For example, Hsu & Berger (Reference Hsu and Berger1999) advocate tests of practical equivalence in assessing the toxicity of bovine growth hormone. Oberdoerfer et al. (Reference Oberdoerfer, Shillito, Beuckeleer and Mitten2005) were the first to apply equivalence testing to crop safety evaluation. Practitioners, on the other hand, are often attracted by the mathematical convenience in testing for difference and avoid equivalence testing due to complications in defining appropriate ELs. Blunt advice against reliance on such ill-posed hypotheses is to cite Hsu (Reference Hsu1996): ‘an approximate solution to the right problem … will be better than an exact solution to the wrong problem, the latter accounting for a significant portion of the abuse.’

Table 1 shows a consensus that $\mu _{\rm T}^m - \mu _{\rm C}^m $ must be small to establish equivalence. The present work proposes to compare GMO with the non-GM counterpart at endpoint m by testing

(4) $$H_{{\rm TC}}^m :\left \vert {\mu _{\rm T}^m - \mu _{\rm C}^m} \right \vert \ge \delta _0\;v.\;K_{{\rm TC}}^m :\left \vert {\mu _{\rm T}^m - \mu _{\rm C}^m} \right \vert \lt \delta _0$$

Value δ ₀ defines a ‘zone of indifference’ where differences inside it are considered trivial for practical purposes. It establishes similarity, or practical equivalence, but has no direct implications on safety. Because $\mu _{\rm T}^m $ and $\mu _{\rm C}^m $ are parameters in the mixed model analysis of the log-transformed data, [exp (δ ₀) − 1] represents the largest tolerated change of GMO relative to the non-GM counterpart in terms of their back-transformed, i.e. geometric, means. The EFSA guidance used ‘percent change (e.g. −20% and +25%) for relatively small changes, or factors (e.g. ½ and 2) for larger changes’ whereas Oberdoerfer et al. (Reference Oberdoerfer, Shillito, Beuckeleer and Mitten2005) used ±20%. A working δ ₀ could then be set as ln(1.25) or ln(1.20). Hypotheses (4) cannot be used as the sole component for establishing equivalence at a given endpoint, because it does not incorporate crop's natural variation. Hothorn & Oberdoerfer (Reference Hothorn and Oberdoerfer2006) explored several options for defining endpoint-specific ELs. Unfortunately, these definitions fall short in biological validity or statistical rigour.

The current paper addresses natural variation through additional hypotheses that compare GMO with references. As mentioned earlier, crop safety studies employ the multi-site design where various references and GMO are randomized into plots arranged in blocks at each site. Under the linear mixed model of EFSA, natural variation can be easily incorporated into the EL. Vahl & Kang (Reference Vahl and Kang2016) introduced the following hypotheses to assess the equivalence of GMO to references at endpoint m.

(5) $$\hskip--8.15ptH_{{\rm TR}}^m :\left \vert {\mu _{\rm T}^m - \mu _{\rm R}^m} \right \vert \ge \sqrt {\Delta (\vartheta, {\bf \sigma}^m )} \;v.\quad\;K_{{\rm TR}}^m :\left \vert {\mu _{\rm T}^m - \mu _{\rm R}^m} \right \vert \lt \sqrt {\Delta (\vartheta, {\bf \sigma}^m )} $$

Function Δ( · ) corresponds to the EL for $(\mu _{\rm T}^m - \mu _{\rm R}^m )^2$ . Its argument $\vartheta $ is a value to be specified by the regulatory agency. Vector ${\bf \sigma}^m = (\sigma _S^m, \sigma _{{\rm B}(S)}^m, \sigma _{\rm R}^m, \sigma _{\rm E}^m )$ denotes the parameter of natural variation due to site, block-within-site, reference genotype and experimental error terms. Van der Voet et al. (Reference Van der Voet, Perry, Amzal and Paoletti2011) consider ‘appropriate percentiles (e.g. 2·5 and 97·5) of the distribution of reference variety characteristics’ as the limits of $\mu _{\rm T}^m $ . Denote z _0.975 as the 97·5th percentile of the standard normal distribution. Setting $\Delta (z_{0.975},{\bf \sigma} ^m) = z_{0.975}^2 [(\sigma _{\rm R}^m )^2 + (\sigma _{\rm E}^m )^2]$ restricts $\mu _{\rm T}^m $ to be within the 2·5th and 97·5th percentiles of the reference distribution at the site and block level. Vahl & Kang (Reference Vahl and Kang2016) referred to this as conditional scaled average equivalence. Percentiles of the marginal distribution of references are more pertinent to safety. This choice leads to the notion of marginal scaled average equivalence with $\Delta (z_{0.975},{\bf \sigma} ^m) = z_{0.975}^2[(\sigma _{\rm S}^m )^2 + (\sigma _{{\rm B}({\rm S})}^m )^2 + (\sigma _{\rm R}^m )^2 + (\sigma _{\rm E}^m )^2]$ . Nevertheless, current field designs impede the proper quantification of $\sigma _{\rm S}^m $ . Because conditional equivalence implies marginal equivalence, a more practical and conservative EL for hypotheses (5) is $\Delta (z_{0.975},{\bf \sigma} ^m) = z_{0.975}^2 [(\sigma _{\rm R}^m )^2 + (\sigma _{\rm E}^m )^2]$ .

The EFSA guidance also tests non-equivalence, whose hypotheses reverse the null and alternative of the equivalence test. While an improvement over a simple test of zero-difference, this test of non-equivalence is still a mismatch with the proof-of-safety objective. It represents the subconscious reversion to traditional exploratory toxicity studies with a proof-of-hazard purpose (Hothorn & Hasler Reference Hothorn and Hasler2008). Note that $\Delta (\delta, {\bf \sigma} ^m)$ is not an exact rigid threshold for toxicity but rather a conservative soft limit of safety. Non-significant evidence for equivalence logically suggests further evaluation, whereas a conclusion of non-equivalence invites misinterpretation as established true hazard. Moreover, simultaneously testing two opposing objectives (hazard and safety) complicates the statistical decision-making process and hinders straightforward characterization of the consumer's risk. For these reasons, non-equivalence tests should be omitted in crop safety evaluation.

In a three-arm non-inferiority study of drug efficacy, non-inferiority of the test treatment to the active control relies on the assay sensitivity, which can be validated by testing the superiority of the active control over the placebo (Röhmel & Pigeot Reference Röhmel and Pigeot2010 and the literature cited therein). This is founded on the concern that the active control may not be efficacious in the current study. A similar question is whether to compare GMO with references if the non-GM counterpart is not significantly equivalent to references under the same criterion. Unlike the active control and placebo in drug efficacy studies, both the non-GM counterpart and references are safe, a priori. It is unnecessary to test the non-GM counterpart against references and this question is moot.

Grouping hypotheses into families

Tamhane (Reference Tamhane, Ghosh and Rao1996) defines a family as a set of comparisons from which some common conclusions are drawn and states that comparisons within this family must be contextually related but not necessarily statistically dependent. Grouping of individual hypotheses into relevant families should match the study objective and decision rule. Recall that M endpoints are subjected to mixed model analyses. There are 2M individual hypotheses and as many as 2^2M  − 2M − 1 combinations to form composite hypotheses. One possible approach is to place comparisons involving the non-GM counterpart into one family, i.e. $\{ H_{{\rm TC}}^1 \;v.\;K_{{\rm TC}}^1,..., H_{{\rm TC}}^M \;vs.\;K_{{\rm TC}}^M \} $ , and comparisons involving references into another family, i.e. $\{ H_{{\rm TR}}^1 \;v.\;K_{{\rm TR}}^1,..., H_{{\rm TR}}^M \;v.\;K_{{\rm TR}}^M \} $ . The two families each contain M individual hypotheses. The EFSA guidance seems to adopt this approach. Nonetheless, the traditional industry practice and recent regulatory opinions indicate that the safety of a GM crop is evaluated at each specific endpoint (literature listed in Table 1; EFSA 2015a , b , c , 2016). This supports grouping the two comparisons at a given endpoint into a family, i.e. $\{ H_{{\rm TC}}^m \;v.\;K_{{\rm TC}}^m, \;H_{{\rm TR}}^m \;v.\;K_{{\rm TR}}^m \} $ . Consequently, there are M families of hypotheses, each corresponding to an endpoint.

Members of a family can be linked by either intersection or union, which gives rise to two basic types of composite hypotheses: intersection-union (IU) and union-intersection (UI). The IU hypotheses place the intersection of individual alternatives in its alternative and the union of individual nulls in its null. It corresponds to an ‘each-and-every’ type of conclusion. In contrast, the UI hypotheses contain the union of individual alternatives in its alternative and the intersection of individual nulls in its null. It corresponds to an ‘at-least-one’ type of conclusion. Recall that GMO is regarded as safe if it is comparable with either the non-GM counterpart or references. Equivalence at a given endpoint shall then be formulated as the following local UI hypotheses.

(6) $$H^m :H_{{\rm TC}}^m \cap H_{{\rm TR}}^m \;v.\;K^m :K_{{\rm TC}}^m \cup K_{{\rm TR}}^m $$

To those who presume hypotheses (5) as a relaxation of hypotheses (4) (i.e. $K_{{\rm TC}}^m \subset K_{{\rm TR}}^m $ ), hypotheses (6) seem unnecessary since $K_{{\rm TC}}^m \cup K_{{\rm TR}}^m = K_{{\rm TR}}^m $ . However, experience with GMO safety analyses suggests that $K_{{\rm TC}}^m \cup K_{{\rm TR}}^m \supset K_{{\rm TR}}^m $ . Insight on this is gained by depicting hypotheses (4–6) at endpoint m as partitions in a two-dimensional space. Given the known safety of the non-GM counterpart, $\mu _{\rm C}^m $ is expected to be not far from $\mu _{\rm R}^m $ . It is then convenient to express $\mu _{\rm T}^m - \mu _{\rm C}^m $ as $(\mu _{\rm T}^m - \mu _{\rm R}^m ) - (\mu _{\rm C}^m - \mu _{\rm R}^m )$ and examine the parameter space of $(\mu _{\rm T}^m - \mu _{\rm R}^m, \;\mu _{\rm C}^m - \mu _{\rm R}^m )$ . In this way, the entire parameter space of interest, $H^m \cup K^m $ , is simply represented by the horizontal strip in between the two dashed lines in Fig. 3. Boundaries of $H^m \cup K^m $ reflect the fact that the non-GM counterpart is as safe as any non-GM reference variety. They represent the unknown safety limits, which are less stringent than the ELs in hypotheses (5). The $K_{{\rm TC}}^m $ space is represented by the shaded thin parallelogram in Fig. 3(a) and the $K_{{\rm TR}}^m $ space is given by the shaded tall rectangle in Fig. 3(b). Visibly, the $K_{{\rm TC}}^m $ space is not nested within $K_{{\rm TR}}^m $ . Depending on the width of the $K_{{\rm TR}}^m $ rectangle, the $K^m $ space takes two shapes (Figs 3(c) and (d)). The set difference between $K_{{\rm TC}}^m \cup K_{{\rm TR}}^m $ and $K_{{\rm TR}}^m $ , i.e. $(K_{{\rm TC}}^m \cup K_{{\rm TR}}^m )\backslash K_{{\rm TR}}^m $ , is non-empty and serves as a compensation when $\mu _{\rm C}^m $ is not very close to $\mu _{\rm R}^m $ . Particularly, Fig. 3(d) displays the situation of an overly conservative EL. In such a case, hypotheses (5) alone do not adequately represent the concept of substantial equivalence. Testing hypotheses (6) yields a reasonable solution. Tolerance intervals or literature ranges derived from non-GM crops are random quantities with intractable uncertainty. Because their usage is less transparent and prone to manipulation, the present work excludes them from the decision-making process. Figure 4 presents the flow chart for the proposed evaluation. Here the two partial decisions at endpoint m are integrated into a joint decision of overall equivalence by testing the local UI hypotheses. This supports a decision-making process on the basis of pre-specified justification rather than post hoc arguments. It is clearly an improvement over those processes in Figs 1 and 2.

Fig. 3.

Graphic representation of (a) hypotheses (4), (b) hypotheses (5), (c) hypotheses (6) with $\Delta (z_{0.975},{\bf \sigma} ^m)$ representative of the natural variation and (d) hypotheses (6) with a conservative $\Delta (z_{0.975},{\bf \sigma} ^m)$ . Area between the dashed lines is the parameter space of interest. Shaded area represents the alternative space of the corresponding hypotheses.

Fig. 4.

Flow chart for the process that jointly compares genetically modified organisms (GMO) with the non-GM counterpart and references.

Testing hypotheses (6) across M endpoints pertains to a new family of hypotheses. Figure 5 visualizes the organized layout at M = 3. Here the individual hypotheses are grouped into endpoint-specific families of hypotheses, whose UI composites then form a general family. Now the question is how to link members in this general family. Because equivalence at every endpoint implies global equivalence, one might consider the following global IU hypotheses.

(7) $$H:\bigcup\nolimits_{m = 1}^M {H^m} \;v.\;K:\bigcap\nolimits_{m = 1}^M {K^m} $$

Fig. 5.

The organized layout representing the families of hypotheses in establishing substantial equivalence with respect to three endpoints.

Rejecting the null of hypotheses (7) would be a big win for the sponsor. But failing to do so does not result in an automatic regulatory denial, as hazard characterization and exposure assessment will be undertaken for endpoints that cause the null to be retained. Hypotheses (7) do not allow for endpoint-specific conclusions and are therefore a mismatch for crop safety evaluation. Moreover, the type I error in testing hypotheses (7) refers to the assertion of global equivalence when non-equivalence occurs in some endpoints. Its probability does not correspond to the consumer's risk in hazard identification, i.e. the probability of erroneously asserting equivalence on one or more non-equivalent endpoints. According to the theoretical findings of Sonnemann & Finner (Reference Sonnemann, Finner, Bauer, Hommel and Sonnemann1988), strong control on the error of rejecting at least one true H ^m can be achieved by adding the following UI composite hypotheses to the general family (see Fig. 5) and testing each H ^m via the well-established closure principle (a brief introduction on this principle is available in Appendix 1).

(8) $$H^I:\bigcap\nolimits_{m \in I}^{} {H^m} \;v.\;K^I:\bigcup\nolimits_{m \in I}^{} {K^m} $$

The index set I here is a subset of {1, …, M}. For M = 3, I = {1, 2}, {1, 3}, {2, 3}, {1, 2, 3}. Linking K ^m by ‘ $ \cup $ ’ may be difficult for some to comprehend, as these hypotheses are themselves of little practical interest. Nonetheless, expanding the general family to include UI hypotheses allows strong FWER control without compromising the power. In other words, testing such an expanded family of hypotheses is as good as testing the family with individual hypotheses only. It is worthwhile to mention that similar UI formulation of composite hypotheses is becoming increasingly common in multi-endpoint bioequivalence and non-inferiority studies of drugs (Quan et al. Reference Quan, Bolognese and Yuan2001; Logan & Tamhane Reference Logan and Tamhane2008; Hasler & Hothorn Reference Hasler and Hothorn2013; Hua et al. Reference Hua, Xu and D'Agostino2015). This stems from the need for inference on individual endpoints.

The multiple endpoints in drug safety evaluation, i.e. types of adverse events, can be categorized into several body systems on the basis of prior clinical understanding. Such grouping has been used to reduce false positive findings (Berry & Berry Reference Berry and Berry2004; Mehrotra & Heyse Reference Mehrotra and Heyse2004). However, the intertwining relationship of plant metabolic pathways prevents any grouping of endpoints measured in crop safety evaluation. Consequently, there is no intermediate grouping of hypotheses (6) into subfamilies. Only the family containing hypotheses (6) across all endpoints is relevant for GMO testing.

A multiple testing procedure for genetically modified organism safety evaluation

There are three fundamental principles governing multiple testing in the UI context. The closure principle establishes the general method for MTP construction. The sequential rejection principle and the partition principle exploit the relationships among individual hypotheses to improve power. A brief tutorial of these principles and their applications is given in Appendix 1. As mentioned previously, hypotheses (4) and (5) at a given endpoint are not nested. Neither sponsors nor regulatory agencies have articulated any decision path among endpoints measured in crop safety evaluation. This prevents hypotheses (6) from being arranged, in advance, into a fixed sequence. Sequential testing in a data-driven order is therefore the best available approach here. Practitioners accustomed to the analysis of variance setting may be familiar with Dunnett's parametric stepwise procedures used in comparing several treatments with a control (Dunnett & Tamhane Reference Dunnett and Tamhane1991, Reference Dunnett and Tamhane1992). Construction of these MTPs relies on the knowledge that the individual test statistics have a joint multivariate t distribution. The distinct formulation of hypotheses (4) and (5) makes it difficult to characterize such a joint distribution. The Simes-Hochberg (SH) procedure is a classical semi-parametric sequential MTP (Westfall et al. Reference Westfall, Tobias, Rom, Wolfinger and Hochberg1999; Dmitrienko et al. Reference Dmitrienko, D'Agostino and Huque2013). It circumvents the derivation of any joint distribution and is readily applicable to GMO testing. Appendix 1 provides the generic formula for the SH adjustment. Multiplicity adjustment can be carried out by either raising the P values or dropping the significance levels of individual tests. Because these two approaches yield identical conclusions, only the P-value-based approach is presented.

To accommodate the hypothesis layout in Fig. 5, the present work proposes a two-stage MTP that executes the SH adjustment in both stages. The first stage derives the raw P value for testing hypotheses (6) at each endpoint. Suppose that the ordered P values for testing hypotheses (4) and (5) at endpoint m are $P_{(1)}^m $ and $P_{(2)}^m, P_{(1)}^m \le P_{(2)}^m $ . According to the SH adjustment, the raw P value for testing hypotheses (6) is:

(9) $$P^m = \min (2P_{(1)}^m, P_{(2)}^m )$$

Let $P^{(m)} $ be the m ^th smallest P value of $\{ P^1,..., P^M\} $ , i.e. $P^{(1)} \le \ldots \le P^{(M)}$ . The second stage controls the consumer's risk by correcting for multiplicity across endpoints. The SH method is applied for a second time and P values for hypotheses (6) are adjusted sequentially as:

(10) $$\eqalign{&\tilde{P}^{(m)} =\cr & \left\{\matrix{P^{(M)} &m = M \cr \min [\tilde{P}^{(m + 1)}, (M - m + 1)P^{(m)}] &1 \le m \le M - 1} \right.}$$

To preclude logical incoherence, this present work applies the closure principle and readjusts P values for individual hypotheses as:

(11) $$\eqalign{& \tilde P_{(1)}^{(m)} = \max (\tilde P^{(m)}, P_{(1)}^{(m)} ) \cr & \tilde P_{(2)}^{(m)} = \max (\tilde P^{(m)}, P_{(2)}^{(m)} )} $$

This assures that at a given endpoint, the P value for testing hypotheses (6) will be no greater than either of the two for testing hypotheses (4) and (5). Consequently, if the null of hypotheses (6) is retained, the corresponding nulls of hypotheses (4) and (5) are both retained.

The SH procedure has been demonstrated to strongly control the FWER for a general class of positively correlated multivariate distributions, many of which are encountered in multiple testing situations (Sarkar & Chang Reference Sarkar and Chang1997; Sarkar Reference Sarkar1998, Reference Sarkar, Balakrishnan, Peña and Silvapulle2008). Excessive ‘handwringing’ on its analytical conditions is counterproductive in practice. Biological rationale for the positive relationship among multiple inferences should alleviate any concern on its validity here. In particular, proper selection of GMO comparators, as articulated by EFSA (2011), assures $\mu _{\rm C}^m $ to be fairly close to $\mu _{\rm R}^m $ . Results of the two within-endpoint comparisons are therefore expected to be positively correlated. Owing to the intricacy of plant system biology, a positive relationship is also expected from results of equivalence testing across endpoints. It is therefore anticipated that the proposed MTP will restrain the FWER effectively. The large number of hypotheses tested (2M >100) obstructs non-parametric adjustments carried out either by the single-step Bonferroni (SSB) procedure or the sequential Bonferroni-Holm (BH) procedure. A similar viewpoint has been expressed by Hothorn & Oberdoerfer (Reference Hothorn and Oberdoerfer2006), who employed the SH procedure in their multiple testing of hypotheses (4) across compositional endpoints. For side-by-side comparisons of these procedures for a single family of hypotheses, see the corresponding formulas in Appendix 1. Let α be the desired FWER. The benefit of the proposed MTP is best seen when P ¹, …,P ^M are all less than α. In this case, some of SSB- or BH-adjusted P values might be greater than α, which seems counterintuitive. The SH-adjusted P values, on the other hand, would always remain less than α. Such consistency among outcomes was recognized by Dmitrienko et al. (Reference Dmitrienko, D'Agostino and Huque2013) as a desirable property. It has served as an incentive for Quan et al. (Reference Quan, Bolognese and Yuan2001) and Hasler & Hothorn (Reference Hasler and Hothorn2013) to apply the SH adjustment in clinical equivalence and non-inferiority studies.