Definition of Single-Case Experiment

If we want a generic model for the analysis and meta-analysis of SCEs, then it is crucial to be crystal clear about the kind of data we are dealing with. We need to know exactly what the experimental units are, which units are considered in the statistical analysis, and which units are considered in the meta-analysis. In this first section, we reflect on the SCE definition and comment on the essential ingredients and their implications.

SCEs are defined as ‘experiments in which one entity is observed repeatedly during a certain period of time, under different levels (‘treatments’) of at least one independent variable’ (Onghena, Reference Onghena, Everitt and Howell2005, p. 1850). There are two essential features in this definition. First, only one entity is involved (it is a single case), and second, there is a manipulation of the independent variable(s) (it is an experiment). Two other features are consequences of this set-up: the entity is exposed to all levels of the independent variable (like in a within-subject design) and there are repeated observations or measurements (like in a longitudinal or time series design).

For the purpose of this paper it is important to reflect on the main terms and implications of this definition. We summarise our reflection in seven comments:

1. 1. The definition refers to the generic term ‘entity’ to emphasise that any experimental unit at any level of aggregation may be involved. In the health sciences, the entity usually is a patient but the entity can also be a group of patients in a hospital ward, a part of a patient (e.g., a brain region), or a more abstract unit, such as a rehabilitation centre as an organisation (Barlow, Nock, & Hersen, Reference Barlow, Nock and Hersen2009; Kazdin, Reference Kazdin2011; Onghena, Reference Onghena, Everitt and Howell2005).

2. 2. The definition and the following discussion of statistical techniques are restricted to ‘experiments’. By referring to one or more manipulated variables, SCEs are distinct from qualitative and descriptive case studies and from observational time series studies (Onghena & Edgington, Reference Onghena and Edgington2005; Onghena & Struyve, Reference Onghena, Struyve, Balakrishnan, Brandimarte, Everitt, Molenberghs, Piegorsch and Ruggeri2015).

3. 3. Notice that the ‘observed repeatedly’ in the definition does not need to refer to measurements of quantitative attributes. Also qualitative and naturalistic observations are possible without losing the experimental nature of the SCE. However, if the data have to be analysed statistically, all observations have to be quantitatively coded in one or more outcome variables. In the remainder of this article, we assume that measurement of a quantitative attribute is at the basis of at least one outcome variable. This is the most characteristic way that SCEs are presented, and is most feasible with the rising popularity of health care wearables and ecological momentary assessment (McDonald & Davidson, Reference McDonald and Davidson2016; Piwek, Ellis, Andrews, & Joinson, Reference Piwek, Ellis, Andrews and Joinson2016).

4. 4. The internal and statistical-conclusion validity of an SCE can be strengthened by including randomisation in the design (Dugard, File, & Todman, Reference Dugard, File and Todman2011; Edgington, Reference Edgington1996; Heyvaert, Wendt, Van den Noortgate, & Onghena, Reference Heyvaert, Wendt, Van den Noortgate and Onghena2015; Kratochwill & Levin, Reference Kratochwill and Levin2010), but randomisation is not a necessary feature in the SCE definition. Randomisation can be performed with respect to the treatment order or the moments of phase change or a combination of both (Levin, Ferron, & Gafurov, Reference Levin, Ferron and Gafurov2014; Onghena, Vlaeyen, & de Jong, Reference Onghena, Vlaeyen, de Jong and Sawilowsky2007).

5. 5. External validity and generalisability to other patients is not always the first aim of an SCE (Barlow et al., Reference Barlow, Nock and Hersen2009; Gast & Ledford, Reference Gast and Ledford2014; Kazdin, Reference Kazdin2011). In a purely idiographic approach, the clinician might just be interested to understand and help the single patient under investigation, and may use a customised treatment and a customised symptoms checklist. However if generalisability is an aim, it can be accomplished by replications using identical (or similar) treatments and outcome measures. Replicated demonstrations of a similar treatment effect in consecutive patients provide evidence for an identical underlying process or a general law.

6. 6. Many experimental design schedules are possible for SCEs. A useful classification has been proposed by Shamseer et al. (Reference Shamseer, Sampson, Bukutu, Schmid, Nikles and Tate2015), Tate et al. (Reference Tate, Perdices, Rosenkoetter, McDonald, Togher, Shadish and Vohra2016a, Reference Tate, Perdices, Rosenkoetter, Shadish, Vohra, Barlow and Wilson2016b), and Vohra et al. (Reference Vohra, Shamseer, Sampson, Bukutu, Schmid and Tate2015). They distinguish:

   • • Withdrawal or reversal designs (such as ABA, ABAB, ABACAD with each letter representing a phase of repeated measurements for the same level of the manipulated variable),

   • • Alternating treatments designs

   • • Multiple baseline designs, and

   • • Changing criterion designs.

   A two-phase AB design is not included in this classification, although the use of this design in an experimental context would match the definition of an SCE. Our proposal would be to consider studies using an AB design as SCEs, but to acknowledge that this design is the poorest design in terms of methodological quality. Two-phase AB designs suffer from the fact that there is only one demonstration of the treatment effect and that effect might be due to an external event accidentally coinciding with the moment of phase change (Kratochwill et al., Reference Kratochwill, Hitchcock, Horner, Levin, Odom, Rindskopf and Shadish2010, Reference Kratochwill, Hitchcock, Horner, Levin, Odom, Rindskopf and Shadish2013; Tate et al., Reference Tate, Aird and Taylor2013). This should not mean that applied studies using an AB design are, by definition, poorest in terms of methodological quality. SCEs using an AB design may involve methodological add-ons to be able to provide solid evidence, such as a randomly timed intervention, a large number of reliable observations, and replications across patients to obtain sufficient statistical power (Barlow et al., Reference Barlow, Nock and Hersen2009; Heyvaert et al., Reference Heyvaert, Hannes and Onghena2017; Heyvaert, Wendt, Van den Noortgate, & Onghena, Reference Heyvaert, Wendt, Van den Noortgate and Onghena2015).

7. 7. In the medical literature, the term ‘N-of-1 trial’ has been adopted to refer to a prospectively planned, multiple crossover trial in a single patient to determine the effect of an intervention (Guyatt, Jaeschke, & McGinn, Reference Guyatt, Jaeschke, McGinn, Guyatt, Rennie, Meade and Cook2002; Schork, Reference Schork2015; Shamseer et al., Reference Shamseer, Sampson, Bukutu, Schmid, Nikles and Tate2015; Vohra et al., Reference Vohra, Shamseer, Sampson, Bukutu, Schmid and Tate2015). N-of-1 trials are ‘biomedical’ SCEs, usually using a withdrawal or reversal design. Furthermore, N-of-1 trials often evaluate pharmacological treatments and include placebo control, randomisation, and blinding (Onghena, Reference Onghena and Berger2007, Reference Onghena and Berger2016). Consequently N-of-1 trials can be considered a subset of SCEs (Tate et al., Reference Tate, Perdices, Rosenkoetter, McDonald, Togher, Shadish and Vohra2016a, Reference Tate, Perdices, Rosenkoetter, Shadish, Vohra, Barlow and Wilson2016b).

An Example

An example of a replicated SCE with two patients was published in this journal by Douglas, Knox, De Maio, and Bridge (Reference Douglas, Knox, De Maio and Bridge2015). They used an AB design with follow-up to evaluate the effectiveness of a new treatment, Communication-specific Coping Intervention (CommCope-I), which targets coping in the context of communication breakdown after traumatic brain injury. The study was carried out with two patients: Samantha,Footnote ¹ a 30-year-old woman, and Thomas, a 34-year-old man, who both had sustained severe traumatic brain injury several years before, and still experienced substantial communication problems. The primary outcome variable in this study was the Discourse Coping Scale – Clinician Rating, which uses a 10-cm visual analogue scale to measure the appropriate use of communication-specific coping strategies.

The raw data of the first A phase and the B phase are shown in Tables 1 and 2 for Samantha and Thomas respectively. As can be seen in the tables all scores in the B phase are larger than the scores in the A phase and the authors conclude that: ‘This study provides sound phase-1 evidence for the effectiveness of CommCope-I’. (p. 190).

TABLE 1

Weekly Scores on the Discourse Coping Scale – Clinician Rating for Samantha During a 9-Week SCE with a 3-Week Baseline (A phase) and a 6-Week Communication-Specific Coping Intervention (B phase)

Note. There was one missing week and data point in the A phase.

TABLE 2

Weekly Scores on the Discourse Coping Scale – Clinician Rating for Thomas During a 9-Week SCE with a 4-Week Baseline (A phase) and a 6-Week Communication-Specific Coping Intervention (B phase)

In the remainder of this article we will use the data in Tables 1 and 2 for a simple numerical demonstration of the statistical analysis and meta-analysis of SCE data. In most applications, the data are more numerous, complicated, diverse or fragmented and we will reflect on a few of these complicating issues (and their solutions) in the Discussion section. Furthermore, Table 1 and Table 2 contain two simplifications compared to the original data in Douglas et al. (Reference Douglas, Knox, De Maio and Bridge2015):

1. 1. We ignored the missing data point in the A phase of Samantha because dealing with missing data is a separate and complicated issue that would distract from the main ideas in the demonstration.

2. 2. Douglas et al. (Reference Douglas, Knox, De Maio and Bridge2015) referred to their design as ‘an ABA design’ (p. 194), but as the authors acknowledged: ‘The withdrawal design is somewhat limited in the case of skill-based interventions like CommCope-I which are designed to facilitate lasting change in behaviour and are thus not strictly amenable to evaluation using return to baseline single-case designs’. (p. 199) If the data in the follow-up phase were modelled as ‘a return to baseline phase’ then the results would be opposite to what is expected. Consequently, for both Samantha and Thomas, we did not include the follow-up data to avoid this interpretational pitfall.

We will also return to the implications of these simplifications and refer to some statistical options to deal with missing data and withdrawal designs in the Discussion section. The Appendix presents R code for all computations that are used in this manuscript.

A Simple Statistical Model for the AB Design: Model 1

Taking into account the essential and implied features of the SCE definition, a simple statistical model for data from a single patient considers the patient as the population and the repeated measurements as a random sample (i.e., realisations of a random outcome variable). In an AB design, the manipulated variable takes two values: one value for the baseline phase and one value for the treatment phase.

Given these features, some authors have been analysing data from SCEs using the following simple statistical model (Gentile, Roden, & Klein, Reference Gentile, Roden and Klein1972; Shine & Bower, Reference Shine and Bower1971):

with

• Score _t referring to the value of the outcome variable at time t; the outcome variable Score indicates the repeated measurements, with the index t going from 1 to N (for N repeated measurements available from the SCE),

• (Treat) _t , referring to the manipulated variable, taking a value of 0 for each measurement occasion in the baseline phase and a value of 1 for each measurement occasion in the treatment phase,

• β₀ referring to the baseline mean (i.e., the value of Score _t when (Treat) _t and ε _t equal 0),

• β₁ referring to the difference between the treatment mean and the baseline mean (i.e., the treatment effect),

• ε _t referring to the residual at time t, which is the deviation of Score _t from the average value of the outcome variable in the corresponding phase, ε _t = Score _t − [β₀ + β₁(Treat) _t ].

The values for Score _t are observed by conducting the study and the values of (Treat) _t are determined by the chosen design; the values for β₀ and β₁ have to be derived following a particular optimisation criterion. The classical optimisation criterion is the minimisation of the squared residuals ε² _t , resulting in a so-called ‘Ordinary Least Squares’ (OLS) regression (or a common Analysis of Variance, using slightly different notation).

For the data of Samantha in Table 1, we have N equal to 9, and the means and standard deviations as shown in Table 3.

TABLE 3

Summary Statistics for the Samantha Data in Table 1

For Model 1 we obtain β₀ = 4.77 and β₁ = 2.75 (see the first two columns of Table 4), so there is an average addition of 2.75 points for the scores in the B phase as compared to the A phase, which can also be verified in Table 3 (4.77 + 2.75 = 7.52). Notice that no distributional assumptions are needed for the derivation of the OLS estimates.

TABLE 4

Parameter Estimates (and Standard Errors) of the Models Specified in Equations 1–3 for the Samantha Data in Table 1

Note. *p < .05. **p < .01.

A distributional model is usually invoked if questions like: ‘Is this value of β₁ = 2.75 statistically significantly different from zero?’ or ‘What is the 95% confidence interval for this treatment effect estimate?’ need an answer. The most common model arises if the ε _t values are assumed to be independent and identically distributed Gaussian deviates with a mean of 0 and an unknown variance σ² _ε (ε _t ~ N(0, σ² _ε)). Under this assumption, the treatment effect for the Samantha data is considered statistically significant, β₁ = 2.75, t(7) = 3.22, p = .0146, with [0.73; 4.77] being the 95% confidence interval.Footnote ² Notice that the same t- and p-values would be obtained if a classical two-sample Student t test was performed or an equivalent F (= t²) and identical p-value for an Analysis of Variance with two independent groups.Footnote ³ Notice also that the statistical test for the β₀ parameter is usually not relevant because it is testing a hypothesis that is irrelevant for most applications, namely the null hypothesis that the baseline mean is zero, H₀: β₀ = 0.

The assumption with regard to the residuals can be relaxed by only assuming that the residuals are exchangeable, so without specifying any population distributional shape. In that case an exact regression permutation test can be performed (Ottoboni, Lewis, & Salmaso, Reference Ottoboni, Lewis and Salmaso2017; Pesarin & Salmaso, Reference Pesarin and Salmaso2010; Wheeler & Torchiano, Reference Wheeler and Torchiano2016), with a result that is very similar to the Gaussian theory test, perm t(7) = 4.037, p = .0238. This value can easily be verified because there is no overlap between the scores in the two phases. In that case, the observed test statistic provides the most extreme value and the exact one-sided permutation test p-value can be obtained by inverting the binomial coefficient ${( \scriptsize{\begin{array}{@{}*{1}{c}@{}} N\\ n \end{array} )^{ - 1}}}$ . The exact two-sided permutation test p-value can be obtained by doubling this value. For the Samantha data 2/ $(\scriptsize{\begin{array}{@{}*{1}{c}@{}} 9\\ 3 \end{array}} )$ = .0238. This procedure is equivalent to the procedure in a two-sample permutation test (Edgington & Onghena, Reference Edgington and Onghena2007; Fay, Reference Fay2010), so also for this test the same p-value of .0238 would be obtained.

Modelling Trend: Model 2

Equation (1) is overly simplistic because it only models a difference between the A phase average and the B phase average. By contrast, an SCE always involves a time component and usually there is an evolution in time within the phases, a so-called ‘trend’ (Shadish & Sullivan, Reference Shadish and Sullivan2011; Solomon, Reference Solomon2014). Therefore, a more appropriate statistical model for time series data includes a parameter β₂ to capture this trend:

with

• Score _t and (Treat) _t identical as in Equation 1,

• (Time) _t referring to the value of the time variable at time t; for the models in this manuscript the time variable takes the same values as the index t (viz, from 1 to N for Week 1 to Week N), but for more sophisticated models other values can be used,Footnote ⁴

• β₀ referring to the intercept, that is the expected value of the outcome variable Score when (Treat) _t and (Time) _t are zero,

• β₁ referring to the change in the expected value of the outcome variable Score when going from the baseline phase to the treatment phase (i.e., the additive treatment effect),

• β₂ referring to the change in the expected value of the outcome variable for each unit change in (Time) _t (i.e., the trend), and

• ε _t referring to the residual at time t, which is the deviation of Score _t from the best fitting equation at time t, ε _t = Score _t − [β₀ + β₁(Treat) _t + β₂(Time) _t ].

A trend can most conveniently be spotted by making a time series plot of the data. Figure 1 shows the time series plot of the Samantha data. There seems to be a steady increase in Samantha's communication from Week 1 to Week 9, but it is not clear whether that increase is statistically significant. The parameter estimates and their standard errors for this Equation (2) can be found in Table 4, in the column with ‘Model 2’ as the header.

FIGURE 1

Weekly scores on the Discourse Coping Scale – Clinician Rating for Samantha during a nine-week SCE with a three-week baseline (A phase) and a 6-week Communication-specific Coping Intervention (B phase).

If we assume that ε _t ~ N(0, σ² _ε) then the additive treatment effect for the Samantha data is not considered statistically significant anymore, β₁ = 0.94, t(6) = 0.70, p = .5109, with [−2.35; 4.22] being the 95% confidence interval after taking the linear trend into account. Also the trend itself is not statistically significant, β₂ = 0.40, t(6) = 1.64, p = .1518, with [−0.20; 1.00] being the 95% confidence interval. The permutation test p-values are .5050 and .1550 (see Wheeler & Torchiano, Reference Wheeler and Torchiano2016, and the Appendix for the calculation of the permutation test p-values).

Notice that the β₁ estimate in Equation (2), 0.94, is smaller than the β₁ estimate in Equation (1), 2.75. The reason for this difference is that Equation (2) also includes a trend parameter β₂ and that in the estimation all components of the equation are taken into account. The linear trend estimate in this example indicates that there is a score increase of β₂ = 0.40 for each unit of change in the time variable, even if there is no additive treatment effect. For the A phase, (Treat) _t = 0, this means that the expected scores are: 3.96 + (1)(0.40) = 4.36, 3.96 + (2)(0.40) = 4.76, and 3.96 + (3)(0.40) = 5.16. The average of these three numbers is 4.76, which is the A phase average (within rounding error) that can be found in Table 3. For the B phase, (Treat) _t = 1, the expected scores from Equation (2) become: 3.96 + 0.94 + (4)(0.40) = 6.5, 3.96 + 0.94 + (5)(0.40) = 6.9, 3.96 + 0.94 + (6)(0.40) = 7.3, 3.96 + 0.94 + (7)(0.40) = 7.7, 3.96 + 0.94 + (8)(0.40) = 8.1, and 3.96 + 0.94 + (9)(0.40) = 8.5. The average of these six numbers is 7.5, which is the B phase average (within rounding error) that can be found in Table 3. This demonstrates that the phase averages following Equation (2) are constructed by adding three components: an intercept, a time trend, and an additive treatment effect. Consequently, the β₁ estimate in Equation (2) should be interpreted differently than the β₁ estimate in Equation (1). In Equation (2) the β₁ estimate does not represent the raw difference between the phase averages but rather represents the score that is added in the B phase after taking the linear trend into account.

Another important difference between the results using Equation (2) as opposed to the results using Equation (1) is that the treatment effect has become statistically non-significant. This means that the treatment effect cannot be distinguished from the mere increment of the scores as a function of time. Although Figure 1 may give the impression that there is an additive treatment effect in addition to a small trend, none of these effects is large enough to be statistically significant. The observed data using this design and this (small) number of repeated measures are not convincing enough to attribute the larger average in the B phase to either an additive treatment effect or to a trend.

Modelling Serial Dependency: Model 3

A simplifying assumption that we made in the previous analyses is that the residuals are independent or exchangeable, given that all relevant parameters are included in the model. This assumption is not obvious for time series data (Harrington & Velicer, Reference Harrington and Velicer2015; Jones, Weinrot, & Vaught, Reference Jones, Weinrot and Vaught1978) and therefore Equation (2) is sometimes extended using an autoregressive parameter for the residuals:

and with all the other parameters as defined in Equation (2).

The results are shown in Table 4, in the column with ‘Model 3’ as the header. The autoregressive parameter appears to be very small, and the results for Model 3 are essentially identical to the results for Model 2.

More Complex Statistical Models for the AB Design

It is possible to extend Equations (2) and (3) in several ways. An elaboration of these extensions is beyond the scope of the present article, but we give some references for the reader who is interested to explore these extensions.

A first possible extension is the inclusion of an interaction term (Treat) _t × (Time) _t to assess whether the trend changes from the baseline to the treatment phase. Issues regarding specification of the linear model, coding of the variables and interpretation are presented in a very accessible way by Huitema and McKean (Reference Huitema and McKean2000).

A second possible extension is the inclusion of non-linear terms with respect to time, such as (Time²) _t to model quadratic trends. The interested reader is referred to Aiken, West, and Pitts (Reference Aiken, West, Pitts, Schinka and Velicer2003).

A third possible extension is the inclusion of additional terms to model the serial dependency. A very general methodology to accomplish this involves the use of so-called ‘Autoregressive Integrated Moving Average’ models (see Box, Jenkins, Reinsel, & Ljung, Reference Box, Jenkins, Reinsel and Ljung2016, for a modern presentation of this methodology). Those models make provision for higher-order autoregressive parameters and moving average parameters, based on repeated cycles of model identification, parameter estimation, and diagnostic checking. Furthermore, lagged predictor variables can be added to examine the relationship between the outcome variable and other manipulated or non-manipulated predictor variables with possible time delays. For example, if data about weekly mood were collected, a third predictor variable (Mood) _{t − 1}, with the corresponding parameter β₃, could be added to Equation (3) to examine the relationship between communication at week t and Mood the week before (t – 1).

For some applications, those extensions may be relevant, but essentially any modelling attempt will boil down to the three basics: modelling the treatment effect, modelling the trend and modelling the serial dependency, as demonstrated in the previous sections. Finally, notice that an important restriction in Equations (1), (2) and (3) is that a numerical outcome variable is considered. If the outcome variable is categorical, or even dichotomous, generalised models are needed (McCullagh & Nelder, Reference McCullagh and Nelder1989).

Meta-Analysis: Modelling the Nested Structure for Replicated AB Designs in Models 4, 5 and 6

The same analyses can be performed on Thomas's data and the results can be found in Figure 2, Tables 5 and 6. For Thomas, Model 3 indicates a statistically significant treatment effect, a statistically significant linear trend, and a statistically significant negative autoregressive parameter.

FIGURE 2

Weekly scores on the Discourse Coping Scale – Clinician Rating for Thomas during a 10-week SCE with a 4-week baseline (A phase) and a 6-week Communication-specific Coping Intervention (B phase).

TABLE 5

Summary Statistics for the Thomas Data in Table 2

TABLE 6

Parameter Estimates (and Standard Errors) of the Models Specified in Equations 1–3 for the Thomas Data in Table 2

Note. *p < .05. **p< .01.

In this example, there are two patients, but in general, there might be J patients and for each patient j (j = 1, . . ., J) such a separate analysis can be performed. Moreover, if J is large enough and the SCEs are similar enough in design and research focus, then it might also be interesting to combine and compare the results of these multiple patients. Such a pooled analysis is called a ‘meta-analysis’ because it is an analysis at a higher level. Notice that the SCEs can be conducted simultaneously (as in multiple baseline designs across participants), at different points in time (as in replicated AB designs), at different research institutes, and the data may be extracted from different publications.

For the meta-analysis of these SCEs we will present the multilevel approach proposed by Van den Noortgate and Onghena (Reference Van den Noortgate and Onghena2003a, Reference Van den Noortgate and Onghena2003c). For illustration purposes, we will use the data from Samantha and Thomas, but judicious application of this methodology should involve samples much larger than two (see Discussion section for the factors involved in determining the appropriate number of patients). In addition, we will assume that all raw data are available; an assumption that is not unrealistic for SCEs because in the SCE literature it is good practice to have the raw data published in a table or a graph.Footnote ⁵ Alternatively researchers can be asked to share the raw data upon request. If the raw data are not available and the meta-analysis has to be performed on summary statistics then the effect size model proposed by Van den Noortgate and Onghena (Reference Van den Noortgate and Onghena2003b) can be used.

For the meta-analysis, we have to adjust our notation. In order to make a distinction between the models for Samantha and the models for Thomas, an additional index has to be used to specify the particular patient that is involved. So for Samantha, Equation (1) becomes Score _t1 = β₀₁ + β₁₁(Treat) _t1 + ε _t1 and for Thomas Score _t2 = β₀₂ + β₁₂(Treat) _t2 + ε _t2. In general, for J patients, Equation (1) becomes

Furthermore, if we want to include the data of all patients in one overall meta-analysis, we have to take the nested data structure into account: there are J patients and the repeated measures are nested within the patient. So for a meta-analysis, we are considering a sample from a population of patients and in each patient a random sample of repeated measurements. The additional index j is used to indicate that all values can differ between patients, so also the parameters can vary. The variation of the parameters can be modelled with a set of regression equations at the second level:

Together Equations (4a) and (4b) constitute a simple meta-analytical model for SCEs. In this model γ₀ indicates the population average score for the baseline and γ₁ the difference between the population treatment average and the population baseline average. The residuals u _0j and u _1j indicate the deviations between the parameter values for each patient j and the value of the population parameter. By assuming that these residuals follow a multivariate normal distribution with zero means, variances σ² ₀ and σ² ₁, and covariance σ₀₁, in symbols this is:

we can perform statistical tests and construct confidence intervals.

The parameter estimates and standard errors for this Model 4 can be found in Table 7 Footnote ⁶ . In this meta-analytical model, the average treatment effect is considered statistically significant, γ₁= 3.24, t(16) = 4.09, p = .0009, with [1.56; 4.92] being the approximate 95% confidence interval. The estimated standard deviations of the random effects indicate that there is little difference between the patients with respect to their average (σ₀ = 0.04), and that there are larger differences between the patients with respect to the treatment effect (σ₁ = 0.88) and between the repeated measurements after taking the treatment effect into account (σ_ε = 1.03).

TABLE 7

Model Estimates (and Standard Errors) for the Multilevel Meta-Analysis Containing both the Samantha and Thomas Data in Tables 1 and 2

Note 1. *p < .05. **p < .01.

Note 2. Standard errors for the standard deviations of the random effects are not reported because the sampling distribution of these standard deviations is strongly asymmetrical (Pinheiro & Bates, Reference Pinheiro and Bates2000). Certainly for only two patients this standard error would be a very misleading measure of uncertainty. See Bates (Reference Bates2010) for an alternative approach.

^†σ_ω for Model 6.

Similarly, in a meta-analytical context, Equation (2) is extended to a model with three equations at the second level (one additional equation for the differences in time trend between the patients):

and Equation (3) becomes

The results for Model 5 show that the treatment effect is again not statistically significant anymore, γ₁= 1.46, t(15) = 1.68, p =.1141, once the trend is taken into account (Table 7). The approximate 95% confidence interval for the treatment effect is [−0.40; 3.32], which obviously contains zero. The time trend is statistically significant, γ₂= 0.38, t(15) = 2.82, p =.0129, with an approximate 95% confidence interval of [0.09; 0.66], which means that there is a general upward trend. The standard deviations of the random effects show the same pattern as in Model 5, with the addition that there is little difference between the patients with respect to this general time trend (σ₂ < 0.01). As can be seen in Table 7, the results for Model 6 are essentially identical to the results for Model 5, due to the small (and non-significant) value for the autoregressive parameter (φ = −0.03).

According to Douglas et al. (Reference Douglas, Knox, De Maio and Bridge2015, p. 190), these SCEs provide ‘sound phase-1 evidence for the effectiveness of CommCope-I’ (p. 190), but our analysis demonstrates that with this simple design, with this statistical model, only two SCEs, and these data, it is impossible to convincingly distinguish between the treatment effect and the general time trend. Instead of calling it ‘sound phase-1 evidence’ it would be more cautious to label these results as ‘promising’. Of course, we have to acknowledge that our analysis was also tentative, based on questionable assumptions, and for demonstration purposes only. When readers would look at the complete Douglas et al. (Reference Douglas, Knox, De Maio and Bridge2015) study, including the follow-up data and more qualitative information, and they would perform an analysis based on alternative statistical models (e.g., including a (Treat) _t × (Time) _t interaction), then it cannot be ruled out that other conclusions are drawn.

Meta-Analysis: More Advanced Issues and Tools

The multilevel meta-analytic model can be extended in several ways. Without going into the technical details, we mention nine additions and variations, and provide the references for the interested reader:

1. 1. Usually a multilevel meta-analysis is performed not only to combine the patient results, as in the example, but also to compare the patients. Formally, this comparison is carried out by adding a moderator variable at the higher-level regression equation. For example, a comparison between men and women with respect to the treatment effect in Equation (4b) is accomplished by dummy coding men and women in the data set and extending the Equation as β_1j = γ₁₀ + γ₁₁(Sex) _j + u _1j . Notice that there is no t index for this additional variable because sex only varies between patients (i.e., in the second-level equation); this variable is a characteristic of the patients, not a characteristic of the measurement occasions. Notice also that it is possible to include numerical moderator variables. For example, the patient's age can be added and also interactions between sex and age examined: β_1j = γ₁₀ + γ₁₁(Sex) _j + γ₁₂(Age) _j + γ₁₃[(Sex) _j × (Age) _j ] + u _1j . If there is substantial variation in the lower-level parameter between patients (e.g., the β₁’s), then the addition of moderators at the higher level will allow a better understanding of how this variation comes about (Baek et al., Reference Baek, Moeyaert, Petit-Bois, Beretvas, Van den Noortgate and Ferron2014; Van den Noortgate & Onghena, Reference Van den Noortgate and Onghena2003a; Van den Noortgate & Onghena, Reference Van den Noortgate and Onghena2008).

2. 2. Multilevel meta-analysis of SCEs need not be restricted to two levels (repeated measurements within patients) as in the Samantha-Thomas example. The model can also be applied to three or more levels, for example, when a literature review involves multiple SCEs nested within each publication (three levels: repeated measurements within patients within publications) (Moeyaert, Ferron, Beretvas, & Van den Noortgate, Reference Moeyaert, Ferron, Beretvas and Van den Noortgate2014; Moeyaert, Ugille, Ferron, Beretvas, & Van den Noortgate, Reference Moeyaert, Ugille, Ferron, Beretvas and Van den Noortgate2013a, Reference Moeyaert, Ugille, Ferron, Beretvas and Van den Noortgate2013b).

3. 3. Multilevel meta-analysis of SCEs need also not be restricted to one outcome variable. In many applications, there are multiple outcome variables or multiple effect size measures and the model has to be adapted accordingly. This can be accomplished in a multilevel meta-analysis by including the outcome variable as a separate level in the model (Moeyaert et al., Reference Moeyaert, Ugille, Beretvas, Ferron, Bunuan and Van den Noortgate2016; Van den Noortgate, López-López, Marín-Martínez, & Sánchez-Meca, Reference Van den Noortgate, López-López, Marín-Martínez and Sánchez-Meca2013, Reference Van den Noortgate, López-López, Marín-Martínez and Sánchez-Meca2015).

4. 4. Multilevel meta-analysis of SCEs is also applicable to other designs than replicated two-phase AB designs (Moeyaert et al., Reference Moeyaert, Ugille, Ferron, Onghena, Heyvaert and Van den Noortgate2015). Moeyaert et al. (Reference Moeyaert, Ferron, Beretvas and Van den Noortgate2014) showed how design matrices can be specified for reversal designs, alternating treatments designs, and multiple-baseline designs. Brosnan et al. (Reference Brosnan, Moeyaert, Brooks Newsome, Healy, Heyvaert, Onghena and Van den Noortgate2017), for example, used such an extended multilevel model for multiple-baseline data in a study evaluating precision teaching to improve the fluency in foundational reading skills for at risk readers.

5. 5. In the example, we performed the analyses at the individual patient level first, and afterwards combined the two analyses in a two-level meta-analysis. The other way around, the two-level meta-analysis also enables a more accurate estimation of the patient-level treatments effects afterwards. The advantage of these ‘empirical Bayes estimates’, using a weighted average of the estimate based on the individual patient data and an estimate based on the average of all patients’ data, is that for short time series data at the patient level it is still possible to obtain reliable estimates by ‘borrowing strength’ from the average estimate (Ferron, Farmer, & Owens, Reference Ferron, Farmer and Owens2010; Van den Noortgate & Onghena, Reference Van den Noortgate and Onghena2003a).

6. 6. As mentioned before, multilevel meta-analysis of SCEs can also handle data sets with only descriptive statistics or effect sizes available. Without having access to the raw data, a multilevel meta-analysis can still be performed if sufficient information is available to estimate the sampling variance of the statistics prior to performing the meta-analysis (Ugille, Moeyaert, Beretvas, Ferron, & Van den Noortgate, Reference Ugille, Moeyaert, Beretvas, Ferron and Van den Noortgate2012; Van den Noortgate & Onghena, Reference Van den Noortgate and Onghena2003b).

7. 7. Because multilevel meta-analysis of SCEs usually involves consecutive patients or consecutive publications, techniques from sequential analysis can be applied. Kuppens and Onghena (Reference Kuppens and Onghena2010) and Kuppens, Heyvaert, Van den Noortgate, and Onghena (Reference Kuppens, Heyvaert, Van den Noortgate and Onghena2011) proposed such a sequential meta-analysis of SCEs, which focuses on the question whether enough cumulative knowledge has been collected to draw convincing statistical conclusions and when data collection can be stopped. Determining the sufficiency of cumulative knowledge avoids the inefficient use of limited resources and prevents the dissemination of treatments with dubious benefits (Kuppens & Onghena, Reference Kuppens and Onghena2012).

8. 8. Multilevel meta-analysis of SCEs can be completed by performing sensitivity analyses and publication bias analyses in a similar way as with standard meta-analysis (Cooper, Hedges, & Valentine, Reference Cooper, Hedges and Valentine2009). Sensitivity analyses are used to examine whether certain researcher decisions (e.g., ways of handling missing data, adding a moderator variable or not) distort the conclusions and the inferences drawn from the analysis. Publication bias analyses are used to examine whether there are indications that the results in the published literature differ systematically from the results that remain unpublished (e.g., by using a funnel plot-, calculating a fail-safe N, or performing Duval and Tweedie's trim and fill analysis). Examples of sensitivity analyses and publication bias analyses in multilevel meta-analyses of SCEs can be found in the applied studies by Heyvaert, Maes, and Onghena (Reference Heyvaert, Maes and Onghena2010), Heyvaert, Maes, Van den Noortgate, Kuppens, and Onghena (Reference Heyvaert, Maes, Van den Noortgate, Kuppens and Onghena2012), and Vanderkerken, Heyvaert, Maes, and Onghena (Reference Vanderkerken, Heyvaert, Maes and Onghena2013).

9. 9. Finally, besides deriving the patient-level treatments effects using ‘empirical Bayes estimates’, it is also possible to perform a fully Bayesian analysis of multilevel data if prior distributions for the parameters can be specified. The fully Bayesian approach is promising for small data sets, offers an elegant solution for imputing missing values, and also lends itself very nicely to a sequential or cumulative meta-analysis (de Vries & Morey, Reference de Vries and Morey2013; Kravitz & Duan, Reference Kravitz, Duan, Duan, Eslick, Gabler, Kaplan, Kravitz, Larson, Pace, Schmid, Sim and Vohra2014; Moeyaert, Rindskopf, Onghena, & Van den Noortgate, Reference Moeyaert, Rindskopf, Onghena and Van den Noortgate2017).

Comment on the Literature Regarding the Analysis and Meta-Analysis of SCEs

When reviewing the literature regarding the analysis and meta-analysis of SCEs, one sometimes gets the impression that the wheel is reinvented. Sometimes it seems that a completely new statistical language is needed with specific overlap measures and inferential techniques. In other publications, statistical techniques seem to be borrowed, and at the same time divorced, from the mainstream statistical literature. These techniques might get nourished in particular niche journals with minor reference to the broader statistical literature, but this is not beneficial for the methodological development of the field.

By contrast, in this article we showed that standard multilevel analysis, common meta-analytical tools, and old school randomisation tests can be used for the analysis and meta-analysis of SCEs. The underlying tenet of our demonstration is an advocacy to connect to the general statistical literature and to consider an SCE data set as any other data set. The only special consideration refers to the specific features that all SCE data have in common: repeated observations or measurements and a manipulated variable with at least two levels. These features imply that particular attention has to be paid to time trends and problems of serial dependence, and that randomisation schedules should be compatible with the phasic or alternation structure of the design.

Connecting the analysis and meta-analysis of SCEs to the general statistical literature has the advantage that more than a century of statistical experience and research is made to bear on these data, and that the results of analyses and meta-analyses of SCEs might become more acceptable/ digestible/convincing to the general scientific community. Furthermore, this connection avoids having to repeat all discussions and controversies regarding model-based versus design-based inference, population versus causal inference, frequentist versus Bayesian inference, parametric and non-parametric inference, the complementarity of visual and statistical techniques, the justified use of exploratory and confirmatory techniques, the combination of quantitative and qualitative data, . . . within an isolated SCE community.

Comments on Statistical Modelling

We already indicated that statistical modelling involves making concessions. These concessions are a consequence of any scientific model being an attempt at providing an insightful reduction of a complex reality, guided by the epistemological principle of parsimony. Furthermore, if this principle is left aside, then statistical modelling might become problematic for trustworthy inferences. For example, the multilevel model is so flexible and has so many options that very complex models can be built, to the extent that the model is just a description of the data. This is exactly what happens if the number of parameters to be estimated equals the number of independent data points.

A corollary of this flexibility is that many decisions have to be made in the modelling enterprise and that some decisions might be guided by the researcher's biases pro or contra certain hypotheses. To counteract these biases, and to control for type I and type II errors in a formal statistical framework, it is necessary to specify the model and its test statistics before the data are inspected. Consequently, it might also be recommended for the SCE community to install formal preregistration of SCEs and their meta-analyses (see e.g., the PROSPERO registry; Booth et al., Reference Booth, Clarke, Dooley, Ghersi, Moher, Petticrew and Stewart2012) to steer clear of too many ‘researcher degrees of freedom’ and of too much opportunistic, misguided and self-deceiving post factum model building (Simmons, Nelson, & Simonsohn, Reference Simmons, Nelson and Simonsohn2011; Wicherts et al., Reference Wicherts, Veldkamp, Augusteijn, Bakker, van Aert and van Assen2016).

Comments on Meta-Analysis

According to Eysenck (Reference Eysenck1978) a meta-analysis is ‘an exercise in mega-silliness’ and he referred to a meta-analysis that was, in his opinion, a compilation of heterogeneous (‘mixing apples and oranges’) and poorly designed studies (‘garbage in, garbage out’). Furthermore, meta-analysis is also sometimes contested as a valid scientific method because the results in the sampled or published studies are most likely unrepresentative of the phenomenon studied (the so-called ‘file drawer problem’), meta-analysis conclusions can disagree with large-scale randomised controlled trials, meta-analyses are not reproducible, meta-analyses are performed poorly, and meta-analyses use inferior statistical tools (Borenstein, Hedges, Higgins, & Rothstein, Reference Borenstein, Hedges, Higgins and Rothstein2009; Lakens, Hilgard, & Staaks, Reference Lakens, Hilgard and Staaks2016; Scargle, Reference Scargle2000; Terrin, Schmid, Lau, & Olkin, Reference Terrin, Schmid, Lau and Olkin2003).

This controversy and skepticism regarding meta-analysis carries over to the meta-analysis of SCEs. While some of the criticisms can easily be debunked (see Borenstein et al., Reference Borenstein, Hedges, Higgins and Rothstein2009, and Cooper et al., Reference Cooper, Hedges and Valentine2009, for excellent responses), single-case researchers should be encouraged to raise the bar with respect to the quality of their meta-analyses. The criticism of ‘garbage in, garbage out’ can be turned into a constructive approach to meta-analysis as ‘waste management’ by using methodological and reporting quality of the SCEs as inclusion criteria or moderator variables. Or more fundamentally, single-case researchers can engage in ‘sustainable research’ by producing less waste and improve the overall quality of the design and analysis of the primary studies. The instrument developed by Tate et al. (Reference Tate, Aird and Taylor2013) for the assessment of the methodological quality of SCEs and the reporting guidelines developed by Tate et al. (Reference Tate, Perdices, Rosenkoetter, McDonald, Togher, Shadish and Vohra2016a, Reference Tate, Perdices, Rosenkoetter, Shadish, Vohra, Barlow and Wilson2016b) are very useful in this respect. By paying more attention to the methodological and reporting quality of the primary studies, the field of SCEs as a whole will benefit (Kratochwill et al., Reference Kratochwill, Hitchcock, Horner, Levin, Odom, Rindskopf and Shadish2010, Reference Kratochwill, Hitchcock, Horner, Levin, Odom, Rindskopf and Shadish2013; Lobo, Moeyaert, Cunha, & Babik, Reference Lobo, Moeyaert, Cunha and Babik2017). In addition, the methodological and reporting quality of the SCE meta-analyses, as such, can be improved by adhering to the general standards and guidelines for meta-analyses and systematic reviews (Moher, Liberati, Tetzlaff, Altman, & the PRISMA group, Reference Moher, Liberati, Tetzlaff and Altman2009; Shea et al., Reference Shea, Hamel, Wells, Bouter, Kristjansson, Grimshaw and Boers2009).

Besides the general controversy and skepticism regarding meta-analysis, two concerns need special care when considering an SCE meta-analysis. The first concern is the relatively high risk for selectivity of cases submitted and published in the literature. If cases are published with a strong preference for spectacular, extreme, or successful cases, then any meta-analysis of these cases will be severely biased in the direction of success. A survey by Shadish, Zelinsky, Vevea, and Kratochwill (Reference Shadish, Zelinsky, Vevea and Kratochwill2016) found indications that the publication bias is larger for SCEs than for other research designs indeed. This raises important concerns with respect to the validity of SCE meta-analyses because no sophisticated statistical analysis can remedy the selectivity of cases.

Another concern is the kind of inference that is aimed at with an SCE meta-analysis. As mentioned in our reflection on the definition of an SCE, some SCEs are performed just for the benefit of a single patient (comment 5). Such SCEs are purely idiographic and do not have the intention to be generalisable. They aim at causal inference for one particular patient. When performing an SCE meta-analysis this should be taken into account. It should be clear to what extent the meta-analysis considers the patients as a random sample from a target population and to what extent the meta-analysis is an aggregation for causal inference, with generalisation based on non-statistical arguments.

Future Research

The analysis and meta-analysis of SCEs represents a thriving part of current methodological and statistical developments and applications. In the context of this article, we mention three topics that need more research than currently available.

The first topic is the number of patients and the number of studies needed to perform a reliable and efficient multilevel meta-analysis. Cools, Van den Noortgate, and Onghena (Reference Cools, Van den Noortgate and Onghena2008, Reference Cools, Van den Noortgate and Onghena2009) developed general software to determine the efficiency of multilevel designs, but a validation in the context of SCE meta-analysis is still lacking. It is to be expected that there are numerous factors involved in determining the appropriate number of patients and studies (e.g., the number of repeated measurements and the size of the effects that are considered clinically relevant) but more guiding principles would be useful. Van den Noortgate and Onghena (Reference Van den Noortgate and Onghena2007) suggested that multilevel meta-analysis of SCEs should include at least about 20 entities at the highest level (patients or studies). Also Kuppens et al. (Reference Kuppens, Heyvaert, Van den Noortgate and Onghena2011) recommended that the initial interim analysis of a sequential single-case meta-analysis should comprise at least about 20 single-case publications including single or multiple cases. By contrast, Heyvaert et al. (Reference Heyvaert, Hannes and Onghena2017) performed an extensive simulation study and found acceptable statistical power values for replicated AB designs with seven patients, 20 repeated measurements each, for both multilevel meta-analysis and probability combining of randomisation test p-values.

A second topic is the combination of SCEs and group-comparison studies in one overall meta-analysis. Although, it would be important for evidence-based practice that all kinds of scientific evidence were given weight in comprehensive systematic reviews, it is not evident how one should proceed. One option is to aim at a quantitative integration of both kinds of evidence. For example, Pustejovsky, Hedges, and Shadish (Reference Pustejovsky, Hedges and Shadish2014) proposed a general framework for defining effect sizes in replicated SCEs that are directly comparable to the standardised mean difference in between-subjects randomised experiments (see Punja et al., Reference Punja, Schmid, Hartling, Urichuk, Nikles and Vohra2016, for an application). Shadish, Rindskopf, and Boyajian (Reference Shadish, Rindskopf and Boyajian2016) found similar results between an SCE meta-analysis and the results of a randomised controlled trial but also mentioned that more work is needed to understand the conditions under which this holds. A second option is to use estimates of between- and within subject variances to convert SCE effect sizes in order to make them comparable to group-comparison study effect sizes, and combining them in a single meta-analysis, as described by Van den Noortgate and Onghena (Reference Van den Noortgate and Onghena2008). A third option is to separately perform the SCE meta-analysis and the group-comparison meta-analysis and to combine the results in a narrative way (looking for convergence, elaboration, or confrontation). The latter option has the advantage that also qualitative case studies can be involved (Heyvaert, Hannes, & Onghena, Reference Heyvaert, Hannes and Onghena2017; Heyvaert, Maes, & Onghena, Reference Heyvaert, Maes and Onghena2013). Future research should examine the potential and pitfalls of these options, both in theory and application.

The third topic is the development of guidelines and standards for the statistical analysis and meta-analysis of SCEs. Many authors have referred to the lack of consensus regarding the statistical analysis and meta-analysis of SCEs (see e.g., Cohen, Feinstein, Masuda, & Vowles, Reference Cohen, Feinstein, Masuda and Vowles2014; Smith, Reference Smith2012), which may even reinforce some researchers in just performing a visual analysis and calculating a simple Percentage of Non-overlapping Data (PND), without any further statistical analysis at all (Horner et al., Reference Horner, Carr, Halle, McGee, Odom and Wolery2005; Kratochwill et al., 2010, 2013; Schlosser, Lee, & Wendt, Reference Schlosser, Lee and Wendt2008; Scruggs & Mastropieri, Reference Scruggs and Mastropieri2013). Hence some guidelines and standards are more than welcome (see Manolov & Moeyaert, Reference Manolov and Moeyaert2017b, for some provisional recommendations).

In developing these guidelines and standards we should not chase the chimera that there is only one optimal way of performing the statistical analysis and meta-analysis of SCEs. Optimality always refers to some criterion, and this criterion in its turn can be the subject of debate. We can emphasise robustness, power, small sample properties, asymptotic properties, unbiasedness, accuracy or some other criterion and depending on the criterion have another preference. As wittingly remarked by John Tukey: ‘the collective noun for a group of statisticians is a quarrel’ (Vallverdú, Reference Vallverdú2016), illustrating that this ‘lack of consensus’ is evident for any non-ritualistic statistical analysis (Tukey, Reference Tukey1969). As Stephen Senn (Reference Senn2017) put it: ‘Consult two medics and you'll get two opinions but consult two statisticians and you could easily get three’ (Senn, Reference Senn2017).

Given the current state of the art, there is only one principle that we want to propose for the development of these guidelines and standards. This principle is the connection of any technique for the analysis and meta-analysis of SCEs to the general statistical literature, as we mentioned before. SCE data sets can be analysed as any other data set, with particular attention to time trends, possible serial dependence, and randomisation schedules that are consistent with specific single-case designs. This guiding principle implies a critical attitude towards an overreliance on visual techniques and PND to analyse and meta-analyse SCEs. For example, no one would seriously consider using PND to compare two groups of a randomised controlled trial. For some strange reason, though, we tend to accept this measure for the analysis and meta-analysis of SCEs, although it does not take into account any of the specific features of SCEs.