Introduction
The drive for new treatments in Alzheimer's disease (AD) is stimulating research across a wide number of areas, including clinical trial methodology. Providing evidence for a fundamental alteration in clinical course raises issues including separating temporary symptom improvement from long-term disease alteration; identifying meaningful outcome measures for patients, carers and health professionals; validating appropriate surrogate biomarkers; identifying potential subgroups of responders from larger populations; and establishing those most likely to progress to AD at the earliest possible point for intervention. Slow progression rates may necessitate long-term studies with an associated tendency to towards high drop-out rates and cost. The relative absence of drugs to slow the disease course may be partly due to deficiencies in trial methodology. For example, recent work by Hobart et al. (Reference Hobart, Posner, Aisen, Selnes, Stern, Thomas, Weiner and Zajicek2009) using the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset suggests that the Alzheimer's Disease Assessment Scale – Cognitive (ADAS-Cog) performs less than adequately in detecting people at the mild end of the AD spectrum. In addition, the reliance on univariate methodology limits the scope for interpretation of complex multivariate data.
This review describes the methodology used in recently published papers of clinical trials in AD and aims to highlight the level of methodological sophistication within current research. We have collated information on the primary statistical analysis methods used as well as other facets of clinical trial design, including trial duration, intervention type and primary outcome measure. These additional design aspects provide an overview of clinical trial methodology.
The review presented here differs from the “traditional” systematic review, which focuses on assessing evidence of effects of particular treatments across different studies. Our focus is on gaining an overview of the statistical methodological aspects of clinical trials used in AD. Similar reviews covering outcome measures and measures of clinical significance have been published (for example, Wolfson et al., Reference Wolfson, Moride, Perrault, Momoli, Demers and Oremus2000; Woodford and George, Reference Woodford and George2007; Craig-Schapiro et al., Reference Craig-Schapiro, Fagan and Holtzman2009; Molnar et al., Reference Molnar, Man-Son-Hing and Fergusson2009). However, these did not address the statistical methodology employed.
Methods
We followed the Cochrane handbook for systematic reviews of interventions (Higgins et al., 2007). The process involved the steps described below. The review criteria were chosen to achieve a broad overview of statistical research methodology and were not directed at a specific intervention.
Search strategy
We limited the review to AD clinical trials publications over the five-year period from January 2004 to August 2009. We chose this time period so that we focused on the most up-to-date methodology and also so that the volume of papers to review was significantly reduced. The search focused on two major research libraries: the Cochrane Library and PubMed. The search strategy for the Cochrane Library used the search term “Alzheimer's disease” within the “title, Abstract and keywords” and was restricted to the “Cochrane Central Registrar of Controlled Clinical Trials’” (Clinical trials) between the data range 2004 and 2009.
The search was then repeated for the PubMed library. We used the search terms “Alzheimer's disease clinical trials” and limited the search by the following:
• Dates: 2004/01 to 2009/08
• Type of article: Clinical trial
• Species: Human
• Subsets: More subsets: Medline, PubMed central
• Text options: Links to full text
• Languages: English
Relevant titles and abstracts were identified for further consideration. We also scanned the reference lists of relevant articles to identify additional publications. The reference lists of relevant articles were updated into a citation manager database and any duplication from the searches removed.
Selection criteria and data extraction
The following inclusion and exclusion criteria were applied to provide a comprehensive overview of the statistical methodology in clinical trials for AD.
Inclusion criteria:
• Clinical trial – the paper must report a randomized clinical trial of an intervention for AD. This does not cover other types of study designs, such as cohort or case control studies.
• Peer reviewed – the paper must be from an established peer reviewed journal. This does not include conference abstracts or poster presentations.
• Outcome measures – the paper must specify the primary outcome measure used and include details of the version or variant if applicable.
• Trial duration – the paper must specify the trial duration of the intervention period.
• Intervention type – in the vast majority of cases the intervention is specified or is clearly apparent, but in any case must be specified.
• Primary statistical analysis – the paper must detail the primary statistical analysis used to analyze the outcome data. If a general technique has been applied which encompasses a number of methods, the specific details should be included for clarity of the methods.
Exclusion criteria:
• Failure to meet any of the inclusion criteria.
• Full electronic version of paper unavailable.
• Papers discussing new methodology rather than discussing a specific clinical trial.
• Papers discussing qualitative research, as this review is interested in quantitative research methodology.
• Other systematic review papers for clinical trials in AD.
• Other types of dementia.
We produced a written protocol to screen each paper identified in the search. Papers meeting the inclusion and exclusion criteria set out above were scrutinized and the relevant details entered into a spreadsheet.
Quality of the evidence
Given our aim of providing a taxonomy of methodology as opposed to providing “best evidence” of a particular effect, all papers that satisfied our inclusion/exclusion were included in our review.
Classification criteria
We used a spreadsheet that contained the following fields: primary outcomes, duration of trial, intervention and primary statistical analysis methodology. Within each field, information was classified as described below in order to produce a taxonomy.
Classification of outcome measures
Outcome measures were classified into nine groups: cognitive, mood, composite outcome, physical, behavior, caregiver burden, biochemical markers, imaging measurements and time-to-event measurements.
Cognitive function
These assessments focus on measurements of specific areas such as, memory, comprehension, orientation or decision-making using cognitive tests. Typically the tests involve a series of items comprising questions or tasks varying from following simple instructions to drawing shapes. Each item results in either an incorrect/correct response or an ordinal rating. These tests are widely used in both clinical trials and in general clinical assessment of AD patients. Many variations and extensions to these tests exist and were encountered in this review.
Mood
These assessments try to judge the mood or level of depression of the patient. The test procedure usually involves a series of questions to which a patient answers incorrect/correct, or alternatively rates themselves on an ordinal rating. In most cases, these tests were not specifically developed for neurodegenerative diseases but as a general mood or depression measure. Many variations and extensions to these tests exist and were encountered in this review.
Behavior
Behavioral assessments measure the frequency and severity of different behavioral disturbances shown by the patient. Typical disturbances include: delusions, hallucinations, dysphoria, anxiety, agitation/aggression, euphoria, disinhibition, irritability/lability, apathy, and aberrant motor activity.
Caregiver burden
Carer-based assessments are designed to assess the patient's carer. For example, they can be used to measure the degree of dependence of the patient on their carer or how the patient's condition affects their life.
Imaging measurements
These measurements are taken using brain scanning technology, and can be subcategorized into two areas: structural imaging and functional imaging output. Structural imaging measures changes in the size and shape of areas of the brain. Functional imaging measures the functionality or mechanics of the chemical and physical processes in the brain. Functional imaging can be measured by monitoring images over a controlled time period.
Both techniques provide information on the condition of the brain and its function. These techniques provide surrogate outcome measures often used in conjunction with clinical outcome measures. There have been a number of publications presenting the correlation between imaging and cognitive functional assessments (see Hanyu et al., Reference Hanyu, Sakurai, Iwamoto, Takaski, Shindo and Abe1998; Fox et al., Reference Fox, Scahill, Crum and Rosser1999; Apostolova et al., Reference Apostolova2006) as well as examining the association between neurodegeneration and either loss of brain function or brain atrophy. These techniques show potential as an assessment tool as the development of improved technology is advancing (see Nestor et al., Reference Nestor, Scheltens and Hodges2004).
Biochemical markers
These are measurements taken from blood, urine or CSF (cerebrospinal fluid) samples. These biochemical markers may correlate with deterioration in cognitive function by monitoring specific proteins and how they change over time. Hampel et al. (Reference Hampel, Bürgerb, Teipel, Bokdea, Zetterberg and Blennow2008) and Craig-Schapiro et al. (Reference Craig-Schapiro, Fagan and Holtzman2009) discuss some of the most widely used biochemical markers in use with AD diagnosis which have featured within our review. These measures have also been considered as potential surrogate outcome measures.
Physical measures
These range from duration of sleep to changes in patients’ disability/physical measures. These measures can often be secondary outcome measures. In some studies their baseline values are included as covariates in the primary analysis. We have created the category to encompass instruments, measures and scales that measure an aspect of physical performance. These can include: instrumental activities of daily living, physical self maintenance scales, the Epworth sleepiness scale or even incidence of injuries, typically fractures.
Time-to-event measures
Time-to-event outcome measures are the time to occurrence of a disease-related event. They are typically measures such as death, institutionalization or drop-out from program.
Composite outcome
A number of available instruments are designed to cover multiple outcome areas. These instruments have evolved using specific elements of other outcome measure instruments to form a composite outcome assessment. These instruments are typically an interview based assessment or a questionnaire.
Classification of intervention
Interventions were subcategorized as follows:
Pharmaceutical Intervention
This describes trials of new or existing drugs. Two papers reported mixtures of pharmaceutical and other interventions which we have included in the pharmaceutical category as indicated in a footnote in the tables.
Other intervention
This includes any type of therapy or treatment that does not directly involve a drug. These interventions are typically physical and psychological therapies, care routines or carer-based training.
Classification of primary statistical analysis
We began with a list of categories covering commonly used statistical methods, which was extended to incorporate methods identified in the search that were not represented in the original categories. The classification was restricted to the primary statistical analysis. We classified the techniques as “univariate”, “multivariate” or “other” to highlight the predominance of univariate techniques in the primary analyses. The statistical definitions used to subcategorize this area are described as follows (Upton and Cook, Reference Upton and Cook2008):
Univariate
Student's T-test /Analysis of variance (ANOVA): A hypothesis test assessing the evidence of differences between two (t-test) or more (ANOVA) means. It is assumed that, after appropriate transformation, the distribution of the response about its mean has constant variance and is normally distributed.
Analysis of covariance (ANCOVA): A hypothesis test assessing the evidence of differences between two or more means allowing for the effect of one or more covariates.
Non-parametric normality test: Hypotheses tests that, unlike the equivalent parametric tests (e.g. t-tests), do not involve distributional assumptions.
Linear regression: Linear regression models the relationship between one or more dependent variables and one or more independent variables, provided the model depends linearly on the unknown parameters to be estimated from the data.
General linear model (including random effects): The term general linear model embraces t-tests, ANOVA, ANCOVA and linear regression within a single framework. In this review, general linear models including random effects have been separated from these other classes. They are used for the assessment of treatment effects allowing for grouping factors.
Generalized linear model (GLM): A generalized linear model is a generalization of the general linear model to deal with responses that have alternative distributional forms. For example, counts, categories or ordered categories.
Survival model: Analysis of time to event data.
Multivariate
Multivariate analysis of variance (MANOVA): MANOVA is a generalized form of the ANOVA. Multiple dependent variables can be used.
Multivariate analysis of covariance (MANCOVA): MANCOVA is an extension of the ANCOVA method when there is more than one dependent variable, and these variables cannot be combined easily.
Results
Figure 1 shows the disposition of papers and their source locations. A substantial proportion (860) of papers resulting from the search failed the inclusion criteria for our review. The main reason for exclusion was that the paper was not a report of a randomized clinical trial. Another common reason was that the paper described trials of other neurodegenerative diseases such as Parkinson's or Huntington's diseases although “Alzheimer's disease” had been mentioned in the abstract.
Summary of search results showing numbers of included and excluded papers.
The papers included in the review reflected the diversity of work in the area of AD, covering publications in clinical, statistical, social science, nutrition and physiotherapy journals. The majority of papers covered Phase II or Phase III trials that were predominantly designed to demonstrate superiority.
Table 1 presents a summary of the papers included in the review according to the statistical methodology applied in the primary analysis. Generally, there is little to suggest any difference between the interventions with regards to the statistical methods used. The statistical methodology applied was predominantly univariate parametric tests (t-test, ANOVA, and ANCOVA). The simpler univariate techniques, specifically the ANOVA and ANCOVA were used most frequently in the primary statistical analysis.
Summary of statistical methodologies by intervention
* Two included pharmaceutical + other combined.
Table 2 presents the summary of outcome measures by intervention. In both intervention groups the most commonly used outcome measure was cognitive testing.
Summary of primary outcome measures by intervention
* Two included pharmaceutical + other combined.
Table 3 presents the summary of primary statistical methodology by duration of follow-up. Both simple univariate and more advanced multivariate statistical methods were generally limited to shorter trials of less than 12 months. A limited number of trials used multivariate methods; however, in some cases multivariate methods were used in secondary analyses, or if specific data were extracted from completed trials, for post hoc analysis.
Summary of statistical methodology by trial duration
Table 4 displays the summary of outcome measures by trial duration. There were no substantive differences in outcome measure with regard to the associated duration of trial. Generally, the longer trials were associated with cognitive and physical outcomes.
Summary of trial duration by primary outcome measures
Table 5 shows that there was little to suggest any difference between outcome and methodology. There was a clear preference for the simpler statistical univariate methods. The more advance modeling was generally used in trials which used cognitive outcome measures.
Summary of statistical methodology by primary outcome measure
Discussion
Focusing on recently published AD clinical trials (between 2004 and 2009), this paper presents a review of methodological aspects of clinical trials in AD. The review shows how research has been dominated by univariate as opposed to multivariate methods for testing the differences between intervention and placebo groups with respect to a single primary outcome. This is hardly surprising given the current best practice guidelines that focus on the use of a pre-specified analysis of a single primary outcome measure (Moher et al., Reference Moher2010). Whilst it is difficult to argue against guidelines aimed at providing good quality evidence that avoids the pitfalls of multiple comparisons and data dredging, it is clear that most trials of AD involve the collection of complex longitudinal multivariate data. This reflects the complexity of the disease. In this context it is far from clear cut what, if any, single primary outcome should be used. It could be argued that there is a role for a more sophisticated approach to the analysis that avoids issues of multiple testing and post hoc comparisons but makes better use of the data (for details see Rencher, Reference Rencher2002).
Cognitive test instruments were the most common form of assessment in the trials, accounting for 38% of the primary outcome measures. Within this subgroup, the most popular instruments were the ADAS-Cog and Mini-Mental State Examination (MMSE), accounting for 48% and 30% of the subgroup respectively. As we previously stated, recent findings using the ADNI dataset suggests that the ADAS-Cog performs less than adequately in people at the mild end of the AD spectrum, hence there appears to be some over reliance on poorly developed cognitive measures in trials (Hobart et al., Reference Hobart, Posner, Aisen, Selnes, Stern, Thomas, Weiner and Zajicek2009).
Mendiondo et al. (2002) showed that the MMSE is also less sensitive to change in both the mild and severe spectrum of AD. The majority of trials (87%) published over the period January 2004 to August 2009 involved follow-up periods of one year or less. When compared to the typical duration of disease progression (greater than 12 months) the potential improvement shown in interventions may not be totally representative.
Cognitive outcomes are generally considered to be most relevant; therefore, if deciding what measures to include in longer-term follow up, these are a popular choice. When compared to some surrogate outcome measures, cognitive measures are relatively simple to collect, again resulting in cognitive measures predominantly showing up in longer-term follow-ups. Consequently, it is common to find these outcome measures used in longer trials that use more advanced modeling techniques to deal with the longitudinal nature of the data.
In shorter trials, simpler analytic techniques were often used as the trial design was constrained to a single primary outcome variable and often measurement was taken at baseline and one follow-up, limiting the available analyses to a range of simple univariate tests.
The data sets from recent clinical trials are a rich resource for increasing our understanding of AD through analysis that reflects the multidimensional longitudinal data and captures the features of the disease. It should then be possible to develop new approaches to design and analysis that are appropriate for AD but avoid the pitfalls of multiplicity and data dredging.
The application of a pre-planned primary analysis that involves multivariate methods has the benefit of formally reflecting the multidimensional nature of AD and the large number of different outcome measures available. The specification of this kind of analysis requires understanding of the different outcome measures and the inter-relationships between them from the statistical and clinical perspectives. A number of researchers (Pocock et al., Reference Pocock, Geller and Tsiatis1987; Tang et al., Reference Tang, Geller and Pocock1993; Zhang et al., 1996) have discussed the implementation of clinical trials with multiple endpoints and the benefits of these approaches.
We are currently looking at data from cohort studies of AD with a view to modeling the multivariate structure of the disease using seven instruments of different outcome measures covering mobility, dependency, cognition, depression, motor function and activities of daily living. This can then be used to produce a structural model for the disease (Skrondal et al., Reference Skrondal and Rabe-Hesketh2004). Hypotheses concerning treatment effects can then be represented within the framework of this mode.
In a recent paper by Dubois et al. (Reference Dubois2010), a revised definition of AD has been reported. The US Federal Drug Agency's approved, revised definitions aim to provide a common lexicon for clinical research in AD. The definitions would have most impact when considering clinical trial design as trials would need to become more targeted to subpopulations of AD. The available outcome measures may become more segregated toward specific aspects of AD, which may help to reduce the arbitrary nature of the primary outcome measure. The implications for the statistical analyses, as a result of more targeted trials, could yield improvements when trying to assess the efficacy of new interventions.
Conflict of interest
None.
Description of author's roles
Paul Thompson designed the search protocol and strategies, conducted the search and collection of papers, and reviewed and wrote up the results. David Wright worked on the production of the paper and reviewed the final draft. Carl Counsell provided critical appraisal of search strategies and designed the review classification criteria. John Zajicek also provided critical appraisal of search strategies and wrote both the introduction and discussion sections.
Acknowledgments
This review presents independent research commissioned by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research scheme (RP-PG-0707-10124). The views expressed in this publication are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. We are grateful to Professor Gavin Giovannoni, Dr. Colin Green and Dr. James Shearer for their input to earlier versions of this review.
