Surrogate endpoints are intended to substitute for final patient-relevant endpoints that directly measure how patients feel, function or survive in clinical trials (Reference De Gruttola, Clax and DeMets1). Evidence from surrogate endpoints may not only expedite the regulatory approval of new health technologies but also inform coverage and reimbursement decisions. In the United Kingdom, several recommendations of the National Institute for Health and Care Excellence (NICE) have been based on cost-effectiveness analyses entirely based on treatment effects derived from clinical trials assessing surrogate endpoints (Reference Elston and Taylor2). Moreover, this type of evidence may still be relied upon even when patient-relevant endpoints are available, for example in clinical trials that have terminated prematurely or for which data on the final endpoint are not fully mature. Nevertheless, relying on evidence from surrogate endpoints poses a serious challenge for decision makers, as several failures of candidate surrogate endpoints have been reported over the last decades (Reference Yudkin, Lipska and Montori3–Reference Fleming and DeMets5); such failures have arisen not only from discrepancies in the magnitude of treatment effects between surrogate and final endpoints (Reference Ciani, Buyse and Garside6), but also in their directions (Reference Fleming and DeMets5). Hence, in order for policy makers to use a surrogate endpoint with confidence, there must be a process of “surrogate validation.”
The statistical validation of surrogate endpoints has been a major focus of research activity over the last 2 decades (Reference Lassere7;Reference Weir and Walley8), but no consensus exists with respect to the standards needed to identify valid surrogates. Nevertheless, two key tenets dominate current views on the issue, namely the “correlation” and the “meta-analytic” approaches (Reference Buyse, Molenberghs, Burzykowski, Renard and Geys9;Reference Buyse, Sargent, Grothey, Matheson and de Gramont10). According to these two tenets, the core goal of surrogate validation is to demonstrate a correlation between the surrogate and the final endpoint in the context of a clinical trial as well as between treatment effects on the surrogate and on the final endpoint within the context of a meta-analysis of randomized controlled trials (RCTs) (Reference Buyse, Sargent, Grothey, Matheson and de Gramont10). The uptake of surrogate validation methods in technology assessment and coverage decisions is limited (Reference Velasco Garrido and Mangiapane11), a potential explanation being the lack of harmonization of statistical techniques that should be used. Moreover, while decision tools have been proposed to assist policy makers in judging the strength of such validation evidence for a candidate surrogate, there has been little or no empirical testing of these decision tools to date (Reference Elston and Taylor2;Reference Lassere7;12).
Cancer trials are one of the areas in which surrogate endpoints have become most common (Reference Ellenberg and Hamilton13–Reference Shi and Sargent16). Progression-free survival (PFS), measured as the time from randomization to either documented tumor progression or death, is often used as primary endpoint in RCTs as a surrogate for overall survival (OS) (Reference Saad, Katz and Buyse17). Tumor progression includes radiographic evidence (Reference Jaffe18–Reference Eisenhauer, Therasse and Bogaerts20) and, in some instances, non-radiographic criteria such as “symptomatic progression” or “clinical deterioration” determined by a clear, unequivocal worsening of the symptoms and signs of disease that are not evident on radiographic assessment (Reference Dancey, Dodd and Ford21). Some trials use time-to-progression (TTP) rather than PFS, the difference being that in TTP, patients are censored at the time of death with no prior documentation of disease progression. Other surrogate endpoints have been adopted in oncology, the most common being tumor response rate. However, TTP and PFS are more often used in phase III clinical trials (Reference Saad, Katz and Buyse17) and cost-effectiveness analyses of treatments for metastatic solid tumors (Reference Tappenden, Chilcott, Ward, Eggington, Hind and Hummel22), whilst tumor response rate better served for this purpose in hematologic malignancies (Reference Shi and Sargent16;Reference Sridhara, Johnson and Justice23).
In this study, we review current statistical methods of surrogate-endpoint validation that use a meta-analytic framework. In addition, we assess the strength of evidence for PFS and TTP as surrogates for OS and test the application of current surrogate validation decision tools to the evidence base in several advanced solid tumors.
METHODS
Study Identification and Selection
Meta-analyses of RCTs quantifying the statistical association between PFS or TTP and OS in advanced solid tumors were sought. Conventional literature searches of electronic bibliographic databases returned a large number (>3,000) of references, and attempts to make the search more specific resulted in the exclusion of many of the papers already known to the authors. Therefore, we used a “citation pearl-growing” approach to study identification (Reference Hartley, Keen, Large and Tedd24), with backward and forward citation searching from an initial list of six papers known to the authors (Reference Bowater, Bridge and Lilford25–Reference Sherrill, Amonkar and Wu30). The citation searches were conducted using Medline and the Science Citation Index in March 2012, and forward citation searching up until December 2012. Two reviewers independently screened titles and abstracts. We excluded conference abstracts, letters to the editor, papers reporting results from single trials, meta-analyses reporting treatment effects on PFS or TTP and OS without assessing an association between them, and descriptive reviews (Reference Saad, Katz and Buyse17;Reference Buyse31;Reference Sherrill, Kaye, Sandin, Cappelleri and Chen32). Meta-analyses that focused on oncology treatments with curative intent were also excluded, as PFS and TTP are not relevant endpoints in this case (Reference Lee, Wang and Crump33;Reference Tibaldi, Barbosa and Molenberghs34).
Data Extraction and Analysis
Three levels of data were extracted from included meta-analyses using standardized pro-formas: information on the general characteristics of each meta-analysis (authors and date of publication, criteria for inclusion of studies, number and nature of studies included, number of patients included, and type of tumor and interventions considered); details of the statistical methods reported to assess the association between surrogate and final endpoints, the results of these analyses, and each study authors’ conclusions based on the results; and details of the literature search performed to identify included studies. Data were extracted by a single reviewer and checked by a second. Finally, we sought to analyze the suitability of PFS and TTP as surrogates for OS using established surrogate validation frameworks. Three surrogate validation frameworks identified by a recent review of surrogate-endpoint methods were applied to each meta-analysis (35); they are outlined briefly below. To ensure consistency, they were applied to each meta-analysis by a single reviewer and checked by a second reviewer, and discrepancies resolved with involvement of a third reviewer.
Elston and Taylor's Framework
In 1999, Bucher and colleagues (Reference Bucher, Guyatt, Cook, Holbrook and McAlister36) proposed a set of validity criteria to inform the use of an article measuring the effect of an intervention on surrogate endpoints in clinical practice. These criteria were adapted by Elston and Taylor (Reference Elston and Taylor2) into a three-level evidence hierarchy: Level 1, evidence demonstrating that treatment effects on the surrogate (i.e., the change on the surrogate endpoint of treatment versus control arm) correspond to treatment effects on the final patient-relevant endpoint (from RCTs); Level 2, evidence demonstrating a consistent association between surrogate outcome and final patient-relevant outcome (from at least epidemiological/observational studies); Level 3, evidence of biological plausibility of a relationship between surrogate and final patient-relevant outcomes (from pathophysiologic studies and/or understanding of the disease process).
German Institute of Quality and Efficiency in Health Care (IQWiG) framework
In 2011, the Institute of Quality and Efficiency in Health Care (IQWiG), an independent health technology assessment (HTA) agency that assesses the benefits and harms of drug and non-drug technologies on behalf of the German Federal Joint Committee and the Federal Ministry of Health, published a framework for the validation of surrogate endpoints in oncology (12). The IQWiG framework proposes that two levels of consideration are required to judge the suitability of a surrogate endpoint in the assessment of cancer therapy: the reliability of the evidence and the strength of evidence for surrogate validation. Reliability is measured as high, limited, moderate, or low on the basis of the following aspects: (i) application of a recognized approach described in the specialized statistical literature, (ii) conduct of analyses to test the robustness and generalizability of results, (iii) systematic compilation of data, (iv) sufficient restriction of indications or degrees of disease severity and of interventions, and (v) clear definitions of the endpoints investigated. The strength-of-evidence criterion considers the degree of correlation of effects on the surrogate and the patient-relevant endpoint according to predefined thresholds (i.e., high correlation, when the lower limit of the 95 percent confidence interval for R ≥ 0·85; low correlation, when the upper limit of the 95 percent confidence interval for R ≤ 0·7; and medium correlation otherwise). Depending on the categorization produced by an algorithm that takes into account both levels of consideration, a conclusion about the validity of the surrogate endpoint is drawn and expressed as proof, indication, hint or no proof of an effect on the patient-relevant endpoints as derived from an observed effect on the surrogate endpoint. While the IQWiG framework provides a list of elements that contribute to “reliability”, we needed to introduce a system of scoring that enabled us to categorize this dimension in a reproducible manner (e.g., a “high” score required all contributing elements to be met).
Biomarker-Surrogacy Evaluation Schema (BSES3)
The Biomarker-Surrogacy Evaluation Schema (BSES3) (Reference Lassere, Johnson, Schiff and Rees37) is a revised version of a previous scheme (BSES) (Reference Lassere, Johnson and Boers38), proposed in 2010. The BSES3 validation framework consists of four domains: study design, target endpoint, statistical evaluation, and generalizability. Details of the elements that comprise these domains are shown in the online data supplement (Supplementary Table 1 which can be viewed online at http://dx.doi.org/10.1017/S0266462314000300). Each domain is ranked from 0 to 3 and combined to determine an overall score (ranging from 0 to 12). A hierarchical scale of validity is attached to the overall score, with “A” corresponding to highest validity (i.e., overall score 12) and “F-” to lowest. The developers suggest that an overall score of 9 or above, equivalent to a category of “A” or “B,” is required to identify a good level of evidence of surrogate validation.
RESULTS
Characteristics of Included Meta-analyses
Of the 758 papers identified by citation searching, thirty-one publications were included. Figure 1 summarizes the selection process, whereas Table 1 presents a summary of the characteristics of the included meta-analyses. Details for each meta-analysis are provided in Supplementary Table 2, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000300. The majority of them (N = 24, 77 percent) restricted their analyses to a single tumor type, although some reported separate analyses for two (Reference Burzykowski and Buyse26;Reference Johnson, Ringland and Stokes29;Reference Bowater, Lilford and Lilford39;Reference Burzykowski, Molenberghs, Buyse, Geys and Renard40), or more tumor types (Reference Bowater, Bridge and Lilford25;Reference Amir, Seruga, Kwong, Tannock and Ocana41;Reference Wilkerson and Fojo42). Two meta-analyses (Reference Polley, Lamborn and Chang43;Reference Ballman, Buckner and Brown44) of patients with glioblastoma multiforme were included; the poor median survival and the fact that metastases are seldom found in this disease suggest that PFS and OS would be important endpoints. The most frequent tumor types examined were colorectal cancer (Reference Bowater, Bridge and Lilford25–Reference Johnson, Ringland and Stokes29;Reference Bowater, Lilford and Lilford39–Reference Wilkerson and Fojo42;Reference Louvet, de Gramont, Tournigand, Artru, Maindrault-Goebel and Krulik45–Reference Buyse, Burzykowski and Carroll47) non–small-cell lung cancer (NSCLC) (Reference Bowater, Bridge and Lilford25;Reference Johnson, Ringland and Stokes29;Reference Amir, Seruga, Kwong, Tannock and Ocana41;Reference Hotta, Fujiwara and Matsuo48–Reference Hotta, Kiura and Fujiwara52), breast cancer (Reference Bowater, Bridge and Lilford25,Reference Sherrill, Amonkar and Wu30,Reference Bowater, Lilford and Lilford39,Reference Amir, Seruga, Kwong, Tannock and Ocana41,Reference Wilkerson and Fojo42,Reference Burzykowski, Buyse and Piccart-Gebhart53–Reference Miksad, Zietemann and Gothe55), and ovarian cancer (Reference Buyse, Molenberghs, Burzykowski, Renard and Geys9,Reference Burzykowski and Buyse26,Reference Burzykowski, Molenberghs, Buyse, Geys and Renard40–Reference Wilkerson and Fojo42,Reference Rose, Tian and Bookman56,Reference Sundar, Wu, Hillaby, Yap and Lilford57).
Process of screening and identification of included meta-analyses.
Summary of the Characteristics of Included Meta-analyses, N = 31
IPD, individual patient data; NSCLC, non-small-cell lung cancer; PFS, progression-free survival; TTP, time to progression.
aPFS and TTP analyzed as two distinct endpoints.
bPFS and TTP analyzed as single endpoint.
Eighteen meta-analyses were based on aggregate data, while thirteen used individual patient data (IPD). In the aggregate-data meta-analyses, the number of included trials per meta-analysis ranged from 13 to 191 (median, 39) and the number of patients per meta-analysis ranged from approximately 4,300 to 44,000 (median, 15,850). For IPD meta-analyses, these numbers were lower, ranging from two to 27 trials (median, four) and 193 to 3,953 patients (median 1,158). Aggregate-data meta-analyses frequently reported using a systematic literature search to identify included studies (15/18, 84 percent) whereas none of the IPD meta-analyses stated so. The criteria used to select included trials varied markedly across meta-analyses. The scope of meta-analyses were determined by type of intervention (e.g., gefitinib or erlotinib monotherapy) (Reference Li, Liu, Gu and Wang49), line of therapy (e.g., first-line) (Reference Johnson, Ringland and Stokes29;Reference Louvet, de Gramont, Tournigand, Artru, Maindrault-Goebel and Krulik45;Reference Tang, Bentzen, Chen and Siu46;Reference Hotta, Fujiwara and Matsuo48;Reference Hotta, Kiura and Fujiwara52), or other trial characteristics (e.g., sample size) (Reference Chirila, Odom and Devercelli27;Reference Louvet, de Gramont, Tournigand, Artru, Maindrault-Goebel and Krulik45;Reference Tang, Bentzen, Chen and Siu46).
Statistical Methods to Assess the Association between Surrogate and Final Endpoints
A wide variety of differing methods to examine the association between surrogate and final endpoints were used across the thirty-one meta-analyses. Two broad criteria may be used to summarize these statistical methods. The first criterion is the type of meta-analysis, as noted above (meta-analyses using aggregate data and those using IPD). The second criterion is the level of association reported: ten meta-analyses (32 percent) reported on the “observation-level association” or Level-2 evidence (Reference Elston and Taylor2) or “individual-level surrogacy” (Reference Buyse, Sargent, Grothey, Matheson and de Gramont10), i.e., the association between surrogate and final endpoints regardless of the treatment effect on each of the endpoints; twelve meta-analyses (39 percent) reported the “treatment-level association” or Level-1 evidence (Reference Elston and Taylor2) or “trial-level surrogacy” (Reference Buyse, Sargent, Grothey, Matheson and de Gramont10), that is, the association between the treatment effect on the surrogate and the treatment effect on the final endpoint; and nine studies (29 percent) reported both levels of association. Combining these two criteria allowed for four core categorizations of the assessment and reporting of the association between PFS/TTP and OS: (i) meta-analyses that reported an observation-level association based on aggregate data (Reference Chirila, Odom and Devercelli27;Reference Louvet, de Gramont, Tournigand, Artru, Maindrault-Goebel and Krulik45;Reference Tang, Bentzen, Chen and Siu46;Reference Li, Liu, Gu and Wang49;Reference Hayashi, Okamoto, Taguri, Morita and Nakagawa51;Reference Hotta, Kiura and Fujiwara52;Reference Shitara, Ikeda and Yokota58), for example, single-arm median PFS/TTP versus median OS; (ii) meta-analyses that reported an observation-level association based on IPD (Reference Buyse, Molenberghs, Burzykowski, Renard and Geys9;Reference Green, Yothers and Sargent28;Reference Burzykowski, Molenberghs, Buyse, Geys and Renard40;Reference Polley, Lamborn and Chang43;Reference Ballman, Buckner and Brown44;Reference Buyse, Burzykowski and Carroll47;Reference Mandrekar, Qi and Hillman50;Reference Burzykowski, Buyse and Piccart-Gebhart53;Reference Rose, Tian and Bookman56;Reference Foster, Qi and Shi59–Reference Heng, Xie and Bjarnason61), for example, patients’ TTP versus survival time; (iii) meta-analyses that reported a treatment-level association using aggregate data (Reference Bowater, Bridge and Lilford25;Reference Chirila, Odom and Devercelli27;Reference Johnson, Ringland and Stokes29;Reference Sherrill, Amonkar and Wu30;Reference Bowater, Lilford and Lilford39;Reference Amir, Seruga, Kwong, Tannock and Ocana41;Reference Wilkerson and Fojo42;Reference Tang, Bentzen, Chen and Siu46;Reference Hotta, Fujiwara and Matsuo48;Reference Hackshaw, Knight, Barrett-Lee and Leonard54;Reference Miksad, Zietemann and Gothe55;Reference Sundar, Wu, Hillaby, Yap and Lilford57;Reference Shitara, Ikeda and Yokota58;Reference Delea, Khuu, Heng, Haas and Soulieres62), for example, hazard ratio (HR) for PFS/TTP versus HR for OS; and (iv) meta-analyses that reported a treatment-level association using IPD (Reference Buyse, Molenberghs, Burzykowski, Renard and Geys9;Reference Burzykowski and Buyse26;Reference Green, Yothers and Sargent28;Reference Burzykowski, Molenberghs, Buyse, Geys and Renard40;Reference Buyse, Burzykowski and Carroll47;Reference Burzykowski, Buyse and Piccart-Gebhart53;Reference Foster, Qi and Shi59). An overview of the statistical methods used presented according to these four categorizations is provided in a supplementary technical note, with further details shown in Supplementary Tables 2 and 3, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000300.
Assessment of the Validity of PFS and TTP as Surrogates for OS
The main results of meta-analyses on the potential role of PFS or TTP as surrogates for OS are presented in Tables 2 and 3, respectively. The validity of these candidate surrogates was assessed according to the Elston and Taylor’s, IQWiG, and BSES3 frameworks applied to each meta-analysis, grouped according to the tumor type. An extract of the original authors’ conclusions on the surrogacy of PFS or TTP is also presented for each meta-analysis. The four most frequently evaluated advanced solid tumors were colorectal cancer, NSCLC, breast cancer, and ovarian cancer. While the available evidence consistently shows an association between treatment effects on PFS or TTP and treatment effect on OS (i.e., Level 1 evidence according to Elston and Taylor's framework) in metastatic colorectal cancer, the validity of these surrogate measures appear relatively low when rated by both the IQWiG and BSES3 frameworks (Tables 2 and 3). However, four studies (Reference Burzykowski and Buyse26–Reference Green, Yothers and Sargent28;Reference Buyse, Burzykowski and Carroll47) provide an “indication” of an effect on the final endpoint given the effect observed on PFS, according to the IQWiG framework (Table 2). Nevertheless, as these analyses were limited to trials within a specific treatment setting (i.e., the comparison of fluorouracil (FU) plus leucovorin with either FU alone or with raltitrexed) (Reference Buyse, Burzykowski and Carroll47) and did not provide evidence across different risk populations and drug-class mechanisms, they were scored down on the BSES3 framework. For advanced lung cancer, three meta-analyses (Reference Li, Liu, Gu and Wang49,Reference Mandrekar, Qi and Hillman50,Reference Heng, Xie and Bjarnason61) only reported observation-level association between PFS and OS in NSCLC; in small-cell lung cancer, Foster et al. (Reference Foster, Qi and Shi59) reported high correlation (R2 trial = 0·79) between HR observed on PFS and OS (on the log scale), thus providing an “indication” for an effect on OS having observed an effect on PFS according to the IQWiG framework (Table 2). TTP does not appear to be a good surrogate measure in advanced lung cancer according to any of the three frameworks (Table 3). In metastatic breast cancer, despite the moderate to high quality of the meta-analyses assessed (Reference Wilkerson and Fojo42;Reference Burzykowski, Buyse and Piccart-Gebhart53;Reference Miksad, Zietemann and Gothe55), PFS is not judged to be a valid surrogate for OS according to the three evaluation frameworks adopted (Table 2). However, Hackshaw and colleagues (Reference Hackshaw, Knight, Barrett-Lee and Leonard54) reported a medium association between TTP and OS (R2 = 0·56) in trials of first-line chemotherapy, which provided a “hint” for an effect on the final endpoint according to the IQWiG framework (Table 3). In metastatic ovarian cancer, three IPD meta-analyses, two related to PFS (Table 2) (Reference Burzykowski and Buyse26;Reference Burzykowski, Molenberghs, Buyse, Geys and Renard40), and one to TTP (Reference Buyse, Molenberghs, Burzykowski, Renard and Geys9) (Table 3) show an indication of an effect on OS drawn on the observation of an effect on the two surrogate endpoints, with R2 trial ranging from 0·83 to 0·95. Nonetheless, as according to the BSES3 criteria (see Supplementary Table 1) they lack generalizability, these studies were scored down. The remaining six solid tumor types (renal, prostate, brain, gastric, head and neck, and pancreatic) were each assessed in one or two meta-analyses (Tables 2 and 3). Across these indications, the level of evidence was mixed and the strength was poor; moreover, the endpoints were not always clearly specified, therefore, all scores for strength of surrogacy relationship were low in both the IQWiG and BSES3 frameworks (in brain and gastric cancer Level 2 was the highest level of evidence according to Elston and Taylor's framework).
Assessment of the Validity of PFS as Surrogate for OS: Comparison of Meta-analyses by Tumor Type across Evaluation Frameworks
Note. § indicate meta-analyses using individual patient data.
CRC, colorectal cancer; CRPC, castrate-resistant prostate cancer; EGFR-TKIs, epidermal growth factor receptor tyrosine-kinase inhibitors; FU, fluorouracil; GBM, glioblastoma multiforme; NSCLC, non-small-cell lung cancer; OS, overall survival; PFS, progression-free survival; PPS, post-progression survival; RCC, renal cell carcinoma; RCT, randomized controlled trial; SCLC, small-cell lung cancer; TE, treatment effect, TTP, time to progression.
*Taxanes; ** Anthracyclines.
a Level 1 corresponds to treatment-level association, i.e. evidence showing treatment effects on the surrogate correspond to treatment effects on the final patient-relevant endpoint. Level 2 corresponds to evidence showing association between the two endpoints.
bReliability is assessed according to (i) use of appropriate statistical approach, (ii) robustness and generalizability of results, (iii) systematic compilation of data, (iv) sufficient restriction of indications, degrees of disease severity, interventions and (v) clear definitions of endpoints. Low, moderate, limited and high indicate growing level of reliability. High correlation corresponds to R ≥ 0.85 whilst low correlation to R ≤ 0.70. Correlation is not even assessed if the study is of low reliability. The conclusion about the effect on the final endpoint drawn from the effect observed on the surrogate can be a no proof, hint, indication or proof according to increasing level of validity of the surrogate endpoint.
cOverall score sums up scores from 0 to 3 obtained in each of the four domain (i.e., study design, target endpoint, statistical evaluation and generalizability). Category A and B of level of evidence correspond to good evidence for validity of the surrogate endpoint. If the score is lower than 2 in any domain, the level of evidence drops by one alphabetic category.
Assessment of the Validity of TTP as Surrogate for OS: Comparison of Meta-analyses by Tumor Type across Evaluation Frameworks
CRC, colorectal cancer; EGFR-TKIs, epidermal growth factor receptor tyrosine-kinase inhibitors; NSCLC, non-small-cell lung cancer; OS, overall survival; PFS, progression-free survival; PPS, post-progression survival; RR, response rate; TTP, time to progression.
§ indicate meta-analyses using individual patient data.
aSee Table 2.
bSee Table 2.
cSee Table 2.
DISCUSSION
We sought to review the current statistical approaches to surrogate endpoint validation in advanced solid tumors, as well as to assess the suitability of PFS and TTP as surrogates for OS using currently available validation frameworks (35). Our review included thirty-one meta-analyses (1,363 RCTs enrolling more than 290,000 patients) and showed that a variety of statistical methods have been used to examine the relationship between PFS or TTP and OS. In addition, we observed a degree of variation in validity rating when using different validation frameworks across meta-analyses in general and even within a particular tumor type.
The various statistical methods used thus far in surrogacy research can be summarized in two broad categorizations. First, according to whether the assessment of the statistical association is made between the surrogate and final endpoint (observation-level association, which does not take treatment into account and is, therefore, an assessment of the prognostic role of the candidate surrogate), or between the treatment effects on both surrogate and final endpoints (treatment-level association, which assesses the predictive role of the candidate surrogate by taking treatment into account). Second, according to whether aggregate data or IPD were used. Observation-level association has been reported both using aggregate data and IPD, with different metrics used to quantify the correlation between endpoints (e.g., Spearman's ρ for median PFS versus median OS in the former (Reference Chirila, Odom and Devercelli27;Reference Louvet, de Gramont, Tournigand, Artru, Maindrault-Goebel and Krulik45;Reference Tang, Bentzen, Chen and Siu46;Reference Hayashi, Okamoto, Taguri, Morita and Nakagawa51;Reference Shitara, Ikeda and Yokota58) and R2 individual in the latter) (Reference Buyse, Molenberghs, Burzykowski, Renard and Geys9). In several cancer types, such as metastatic gastric cancer and glioblastoma multiforme, only the observation-level association has been investigated so far. This is acknowledged to be insufficient evidence to establish surrogacy for putative surrogate endpoints (Reference Buyse, Sargent, Grothey, Matheson and de Gramont10). For most tumor types, including colorectal cancer, breast cancer, and NSCLC, both observational-level and treatment-level surrogacy has been investigated, with treatment-level surrogacy being assessed using both IPD and aggregate data. Although treatment-level associations were often reported using the common statistic of R2 trial, this was calculated using different analytic approaches (e.g., meta-regression for aggregate data (Reference Chirila, Odom and Devercelli27;Reference Johnson, Ringland and Stokes29;Reference Sherrill, Amonkar and Wu30;Reference Amir, Seruga, Kwong, Tannock and Ocana41;Reference Wilkerson and Fojo42;Reference Louvet, de Gramont, Tournigand, Artru, Maindrault-Goebel and Krulik45;Reference Tang, Bentzen, Chen and Siu46;Reference Miksad, Zietemann and Gothe55;Reference Delea, Khuu, Heng, Haas and Soulieres62) and hierarchical regression methods for IPD (Reference Buyse, Molenberghs, Burzykowski, Renard and Geys9;Reference Burzykowski and Buyse26;Reference Green, Yothers and Sargent28;Reference Burzykowski, Molenberghs, Buyse, Geys and Renard40;Reference Buyse, Burzykowski and Carroll47;Reference Burzykowski, Buyse and Piccart-Gebhart53;Reference Foster, Qi and Shi59).
There is little literature directly comparing statistical validation of surrogates using IPD compared with aggregate level data (Reference Green, Yothers and Sargent28). Buyse and colleagues have proposed IPD meta-analysis and calculation of both the R2 individual and R2 trial to be the gold standard approach to the statistical surrogate validation (Reference Buyse, Sargent, Grothey, Matheson and de Gramont10). However, only 22 percent of the meta-analyses in this review met this criterion. Such a low proportion is in large part due to the practical challenges of conducting an IPD meta-analysis. Gathering, cleaning and formatting patient data from across clinical trial centers involves substantial resources, as due to commercial or academic restrictions, IPD for some trials are not immediately available in the public domain. While regulatory agencies can require companies to make such data available, this is often not the case for HTA organizations or agencies with a coverage or reimbursement mandate. Hence, while an IPD meta-analytic approach remains the optimal statistical approach to surrogate validation, it is likely that meta-analyses of treatment-level associations reporting the R2 trial or equivalent statistics will continue to be undertaken. There is often a lack of appreciation that the use of aggregate data entails a loss of information that may have a profound impact on the analyses performed, and their interpretation. For instance, several meta-analyses included in our review used the ratio of medians as a measure of treatment effects (Reference Chirila, Odom and Devercelli27;Reference Sherrill, Kaye, Sandin, Cappelleri and Chen32;Reference Hotta, Fujiwara and Matsuo48;Reference Hackshaw, Knight, Barrett-Lee and Leonard54;Reference Miksad, Zietemann and Gothe55). Such an approach could be seriously misleading if the time to event distributions were not exponential and, even if they were, the medians usually have wide confidence intervals and so their ratio is likely to be extremely unstable (Reference Parmar, Torri and Stewart63). Few regression analyses make proper allowance for the estimation error (Reference Buyse, Molenberghs, Burzykowski, Renard and Geys9;Reference Burzykowski and Buyse26;Reference Green, Yothers and Sargent28;Reference Burzykowski, Molenberghs, Buyse, Geys and Renard40;Reference Burzykowski, Buyse and Piccart-Gebhart53) other than through a weighting of the trials by their sample size. Regression analyses that ignore estimation errors are likely to underestimate the true relationship between the treatment effects on the surrogate and the final endpoint. The availability of IPD allows the association between the surrogate and the final endpoints to be modeled, which is theoretically preferable to looking only at the marginal association between the treatment effects on the two endpoints (Reference Buyse, Molenberghs, Burzykowski, Renard and Geys9).
Limitations of This Study
To the best of our knowledge, this is the first study to empirically test the application of current surrogate validation frameworks across a sample of meta-analyses in a disease area. On the other hand, our study has some limitations. First, as we were unable to use a conventional search strategy, we cannot claim to have identified all relevant meta-analyses. A more exhaustive search might have been feasible if we had narrowed our scope to a single tumor type. However, our aim was to keep the scope of the study broad and to identify a sufficient number of meta-analyses to assess a variety of statistical methods. Our list of included meta-analyses appears indeed to be comprehensive when compared with recent reviews in the field (12;Reference Sherrill, Kaye, Sandin, Cappelleri and Chen32). Second, we have not formally appraised the overall quality of each meta-analysis. Given that the focus of the study was not to determine an unbiased estimate of the efficacy or safety of interventions, we believe that this decision was justified. However, to assess potential selection or publication bias, we noted if each included meta-analysis reported undertaking a formal literature search strategy to identify studies. In line with the findings of previous studies, we found that meta-analyses of aggregate data were more likely to undertake a literature search and include more studies than IPD meta-analyses (Reference Hayashi, Okamoto, Taguri, Morita and Nakagawa51). Third, we have not attempted to replicate any of the analyses presented in the included meta-analyses. This might have been useful, as it would allow us to examine whether all of the assumptions made in the presented analyses are supported by the primary data and whether the conclusions change when all relevant trials are considered in a single analysis and after updating for more recently published trials. Fourth, the application of both the IQWiG and BSES3 evaluation frameworks involved an element of subjective judgment. To minimize potential assessment bias, the application of the frameworks undertaken was independently checked by a second reviewer and a third reviewer used to resolve disagreements in judgment of these two reviewers. All of them were HTA analysts with experience in the field of oncology. Finally, although survival is a definitive patient-relevant outcome in the case of most solid tumors, there may be problems with using of OS in the context of surrogate validation. Despite its primacy, OS has been claimed to be unsuitable in detecting treatment benefit in settings for which effective therapy is available after trial participation (Reference Buyse, Sargent and Saad64). Because patients in oncology trials are often permitted to cross over from the control arm to the treatment arm or switch to other therapies due to lack of response or symptoms, the attribution of OS gain to initial treatment allocation may be confounded by these subsequent lines of therapy (Reference Saad, Katz, Hoff and Buyse65).
Implications for Policy and Practice
Surrogate validation studies also have important relevance for the assessment of the cost effectiveness of new treatments (Reference Ciani and Taylor66). Using a reported relationship between OS and the surrogate, decision analysts can estimate the incremental cost per quality-adjusted life-year (QALY) based on the observed treatment effect on the surrogate (Reference Elston and Taylor2).
Our study has important implications for the use of surrogate outcomes in HTA and coverage/reimbursement policy decisions. To appropriately apply evidence of surrogate validation, policy makers need decision frameworks that help them do so. While the IQWiG and BSES3 frameworks are potentially useful tools for clinicians and healthcare decision makers, there are problems in their practical application. Both have elements that require subjective judgment. In addition, they require a high level of association to demonstrate surrogacy, that is, R2 treatment ≥0·60 or Rtreatment ≥ 0·85, raising a query on the origins of such thresholds. With a small number of exceptions, we found the strength of the association between PFS or TTP and OS across meta-analyses to be consistently low (i.e., Rtreatment < 0·7) across tumor types. Indeed, according to the IQWiG and BSES3 validation frameworks, the evidence available about surrogacy of PFS and TTP in metastatic cancer is still insufficient to guide policy. Moreover, we noted a degree of variation in validity rating of IQWiG and BSES3 frameworks across meta-analyses within a particular tumor type. For example, for PFS in colorectal cancer, four meta-analyses (Reference Burzykowski and Buyse26–Reference Green, Yothers and Sargent28;Reference Buyse, Burzykowski and Carroll47) showed an “indication” of an effect on OS given an effect observed on the surrogates, however the highest level of evidence achieved according to BSES3 is C, well below the minimum acceptable level for a good surrogate. We believe that this variation probably reflects differences in the evidence within each meta-analysis due to differences in the precise patient population, definition and assessment of progression, drug therapy and comparator of included trials. Moreover, variation may also be due to differences in the statistical methods applied by each meta-analysis. Finally, the criteria considered by the two evaluation frameworks are different and in some cases opposite; for instance, BSES3 favors generalizability across populations and drug-class mechanisms while the IQWiG framework gives precedence to restricted indications and therapies. Nonetheless, within each indication, different meta-analyses deal with overlapping evidence, and the underlying redundancy may have accounted for similar conclusions. When considering conclusions across indications, our results support the need for a disease-specific approach to the validation of surrogate endpoints, with careful consideration of transferability of results from one disease to the other.
The three evaluation tools used herein were developed through different processes: Elston and Taylor's framework was based on a guide for clinicians proposed by the Journal of the American Medical Association (JAMA) Evidence-Based Medicine Working Group; the algorithm for surrogate endpoints validation in oncology was developed at IQWiG after a systematic search of the literature by the Agency; whereas the BSES3's initial version originated from a literature review followed by a stakeholder workshop that evaluated it for applications in rheumatology (Reference Lassere, Johnson and Boers38). We believe that the development of future surrogate validation tools would benefit from formal consensus methods. Further research is needed to examine the application of surrogate validation frameworks in the context of candidate surrogates both in oncology and other disease areas.
In conclusion, we found that the level of evidence available supporting a relationship between PFS or TTP and OS varies considerably by tumor type and is not always consistent even within one specific type. Overall, the strength of the association between PFS or TTP and OS was relatively low and only PFS in advanced colorectal and ovarian cancers treated with cytotoxic agents was found to be a valid surrogate endpoint according to one of the evaluation frameworks used. Our study emphasizes the importance of building consensus on appropriate statistical techniques to examine surrogacy and on development of evaluation frameworks, not only in oncology but across all areas of medicine, across jurisdictions and scientific communities.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: http://dx.doi.org/10.1017/S0266462314000300
Supplementary Table 2: http://dx.doi.org/10.1017/S0266462314000300
Supplementary Table 3: http://dx.doi.org/10.1017/S0266462314000300
CONTACT INFORMATION
Oriana Ciani, PhD (o.ciani@exeter.ac.uk), University of Exeter Medical School, Veysey Building, Salmon Pool Lane, Exeter, EX2 4SG, United Kingdom
CONFLICTS OF INTEREST
OC was funded through a Peninsula College of Medicine and Dentistry studentship. A preliminary version of this paper was based on a report which was funded by the National Institute for Health and Care Excellence through its Decision Support Unit. The views expressed in the paper are those of the authors. MB declares an association with the International Drug Development Institute and EDS with Dendrix Ltd. All other co-authors declare no conflict of interest.
