Highlights
What is already known?
-
• Estimands are increasingly used to formulate research questions in confirmatory clinical trials and throughout the life cycle of pharmaceutical research.
-
• In evidence synthesis, the population, intervention, comparator, and outcome (PICO) framework is typically used to pose clinical research questions, and the uptake of estimands has been modest.
-
• While PICOs and estimands have overlapping attributes, the estimand framework additionally places emphasis on the relevance of the population-level summary measure and different intercurrent event strategies.
What is new?
-
• We propose a pragmatic framework for the application of estimands to meta-analyses of clinical trials, within the specific context of health technology assessment.
-
• We apply the estimand framework to a network meta-analysis comparing the efficacy of semaglutide versus dulaglutide in type 2 diabetes, exploring the impact of different intercurrent event strategies.
-
• We highlight the value of estimands to systematically address quantitative heterogeneity between trials, and to maximize the applicability of pooled estimates relative to healthcare decision-making contexts.
Potential impact for RSM readers
-
• We highlight the relevance of different intercurrent event strategies to the conduct of meta-analyses.
-
• We advocate for the integration of estimands, as a complement to PICOs, into the planning of meta-analyses.
-
• Estimands can ensure that evidence syntheses are maximally relevant to healthcare decision-makers, while strengthening communication between stakeholders about what evidence syntheses seek to demonstrate.
1 Introduction
There has been a shift from broad to more specific research questions in the design and analysis of confirmatory clinical trials, stimulated by the recent introduction of the estimand framework.Reference Kahan, Hindley, Edwards, Cro and Morris 1 Estimands have been gradually adopted by regulatory agencies worldwide, 2 are now fundamental in the protocols and statistical analysis plans of Phase 3 clinical trials, and have been incorporated into guidelines, such as CONSORT and SPIRIT,Reference Kahan, Hall and Beller 3 , Reference Kahan, Hall and Beller 4 which suggests that the reporting of estimands will become standard practice in clinical trial scientific publications. Estimands now encompass almost the entire life cycle of pharmaceutical research, with applications from early clinical developmentReference Homer, Yap and Bond 5 – Reference Lynggaard, McKendrick and Baird 7 through to observational and post-authorization safety studies.Reference Ma, Haddad, Liu, Snyder, Bennett and Mayo 8 – Reference Luijken, van Eekelen, Gardarsdottir, Groenwold and van Geloven 10
A similar transition from broad to more narrow research questions has taken place in evidence synthesis for healthcare decision-making. In health technology assessment (HTA), evidence synthesis methods are used to inform policy and reimbursement decisions made for specific clinical settings with well-defined target populations.Reference Ades, Welton, Dias, Phillippo and Caldwell 11 , Reference Remiro-Azócar and Gorst-Rasmussen 12 Given this narrow perspective of research questions, evidence synthesis faces two important challenges: (1) quantitative heterogeneity between studies; that is, variation in treatment effects due to trials addressing different research questions, with potential misalignments in patient populations, treatment implementations, and outcomesReference Ades, Welton, Dias, Phillippo and Caldwell 11 and (2) the, often ambiguous, applicability or external validity of pooled treatment effect estimates with respect to the decision-making context.Reference Manski 13 – Reference Rott, Bronfort and Chu 16
The term “external validity” often refers to the extent to which inferences, drawn from a specific analysis, apply to a target population.Reference Degtiar and Rose 17 , Reference Findley, Kikuta and Denly 18 In this article, we take a more general view of external validity or “applicability”—defining these terms as the extent to which inferences apply to the decision-making context (in the HTA case, routine clinical practice in the corresponding jurisdiction). Namely, we do not exclusively relate the terms to the population element of research questions,Reference Egami and Hartman 19 but also to other dimensions corresponding to attributes of estimands. In particular, we focus on intercurrent events and intercurrent event strategies, at the heart of the estimand framework.
To produce results that are most relevant for HTA decision-making, every effort should be made to: (1) mitigate quantitative heterogeneity between the studies in the evidence baseReference Ades, Welton, Dias, Phillippo and Caldwell 11 and (2) optimize the external validity of pooled estimates with respect to the decision-making context.Reference Dahabreh, Robertson, Petito, Hernán and Steingrimsson 20 The estimand framework holds promise to address these challenges. Firstly, it aligns with a “narrow” perspective of research questions, aiming to reduce the heterogeneity stemming from ambiguous questions. Secondly, it is a language that is used in clinical trial design and reporting and will become increasingly prevalent in publications informing future evidence synthesis work.
In this article, we discuss the application of the estimand framework to evidence synthesis, focusing on the specific context of HTA. The following base-case scenario is considered. A pairwise or network meta-analysis, combining the results of several randomized controlled trials (RCTs), is required for HTA. Each RCT has target estimands of its own and has been designed to establish the relative efficacy, safety, and benefit–risk of a novel treatment, thereby supporting regulatory approval. While head-to-head comparisons versus placebo may suffice for regulatory approval, HTA bodies must perform comparisons versus all treatment options in clinical practice. Network meta-analyses of direct and indirect treatment comparisons are typically used for this purpose, as highlighted by methodological guidelines for joint clinical assessments (JCAs) under the new European Union (EU) HTA Regulation. 21
To exemplify the use of the estimand framework in evidence synthesis, we apply estimands to a network meta-analysis comparing the efficacy of semaglutide versus dulaglutide in patients with type 2 diabetes (T2D).Reference Lingvay, Bauer, Baker-Knight, Lawson and Pratley 22 This practical example shows how estimands can be used as a complement to the population, intervention, comparator, and outcome (PICO) framework, to systematically identify, explain, and address key sources of quantitative heterogeneity, and to enhance external validity, thereby ensuring that the synthesized evidence is maximally relevant to healthcare decision-making. While our example features a network meta-analysis, leveraging both direct and indirect evidence, the principles are also applicable to pairwise meta-analyses.
This article results from the work of the Estimands Implementation Working Group (EIWG) sub-team in HTA and real-world evidence (RWE). The EIWG was established by the European Federation of Pharmaceutical Industries and Associations (EFPIA) and the European Federation of Statisticians in the Pharmaceutical Industry (EFSPI), upon recognizing practical challenges for the implementation of estimands in clinical trials and throughout the life cycle of medicinal products. As part of the HTA and RWE sub-team, we have focused on the implementation of estimands across late-phase activities. Nevertheless, our findings are also applicable to the regulatory pre-market authorization setting. For instance, the 2025–2027 three-year work plan of the European Medicines Agency Methodology Working Party lists “guidance on how to align estimand attributes across different trials in the context of a meta-analysis” as a key priority, 23 and (network) meta-analyses of historical trials are often used to determine the margin of confirmatory active controlled trials.Reference Lynggaard, Keene, Mütze and Rehal 24 , 25
2 What is an estimand?
The design and analysis of confirmatory clinical trials are guided by principles from the International Council for Harmonization (ICH) of technical requirements for pharmaceuticals on human use. An addendum to the ICH guideline on statistical principles for clinical trials, ICH E9 (R1), published in 2019, proposes the estimand framework as a systematic approach to specify clinical research questions. 26 An estimand is a precise definition of the treatment effect that is targeted by a clinical trial, which should align the trial’s planning, design, conduct, and analysis with the scientific question posed by investigators. 26
According to the final version of ICH E9 (R1), an estimand is defined by five attributes: (1) population, (2) treatment, (3) variable (or endpoint), (4) population-level summary measure (the addendum uses the term “population-level summary”), and (5) strategies for intercurrent events. An intercurrent event is a post-baseline event that affects either the interpretation or existence of endpoint measurements. Examples of intercurrent events are treatment discontinuation, initiation of rescue medication, and terminal events like death. The estimand framework is explicit in outlining five potential strategies for handling intercurrent events:
-
• Treatment policy includes any effect resulting from the occurrence of the intercurrent event in the definition of the treatment effect.
-
• Hypothetical envisages a hypothetical scenario in which the intercurrent event would not occur.
-
• Composite variable incorporates the occurrence of the intercurrent event into the endpoint definition.
-
• While-on-treatment considers the treatment effect only prior to the occurrence of the intercurrent event.
-
• Principal stratum restricts the population to a subpopulation in which the intercurrent event would not occur.
In this article, we focus on the treatment policy and hypothetical strategies, but see Kahan et al.,Reference Kahan, Hindley, Edwards, Cro and Morris 1 Keene et al.,Reference Keene, Lynggaard, Englert, Lanius and Wright 27 and Clark et al.Reference Clark, Kahan, Phillips, White and Carpenter 28 for more information about different strategies for handling intercurrent events.
ICH E9 (R1) was largely motivated by a perceived lack of clarity in the handling of intercurrent events. In the pre-estimands era, analytical approaches denoted as “intention-to-treat” (ITT) or “per-protocol” handled intercurrent events ambiguously.Reference Keene, Lynggaard, Englert, Lanius and Wright 27 , Reference Clark, Kahan, Phillips, White and Carpenter 28 For instance, the ITT principle clearly aligns with a treatment policy strategy and should include all randomized patients under their randomized treatment allocation. ICH E9 (R1) states: 26
ICH E9 introduced the ITT principle in connection with the effect of a treatment policy in a randomised controlled trial, whereby subjects are followed, assessed and analysed irrespective of their compliance to the planned course of treatment (
$\ldots $
) Multiple consequences arising from the ITT principle can be distinguished. Firstly, that the trial analysis should include all subjects relevant for the research question. Secondly, that subjects should be included in the analysis as randomised. Taken directly from the definition of the ITT principle (see ICH E9 Glossary), a third consequence is that subjects should be followed-up and assessed regardless of adherence to the planned course of treatment and that those assessments should be used in the analysis.
However, many analyses reported as ITT “ignored” intercurrent events by stopping to follow-up participants after the occurrence of the events. As such, outcome data for subjects experiencing the intercurrent events would be excluded from the estimation, thereby diverging from the ITT principle. Moreover, treatment effects may have inadvertently been estimated in hypothetical settings where intercurrent events do not take place, for example, truncating intercurrent events in mixed models for repeated measures or censoring at the intercurrent event in time-to-event settings.Reference Clark, Kahan, Phillips, White and Carpenter 28 – Reference Mitroiu, Oude Rengerink, Teerenstra, Pétavy and Roes 30
ICH E9 (R1) highlights the potential relevance of alternative intercurrent event strategies beyond treatment policy, with Morga et al. recently illustrating the relevance of these strategies to HTA decision-making.Reference Morga, Latimer, Scott, Hawkins, Schlichting and Wang 31 Consider certain cancer trials, where due to perceived ethical reasons, patients randomized to the control group can switch onto the experimental treatment, as rescue medication following disease progression.Reference Manitz, Kan-Dobrosky and Buchner 32 , Reference Latimer, Abrams and Lambert 33 HTA concerns the evaluation of “real-world” clinical effectiveness. However, patients under the current standard of care in clinical practice cannot initiate the experimental intervention, because it is not available in the healthcare system.
As argued by Latimer et al.Reference Latimer, Abrams and Lambert 33 and others,Reference Morga, Latimer, Scott, Hawkins, Schlichting and Wang 31 , Reference Jackson, Ran and Zhang 34 a treatment policy strategy that includes any effect resulting from treatment switching as part of the treatment effect definition leads to a treatment pathway that is not relevant for the HTA decision problem. Key to the selection of any estimand is its relevance to stakeholders, and different intercurrent event strategies are likely to serve different stakeholders.Reference Keene, Ruberg and Schacht 35 , Reference Polverejan, O’Kelly, Hefting, Norton, Lim and Walton 36 Similarly, the relevance of an estimand critically depends on the context. Ideally, estimand choice should not be driven by habitual choices of analytical methods and their convenience but by the clinical research question.
A hypothetical strategy for treatment switching, considering the setting where rescue medication is unavailable, would offer greater external validity relative to the HTA decision-making context.Reference Jackson, Ran and Zhang 34 Nevertheless, its estimation rests on unverifiable assumptions—when imputing outcomes for the control patients that have switched treatment—which may undermine the internal validity offered by randomization. The Institute for Quality and Efficiency in Health Care in Germany has prioritized internal over external validity and is opposed to diverging beyond treatment policy (and composite variable strategies for terminal events like death, where post-intercurrent event outcomes do not conceptually exist).Reference Morga, Latimer, Scott, Hawkins, Schlichting and Wang 31 While these strategies are generally considered easier to estimate, note that their estimation may still require strong assumptions in the presence of missing outcomes.Reference Bell, Drury and Mütze 37
Other HTA agencies, such as the National Institute for Health and Care Excellence (NICE) in England and Wales, acknowledge that departures from treatment policy might be useful.Reference Morga, Latimer, Scott, Hawkins, Schlichting and Wang 31 This is a view which we share, particularly since HTA demands for external validity, and also since payers assess clinical effectiveness at the level of their jurisdiction. Treatment pathways will differ between countries or regions. A treatment policy strategy where patients switch to a treatment that is unavailable in the local context is arguably of little relevance to HTA decision-makers in such context.
3 Estimands for evidence synthesis
3.1 Estimands as complements to PICOs
We focus on the specific context of evidence synthesis for HTA, where the uptake of estimands has been modest, and where the strategy for intercurrent events is often a secondary consideration.Reference Remiro-Azócar and Gorst-Rasmussen 12 For instance, the Cochrane Risk of Bias 2 (RoB 2) tool,Reference Sterne, Savović and Page 38 recommended by HTA guidance to assess RCT risk-of-bias in evidence syntheses, 39 does not reflect estimands and is misaligned with the intercurrent event strategies presented by ICH E9 (R1).Reference Poythress and Morga 40
In evidence synthesis and HTA, the PICO framework is typically used to formulate research questions. PICOs and estimands are closely connected, with the former preceding ICH E9 (R1) by several decades.Reference Richardson, Wilson, Nishikawa and Hayward 41 The population is included in both frameworks; the intervention and comparator elements of PICO correspond to the treatment attribute of estimands; and the outcome element of PICO is similar to the variable (or endpoint) attribute of estimands, but often broader.Footnote 1 While strategies for handling intercurrent events are at the heart of ICH E9 (R1), these—and the population-level summary measure—are not considered by PICO. All else being equal, PICOs formulate broader research questions than estimands, as estimands contain two additional attributes.
In this manuscript, we do not consider the use of PICO to formulate research questions at the individual trial level, but to postulate research questions across the synthesis of multiple trials. PICOs are typically used to guide the data extraction process for systematic literature reviews and meta-analysesReference Frandsen, Nielsen, Lindhardt and Eriksen 43 and are recommended by the PRISMA statement and its network meta-analysis extension.Reference Moher, Liberati, Tetzlaff and Altman 44 , Reference Hutton, Catala-Lopez and Moher 45 In HTA, the PICO framework is well-established to determine the scope of assessments.Reference Jen, Mt-Isa and Chu 46 Notably, the EU JCA scope is based on sets of PICO questions posed by member states.Reference van Engen, Krüger and Parnaby 47
Within the aforementioned context, the relative imprecision of the PICO framework is useful, and potentially desirable, for pragmatic or feasibility reasons. Estimands are selected by sponsors (in agreement with regulators) that can align their trial’s design, data collection, and analysis with the desired research question. Conversely, PICOs are generally policy-driven, not necessarily driven by the available data, 48 and—in an evidence synthesis context—determined after some or most trials have been conducted. As such, “meta-analysts” often rely on the analysis of secondary data, typically with limited input on primary data collection and without full access to subject-level data.Reference Remiro-Azócar, Heath and Baio 49
Moreover, presume that HTA agencies were to ask very narrow research questions, by requesting specific population-level summary measures and intercurrent event strategies as part of the scope. This would produce a sparser evidence base, of fewer trials, potentially resulting in disconnected networks that threaten the feasibility of network meta-analyses. In the EU JCA context, broader research questions can facilitate the consolidation of scopes, particularly given that a wide variability in policy questions across member states is anticipated.Reference van Engen, Krüger and Parnaby 47
Consider the JCA guidance issued by the EU Member State Coordination Group on HTA, “Guidance on the scoping process” 48 and “Guidance on outcomes for joint clinical assessments”. 42 We shall refer to these documents as the “scoping guidance” and the “outcomes guidance”, respectively. The scoping guidance states that the population for the assessment scope PICO “should be as specific as possible and avoid ambiguity.” 48 The population is based on the claimed therapeutic indication; typically, the indication in the health technology developer’s submission to the European Medicines Agency. 48 As such, the population attribute of the estimand submitted to regulators is expected to overlap with that of the PICO in the HTA scope.
While the population attributes are similarly narrow across frameworks, this is not generally the case for the intervention, comparator, and outcome. According to the EU JCA scoping guidance, “variations of the intervention, such as dose or timing of administration, (
$\ldots $
) do not require a separate PICO.”
48
Similarly, the treatments comprising a comparator can be drug classes, and “should not contain information on dosage or regimen.”
48
According to the outcomes guidance, the timing of outcomes does not need to be specified as part of the PICO scope, and outcomes can be measured in different ways without necessarily specifying the way in which they are measured.
42
Conversely, estimands should define precisely the dosage, regimen, and mode of administration of treatments, the time at which endpoints are measured, and may include the use of specific measure instruments indicating how the endpoint is assessed at the subject level.
The breadth of research questions in HTA scoping is not limited to the EU, but also apparent in other jurisdictions. Consider the NICE assessment of cabozantinib for previously treated advanced differentiated thyroid cancer, unsuitable for or refractory to radioactive iodine, for which final technology appraisal guidance (TA928) was published in November 2023. 50 The evidence from cabozantinib in the HTA submission was largely based on the results of the pivotal Phase 3 RCT COSMIC-311, also used to obtain marketing authorization worldwide.Reference Brose, Robinson and Sherman 51 For illustrative purposes, we display below the PICO elements (Table 1) in the final scope of the NICE technology appraisal of cabozantinib, 52 and the primary estimands (Table 2) of the COSMIC-311 trial. 53
PICO elements extracted from the final scope of the NICE technology appraisal of cabozantinib for previously treated differentiated thyroid cancer unsuitable for or refractory to radioactive iodine 52
Table 1 Long description
The table consists of two columns and four rows.
Row 1 (Header): The first cell is labeled Population. The second cell describes the population as adults with locally advanced or metastatic differentiated thyroid carcinoma, whose disease is refractory to, or who are unsuitable for radioactive iodine, and whose disease has progressed during or after prior systemic therapy.
Row 2: The first cell is labeled Intervention. The second cell lists Cabozantinib.
Row 3: The first cell is labeled Comparator. The second cell lists Best supportive care.
Row 4: The first cell is labeled Outcomes. The second cell lists five specific outcomes: Overall survival, Progression-free survival, Response rate, Adverse effects of treatment, and Health-related quality of life.
Primary estimands of the COSMIC-311 trial 53
Table 2 Long description
The table consists of three columns. The first column lists the Estimand attributes, while the second and third columns provide details for Objective response rate and Progression-free survival respectively.
* Population: Both estimands target patients with radioiodine-refractory differentiated thyroid cancer who have progressed after prior V E G F R targeted therapy.
* Treatments: Both involve oral cabozantinib 60 mg once daily versus a matching placebo.
* Variable endpoint: For Objective response rate, it is radiographic response per R E C I S T 1.1. For Progression-free survival, it is the duration of radiographic progression-free survival.
* Population-level summary measure: For Objective response rate, the difference in proportions of subjects with a best overall response of confirmed complete or partial response. For Progression-free survival, the difference in survival functions between treatment conditions.
* Intercurrent event strategies: Both use a Treatment policy for local radiation to bone, surgical resection of non-target lesions, or non-protocol medications for other diseases. For target lesion resection or systemic non-protocol medications, the Objective response rate uses a While on treatment strategy, whereas Progression-free survival uses a Hypothetical strategy.
The population attributes are almost identical across frameworks and are narrowly defined, corresponding to the claimed therapeutic indication. Conversely, while the treatment attribute of the estimand specifies dosage, regimen, and mode of administration, the intervention and comparator PICO attributes do not. Similarly, the estimand variables (endpoints) are more specific than the PICO outcomes, which are arguably more “patient-relevant”, 42 featuring overall survival, adverse effects of treatment, and health-related quality of life. In all cases, the estimand attributes are encompassed by their corresponding PICO elements. While the estimand framework explicitly addresses strategies for intercurrent events and defines a population-level summary measure, the PICO framework does not. The tables should demonstrate the greater precision employed by the estimand framework in specifying the clinical research question.
With this article, we do not intend to incriminate the use of PICOs, or to suggest indiscriminately replacing PICOs with estimands. We acknowledge that the breadth of PICOs is attractive for scoping, to describe the “totality” of evidence in qualitative evidence syntheses and HTA. As illustrated in Figure 1, we regard PICOs as sets of different estimands, with a given PICO potentially containing several estimand definitions.Reference Remiro-Azócar and Gorst-Rasmussen 12 Our aim in the next sections is to promote the coexistence of estimands with PICOs and the use of estimands “as a complement to the PICO framework”, as proposed by the EU JCA outcomes guidance. 42 Our proposal departs from the original idea by Remiro-Azócar of completely replacing the PICO framework with a “PICOSI” specifying the additional “Summary measure” and “Intercurrent event strategies” estimand attributes,Reference Remiro-Azócar 54 now recommended by Canada’s Drug Agency in all evidence submitted for reimbursement decision-making. 55
PICOs can be viewed as sets of different estimands, with each PICO encompassing multiple estimand definitions.Reference Remiro-Azócar and Gorst-Rasmussen 12 The “meta” level refers to the meta-analytical level, across the trials in base defined by the assessment scope PICO.
Figure 1 Long description
A large rounded rectangle labeled Meta level in the top right corner contains the entire diagram. Inside this container, at the top left, is the P I C O assessment scope, which includes a bulleted list:
* Population
* Intervention
* Comparator
* Outcome.
Below this list are three identical rounded rectangles arranged horizontally, each labeled Trial level in a teal box at its top right corner.
From left to right, these are labeled Estimand 1, Estimand 2, and Estimand 3. Each estimand box contains the same five bulleted components:
* Population
* Treatment
* Variable or endpoint
* Population-level summary measure
* Strategies for intercurrent events.
3.2 Estimands for (network) meta-analysis
The focus of our proposed framework is on pairwise and network meta-analyses, typically used to pool treatment effects across multiple RCTs. It is assumed that the RCTs have been primarily designed to support regulatory approval, each RCT with target estimands of its own. Meta-analyses of direct and indirect treatment comparisons are then required for HTA purposes, as highlighted by JCA methodological guidelines for the new EU HTA Regulation. 21
There has been limited discussion in the evidence synthesis literature to date about potential implications of the estimand framework for (network) meta-analysis. In our view, the practice of pairwise and network meta-analysis faces two important challenges in healthcare decision-making: (1) quantitative heterogeneity between trials; that is, variation in treatment effects due to trials addressing different research questionsReference Ades, Welton, Dias, Phillippo and Caldwell 11 and (2) the often ambiguous external validity or applicability of pooled treatment effect estimates with respect to specific clinical research questions.Reference Manski 13 , Reference Sobel, Madigan and Wang 14 We suspect both challenges to be more problematic in the synthesis of trials conducted by different sponsors than in the pooling of trials within the same Phase 3 development program.
The estimand framework holds promise to address these two challenges by: (1) identifying, explaining, and potentially reducing quantitative heterogeneity between the trials in the evidence base and (2) optimizing the relevance of pooled estimates with respect to the decision-making context. While it suffices to think of estimands at the individual trial level for the first challenge, the second challenge requires viewing estimands at two different hierarchical levels: (1) the trial level and (2) the meta-analytical (“meta”) level.Reference Remiro-Azócar 54
3.2.1 Aligning estimands at the trial level
The ICH E9 (R1) addendum, while lacking explicit guidelines for estimands in meta-analysis, already warns against a “naïve comparison between data sources, or integration of data from multiple trials without consideration of the estimand that is addressed in each.” 26 Individual trials are not necessarily planned with similar estimands, nor in anticipation of a meta-analysis. Almost invariably, there will be some misalignment between estimand components across trials; in patient populations, treatment implementations, endpoint definitions, reported population-level summary measures, intercurrent events, or strategies for intercurrent events.
Critically, discrepancies between estimand attributes may arise even where the high-level estimand descriptions appear to be the same. Patient populations that are seemingly identical, based on estimands’ population attribute, can differ due to discrepancies in trial selection criteria. Versions of administered treatment can differ in dosing regimen, delivery mechanism, or concomitant background medications, which may also change over time. Endpoints assessed at similar time points may have been measured using different instruments. Even the seemingly consistent definition of an intercurrent event strategy like treatment policy for treatment switching can give way to heterogeneous treatment regimens across studies, since standard of care and rescue medication are likely to change over time and over trials.
There is a need to align estimand attributes across trials, but this can be challenging, particularly under limited access to subject-level data. A manufacturer submitting evidence to HTA bodies typically has access to subject-level data from at least one “index” RCT, comparing the efficacy of its novel active treatment versus a control, but only aggregate-level summary data from “competitor” RCTs. In this scenario, the manufacturer is able to align estimands with respect to the competitor trials, but, awkwardly, not with respect to its own RCT(s).Reference Phillippo, Ades, Dias, Palmer, Abrams and Welton 56 In any case, what estimands do provide is a framework for transparent and harmonized reporting at the trial level, and for more standardized treatment effect definitions within therapy areas. As illustrated by our case study in Section 4, this is useful to explicitly identify potential misalignments in the evidence base, and to explain key sources of quantitative heterogeneity between trials, using a language that is well-established in trial design and reporting.
3.2.2 Defining estimands at the meta-analytical level
If estimands impact the interpretation of individual clinical trials, one can conclude that they will impact the interpretation of syntheses of such trials. Behind every trial-level analysis or estimator, there is an estimand being targeted. This is no different for meta-analyses. The pooling of estimates targeting different estimands results in meta-analyses that target neither of the estimands, producing an ambiguous average with dubious applicability.Reference Metcalfe, Remiro-Azócar and Vuong 57 This raises the question of whether one can define an overarching estimand at the “meta” level if there is substantial cross-trial heterogeneity between estimands.
We propose defining an “ideal” meta-analytical estimand, which can be used to restrict the evidence base and perform more targeted (network) meta-analyses, enhancing applicability relative to a specific clinical research question. One can formulate a PICO for the assessment scope, then narrow the evidence base using a target meta-analytical estimand, which more precisely defines the relevant question for healthcare decision-makers. In current practice, such target estimand is sometimes implicitly defined. For instance, where HTA agencies oppose intercurrent event strategies that diverge beyond treatment policy,Reference Morga, Latimer, Scott, Hawkins, Schlichting and Wang 31 they are implicitly imposing a treatment policy meta-analytical estimand.Reference Metcalfe, Remiro-Azócar and Vuong 57
Admittedly, selecting an estimand based on the available data counters the ICH E9 (R1) “estimand thinking process”, where the estimand is selected first, and data collection is performed to align with the estimand.Reference Mütze, Bell and Englert 58 , Reference Fletcher, Hefting and Wright 59 This is a by-product of the evidence synthesis context, where meta-analyses are often performed based on secondary data of previously conducted studies. A notable exception is the planned “integrated analysis” of pivotal RCTs, typically pre-specified prior to data collection, for which the cross-trial alignment of estimands and the definition of a target meta-analytical estimand might be less challenging.Reference Hedman, Kordzakhia, Li and Nyström 60
4 Case study: Semaglutide versus dulaglutide for T2D
To illustrate the use of the proposed framework in Section 3, we apply it to a network meta-analysis comparing the efficacy of semaglutide versus dulaglutide in patients with T2D, based on prior work by Lingvay et al., who performed an exhaustive network meta-analysis, including meta-regression and several scenario analyses.Reference Lingvay, Bauer, Baker-Knight, Lawson and Pratley 22 This network meta-analysis was selected because T2D is the therapeutic area including some of the first RCTs with information on estimands in trial publications, protocols, and statistical analysis plans.
4.1 Background and evidence base
Long-acting glucagon-like peptide-1 receptor agonists (GLP-1 RAs), such as semaglutide and dulaglutide, are established treatment options for patients with T2D. Current treatment guidelines recommend the use of GLP-1 RAs as an effective treatment option for patients with T2D not adequately controlled on metformin alone.Reference Buse, Wexler and Tsapas 61 Semaglutide and dulaglutide, both available for once-weekly administration via subcutaneous injection, can provide improved glycemic control for T2D patients.Reference Leiter and Nauck 62 Semaglutide and dulaglutide were initially approved for lower-dose maintenance use: 0.5 mg and 1.0 mg for semaglutide and 0.75 mg and 1.5 mg for dulaglutide. A head-to-head comparison of both treatments, at these doses, was performed in the open-label, multinational, Phase 3b RCT, SUSTAIN 7 (ClinicalTrials.gov registration number NCT02648204), with semaglutide demonstrating superior glycemic control than dulaglutide at week 40 and a similar safety profile.Reference Pratley, Aroda and Lingvay 63
The results of SUSTAIN 7 and other trials suggest that higher GLP-1 RA doses can provide increased efficacy, with treatment intensification often required for the maintenance of glycemic control.Reference Fonseca 64 Consequently, higher maintenance doses have been investigated, with semaglutide 2.0 mg approved by regulators based on the results of the double-blind, multinational, Phase 3b RCT, SUSTAIN FORTE (NCT03989232)Reference Frías, Auerbach and Bajaj 65 ; and dulaglutide 3.0 mg and 4.5 mg approved by regulators based on the results of the double-blind, multinational, Phase 3 RCT, AWARD-11 (NCT03495102).Reference Frias, Bonora and Nevarez Ruiz 66 The absence of head-to-head data between the higher doses of semaglutide and dulaglutide motivated Lingvay et al. to perform an indirect treatment comparison between semaglutide 2.0 mg, dulaglutide 3.0 mg, and dulaglutide 4.5 mg, based on the results of SUSTAIN 7, SUSTAIN FORTE, and AWARD-11.Reference Lingvay, Bauer, Baker-Knight, Lawson and Pratley 22 Understanding how the higher-dose GLP-1 RAs compare is of interest to clinicians and payers.
Table 3 displays the PICO determining the scope of the research question. In this particular case, and contrary to Section 3.1, dosing regimens are included for the intervention and comparators because decision-makers are specifically interested in the impact of dosing regimen on clinical outcomes. Two additional comparators, semaglutide 1.0 mg and dulaglutide 1.5 mg, have been included to connect or “anchor” the evidence network, as recommended by evidence synthesis methodological guidance. 21 For convenience, we have defined semaglutide 2.0 mg as the intervention, but dulaglutide 4.5 mg could have also been defined as the intervention with semaglutide 2.0 mg as a comparator.
PICO defining the scope and broad clinical research question for the case study
Table 3 Long description
The table consists of two columns and four rows.
- Row 1 (Population): Subjects with Type 2 diabetes on a background treatment of metformin.
- Row 2 (Intervention): Semaglutide 2.0 mg Q W.
- Row 3 (Comparators): Dulaglutide 4.5 mg Q W, Dulaglutide 3.0 mg Q W, Dulaglutide 1.5 mg Q W, and Semaglutide 1.0 mg Q W.
- Row 4 (Outcomes): Change from baseline in H b A 1 c and change from baseline in body weight.
A note below the table explains that comparators in italics connect the network of evidence and defines H b A 1 c as hemoglobin A 1 C (glycated hemoglobin) and Q W as once a week.
Note: The comparators in italics have been included to connect the network of evidence. HbA1c: hemoglobin A1C (glycated hemoglobin); QW: once a week.
Figure 2 displays the evidence network. SUSTAIN FORTE compares semaglutide 2.0 mg versus semaglutide 1.0 mg; AWARD-11 compares dulaglutide 4.5 mg and 3.0 mg versus dulaglutide 1.5 mg; and SUSTAIN 7 compares semaglutide 1.0 mg versus dulaglutide 1.5 mg. The evidence network is connected by including SUSTAIN 7, which anchors the comparisons between the higher-dose GLP-1 RAs via the lower-dose semaglutide 1.0 mg and dulaglutide 1.5 mg.
Connected evidence network for the treatments in the PICO scope for the case study. The nodes represent different treatments, the edges connecting the nodes indicate the presence of a direct head-to-head comparison in an RCT, and the shaded region highlights a closed loop due to a trial contributing more than two treatment arms (AWARD-11). The sample size n corresponds to the number of subjects randomized to treatment.
Figure 2 Long description
The diagram consists of five red circular nodes of varying sizes connected by thick black lines representing R C T comparisons.
At the top, three nodes form a triangular closed loop shaded in light blue.
* The top-most node is Dulaglutide 3.0 mg (n=616).
* The left node of the triangle is Dulaglutide 4.5 mg (n=614).
* The right node of the triangle is Dulaglutide 1.5 mg (n=911).
All three edges of this triangle are labeled A W A R D-11.
Moving downward from the Dulaglutide 1.5 mg node, a vertical edge labeled SUSTAIN 7 connects to a node at the bottom right.
* This bottom-right node is Semaglutide 1.0 mg (n=781).
A horizontal edge labeled SUSTAIN FORTE connects the Semaglutide 1.0 mg node to the final node at the bottom left.
* This bottom-left node is Semaglutide 2.0 mg (n=480).
4.2 Estimands at the trial level
We now adopt the estimand framework to systematically identify misalignments between the treatment effects targeted by different RCTs, and attempt to mitigate any quantitative heterogeneity that may threaten the exchangeability of treatment effects across trials and the feasibility of a network meta-analysis. While this exercise may appear reminiscent of general effect modification assessment in risk-of-bias tools for evidence synthesis,Reference Lunny, Higgins and White 67 it enhances these approaches by accounting for estimand attributes, in particular for intercurrent event strategies.
Table 4 displays the trial-level estimands for the RCTs in the evidence base, derived from their publications, protocols, and statistical analysis plans.Reference Pratley, Aroda and Lingvay 63 , Reference Frías, Auerbach and Bajaj 65 , Reference Frias, Bonora and Nevarez Ruiz 66 , 68 – Reference Randomized 73 On an attribute-by-attribute basis:
-
• Population: The RCT populations are similar. All trial populations include T2D patients on a background treatment of (i.e., inadequately controlled with) metformin $\geq 1,500$
mg/d (or a maximal tolerated dose)
$\geq 90$
days or 3 months, with or without sulphonylurea treatment (not explicitly specified for AWARD-11 and SUSTAIN 7). Nevertheless, there are some minor differences in selection criteria, not apparent from the estimands’ population attributes. Namely, the baseline HbA1c inclusion criteria are 7.0%–10.5%, 8.0%–10.0%, and 7.5%–11.0% for SUSTAIN 7, SUSTAIN FORTE, and AWARD-11, respectively; AWARD-11 explicitly requires a baseline body mass index
$\geq 25$
kg/m2; and SUSTAIN FORTE and AWARD-11 explicitly require T2D duration of at least six months. In any case, Lingvay et al. (and ClinicalTrials.gov) report generally similar summary statistics for the baseline characteristics of enrolled patients in all RCTs.Reference Lingvay, Bauer, Baker-Knight, Lawson and Pratley
22
-
• Treatment: The dosing regimen and mode of administration of the common comparators, semaglutide 1.0 mg and dulaglutide 1.5 mg, are identical—subcutaneous QW—across the trials in which they are investigated. All treatments in all trials are taken alongside a background of metformin (as specified for the population attribute).
-
• Variable (endpoint): Change from baseline in HbA1c in percentage points is a primary endpoint for all trials, and change from baseline in body weight in kilograms is a key or confirmatory secondary endpoint for all trials. However, the endpoints are measured at week 40 for the SUSTAIN RCTs, and at week 36 for AWARD-11. This divergence is likely more problematic for body weight than for HbA1c, which is measured over a 12-week time frame.
-
• Population-level summary measure: For all trials, a mean difference between treatment conditions is reported.
-
• Intercurrent event strategies: Overall, two intercurrent events are identified across all trials: premature treatment discontinuation and initiation of anti-diabetic rescue medication. For each of the key primary and secondary trial objectives, all RCTs report estimates of two different estimands, one where both intercurrent events were handled by the treatment policy strategy and another where both were handled by a hypothetical strategy. The similarities and differences between these strategies might not be immediately apparent to readers of the clinical trial journal articles,Reference Pratley, Aroda and Lingvay 63 , Reference Frías, Auerbach and Bajaj 65 , Reference Frias, Bonora and Nevarez Ruiz 66 particularly to those unfamiliar with strategies for intercurrent events, as different terminology has been adopted across trials. We describe the intricacies of the intercurrent event strategies below.
Trial-level estimands, according to the ICH E9 (R1) estimand framework, of the RCTs in the evidence base for the case study
Table 4 Long description
The table consists of five columns: Trial, Estimand name, Strategy for initiation of rescue medication and treatment discontinuation, Statistical model, and Imputation approach and assumption.
* A W A R D 11 trial:
- Efficacy estimand uses a Hypothetical strategy, M M R M model, and M M R M: M A R within randomized treatment group imputation.
- Treatment regimen estimand uses a Treatment policy strategy, Retrieved dropout A N C O V A model, and M I: extended M A R within group defined by randomized treatment and treatment status at week 36.
* S U S T A I N 7 trial:
- De-jure estimand uses a Hypothetical strategy, M M R M model, and M M R M: M A R within randomized treatment group imputation.
- De-facto estimand uses a Treatment policy strategy, Retrieved dropout A N C O V A model, and M I: extended M A R within group defined by randomized treatment and treatment status at week 40.
* S U S T A I N F O R T E trial:
- Hypothetical trial product estimand uses a Hypothetical strategy, A N C O V A model, and M I: M A R within randomized treatment group imputation.
- Treatment policy estimand uses a Treatment policy strategy, Retrieved dropout A N C O V A model, and M I: extended M A R within group defined by randomized treatment and treatment status at week 40.
4.2.1 Intercurrent event strategies
Any similarities and differences between intercurrent event strategies must be made explicit to avoid mixing “apples and oranges” in network meta-analyses. The statistical analysis plan of SUSTAIN 7 features a “de-jure” and a “de-facto” estimand 69 ; that of SUSTAIN FORTE includes a “hypothetical” (denoted “trial product” in the corresponding publication)Reference Frías, Auerbach and Bajaj 65 and a “treatment policy” estimand 71 ; and that of AWARD-11 describes an “efficacy” and a “treatment regimen” estimand.Reference Randomized 73 The de-jure, hypothetical, and efficacy estimands were the primary estimands, except in the USA, where the Food and Drug Administration (FDA) had a preference for the de-facto, treatment policy, and treatment regimen estimands.
The differences in estimand names across trials are due to changes in terminology over time: “de-jure” and “de-facto” estimands refer to “efficacy” (effect if treatments are taken as directed in the protocol) and “effectiveness” (effect of treatment as actually taken), respectively, but now these terms are somewhat outdated. From the full estimand descriptions in the trial statistical analysis plans, 69 , 71 , Reference Randomized 73 we find that all RCTs contain: (1) a primary estimand, which characterizes the treatment effect if patients do not discontinue treatment prematurely or initiate anti-diabetic rescue medication, that is, a hypothetical strategy is adopted for these intercurrent events; and (2) a supplementary estimand, requested by the FDA, characterizing the treatment effect regardless of premature treatment discontinuation or initiation of anti-diabetic rescue medication, that is, a treatment policy strategy is adopted.
While the criteria for initiation of anti-diabetic rescue medication may have differed across trials, the rescue medication itself was similar. In the SUSTAIN RCTs, it was selected in accordance with the American Diabetes Association and the European Association for the Study of Diabetes guidelines,Reference Davies, D’Alessio and Fradkin 74 but excluding other GLP-1 RAs, dipeptidyl peptidase-4 (DPP-4) inhibitors and amylin analogs. 68 , 70 In AWARD-11, the rescue therapy was determined by the investigator, but GLP-1 RAs and DPP-4 inhibitors were also excluded. 72
It is worth noting that the intercurrent event strategies, while very similar, are not exactly identical across trials. The estimands of SUSTAIN FORTE adopt a treatment policy strategy for change in treatment dose. Conversely, SUSTAIN 7 and AWARD-11 do not explicitly consider change in treatment dose as an intercurrent event. While these two approaches may appear consistent, SUSTAIN FORTE listing change in treatment dose as an intercurrent event, addressed via treatment policy, means that every effort has been made to collect all outcome data after the occurrence of the intercurrent event.Reference Bell, Drury and Mütze 37 Because such intercurrent event was not defined for SUSTAIN-7 and AWARD-11, it is not possible to know how many subjects changed the treatment dose in these trials. The frequency of intercurrent events can change over settings and over time, and an overview of the intercurrent event distribution in the trials would be necessary to understand the impact of ignoring change in treatment dose.Reference Lanius, Glocker and Lösch 75
As illustrated in Table 5, different estimation methods are required for different intercurrent event strategies. Selected estimators of the hypothetical strategy set post-intercurrent event outcomes as missing, then impute them to recreate what the outcomes would have been had the intercurrent event not occurred. The imputation is either implicit (mixed model for repeated measures in SUSTAIN 7 and AWARD-11) or explicit (multiple imputation followed by analysis of covariance in SUSTAIN FORTE), relying on a strong “missing-at-random” (MAR) assumption, where missing outcomes depend only upon observed and modeled covariates as well as previous outcomes, within randomized treatment group.
Statistical estimation procedures for the trials included in the evidence base of the case study
Table 5 Long description
The table compares three clinical trials across six estimand attributes.
1. Trial Identifiers:
• A W A R D 11 (N C T 0 3 4 9 5 1 0 2) with Efficacy and Treatment regimen estimands.
• S U S T A I N 7 (N C T 0 2 6 4 8 2 0 4) with De-jure and De-facto estimands.
• S U S T A I N F O R T E (N C T 0 3 9 8 9 2 3 2) with Hypothetical and Treatment policy estimands.
2. Population:
• A W A R D 11: Subjects with Type 2 diabetes inadequately controlled with metformin.
• S U S T A I N 7: Subjects with Type 2 diabetes on a background treatment with metformin.
• S U S T A I N F O R T E: Subjects with Type 2 diabetes on a background treatment of metformin with or without sulphonylurea treatment.
3. Treatments (on top of metformin):
• A W A R D 11: Subcutaneous dulaglutide at 4.5 mg, 3.0 mg, and 1.5 mg Q W.
• S U S T A I N 7: Subcutaneous semaglutide at 1.0 mg and 0.5 mg Q W; Subcutaneous dulaglutide at 1.5 mg and 0.75 mg Q W.
• S U S T A I N F O R T E: Subcutaneous semaglutide at 2.0 mg and 1.0 mg Q W.
4. Variables (endpoints):
• A W A R D 11: Primary is change in H b A 1 c at week 36; Key secondary is change in body weight at week 36.
• S U S T A I N 7 and S U S T A I N F O R T E: Primary is change in H b A 1 c at week 40; Confirmatory secondary is change in body weight at week 40.
5. Population-level summary measure: All trials use the mean difference between treatment conditions.
6. Intercurrent event strategies:
• A W A R D 11 and S U S T A I N 7: Use Hypothetical or Treatment policy strategies for initiation of rescue medication and premature discontinuation.
• S U S T A I N F O R T E: Adds ‘Change in treatment dose’ to the Treatment policy strategy.
Note: ANCOVA: analysis of covariance; MAR: missing-at-random; MI: multiple imputation; MMRM: mixed model for repeated measures
Conversely, estimators of the treatment policy strategy use all observed outcomes and require collecting outcome data following the occurrence of intercurrent events. The estimation approach for the treatment policy strategy in all RCTs is retrieved dropout, which distinguishes between outcomes occurring before and after the intercurrent event, making use of observed post-intercurrent event data to impute missing post-intercurrent event outcomes.Reference Bell, Drury and Mütze 37 This allows for a more flexible “extended MAR” assumption, as described by Bell et al.,Reference Bell, Drury and Mütze 37 which assumes MAR within group defined by randomized treatment and treatment status (intercurrent event occurrence and timing).Reference Bell, Drury and Mütze 37 All RCTs implemented retrieved dropout by multiple imputation, conditional on whether outcomes occur pre- or post-intercurrent event, followed by analysis of covariance.
4.3 Estimands at the meta-analytical level
We now attempt to define estimands at the meta-analytical level, aiming to enhance the relevance of pooled treatment effect estimates with respect to a specific clinical research question, and to maximize their applicability to healthcare decision-making contexts. There is some cross-trial heterogeneity between the estimand attributes in the case study; particularly, in the time at which endpoints are measured. Nevertheless, other sources of heterogeneity have been mitigated, for example, by identifying consistent intercurrent event strategies across studies.
We construct two target meta-analytical estimands (Table 6): one which adopts a hypothetical strategy for premature treatment discontinuation or initiation of anti-diabetic rescue medication (the hypothetical meta-analytical estimand), consistent with the de-jure, hypothetical (trial product), and efficacy estimands from SUSTAIN 7, SUSTAIN FORTE, and AWARD-11, respectively; and another which adopts a treatment policy strategy for premature treatment discontinuation or initiation of anti-diabetic rescue medication (the treatment policy meta-analytical estimand), consistent with the de-facto, treatment policy, and treatment regimen estimands from SUSTAIN 7, SUSTAIN FORTE, and AWARD-11, respectively. Figure 3 illustrates the restriction of the original PICO scope using each target meta-analytical estimand.
Target meta-analytical estimands for the case study
Table 6 Long description
The table consists of three columns: Estimand name, Hypothetical, and Treatment policy.
- Population: Subjects with Type 2 diabetes on a background treatment of (i.e., inadequately controlled with) metformin.
- Treatments: Semaglutide 2.0 mg Q W, Dulaglutide 4.5 mg Q W, Dulaglutide 3.0 mg Q W, Dulaglutide 1.5 mg Q W, and Semaglutide 1.0 mg Q W.
- Variables (endpoints): Change from baseline to week 36 or week 40 in H b A 1 c (percentage points) and change from baseline to week 36 or week 40 in body weight (k g).
- Population-level summary measure: Mean difference between treatment conditions.
- Intercurrent event strategies: For the Hypothetical estimand, a hypothetical strategy is used for initiation of anti-diabetic rescue medication or premature treatment discontinuation. For the Treatment policy estimand, a treatment policy strategy is used for the same events.
Note: The treatments in italics were not of primary interest and were included to connect the evidence network
The evidence base, defined by the PICO scope, can be narrowed using each meta-analytical estimand to perform more targeted network meta-analyses that are relevant to specific clinical research questions.
Figure 3 Long description
At the top, the P I C O scope is defined by four bullet points.
* Population. subjects with Type 2 diabetes on a background treatment of metformin.
* Intervention. semaglutide 2.0 mg.
* Comparators. dulaglutide 4.5 mg, dulaglutide 3.0 mg, dulaglutide 1.5 mg, semaglutide 1.0 mg.
* Outcomes. change from baseline in H b A 1 c and body weight.
Below this text are two large rounded boxes representing meta-estimands.
The left box is labeled Hypothetical meta estimand. It contains three smaller boxes.
* Top left. A W A R D-11 efficacy.
* Top right. S U S T A I N 7 de-jure.
* Bottom center. S U S T A I N F O R T E hypothetical.
The right box is labeled Treatment policy meta estimand. It contains three smaller boxes.
* Top left. A W A R D-11 treatment regimen.
* Top right. S U S T A I N 7 de-facto.
* Bottom center. S U S T A I N F O R T E treatment policy.
4.3.1 Example network meta-analyses
We conduct two network meta-analyses, exploring the impact of targeting different meta-analytical estimands or intercurrent event strategies. One of the network meta-analyses adopts a hypothetical strategy for premature treatment discontinuation or initiation of anti-diabetic rescue medication. The other adopts a treatment policy strategy for the corresponding intercurrent events. T2D is a therapeutic area where confirmatory Phase 3 RCTs typically report results for both hypothetical and treatment policy estimands. This allows us to conduct the two network meta-analyses based on published aggregate-level results in the clinical trial publications,Reference Pratley, Aroda and Lingvay 63 , Reference Frías, Auerbach and Bajaj 65 , Reference Frias, Bonora and Nevarez Ruiz 66 without necessarily requiring subject-level data.
The network meta-analyses are performed based on trial-level mean differences and standard errors, using fixed effects models with inverse variance weighting,Reference Borenstein, Hedges, Higgins and Rothstein 76 assuming independence between trials and accounting for correlated treatment effects within multi-arm trials,Reference Rücker 77 as implemented in the R package netmeta.Reference Balduzzi, Rücker and Nikolakopoulou 78 Where standard errors were not reported for treatment effects (treatment regimen estimates for AWARD-11), they were approximated based on reported confidence intervals for mean change from baseline for each treatment arm.Reference Higgins, Thomas and Chandler 79 The results of the network meta-analyses for change from baseline in HbA1c, targeting the hypothetical and treatment policy estimands at the “meta” level, are in Figure 4. The corresponding results for change from baseline in body weight are in Figure 5.
Results of the network-meta analyses for change from baseline in HbA1c (%-points). MD: Mean difference; CI: Confidence interval.
Figure 4 Long description
The figure consists of two side-by-side forest plots labeled a and b. Both plots share a horizontal X-axis at the bottom ranging from negative 1 to 1, representing the Mean difference in change from baseline in H b A 1 c percentage points. A vertical line of no effect is positioned at 0.
Panel a, Hypothetical meta-analytical estimand, contains five comparison groups:
1. Semaglutide 2.0 m g vs other: All comparators (Semaglutide 1.0 m g, Dulaglutide 4.5 m g, 3.0 m g, 1.5 m g) show negative M D values ranging from negative 0.23 to negative 0.64, favoring Semaglutide 2.0 m g.
2. Semaglutide 1.0 m g vs other: Shows a positive M D of 0.23 against Semaglutide 2.0 m g and negative M D values against Dulaglutide dosages.
3. Dulaglutide 4.5 m g vs other: Shows positive M D values against Semaglutide dosages and negative M D values against lower Dulaglutide dosages.
4. Dulaglutide 3.0 m g vs other: Shows positive M D values against Semaglutide and Dulaglutide 4.5 m g, and a negative M D of negative 0.17 against Dulaglutide 1.5 m g.
5. Dulaglutide 1.5 m g vs other: All comparators show positive M D values ranging from 0.17 to 0.64.
Panel b, Treatment policy meta-analytical estimand, follows the same structure but with slightly attenuated effect sizes. For example, in the Semaglutide 2.0 m g vs other group, the M D values range from negative 0.18 to negative 0.52. In the Dulaglutide 1.5 m g vs other group, the M D values range from 0.10 to 0.52. Each data point includes a square representing the M D and horizontal whiskers representing the 95% C I.
Results of the network-meta analyses for change from baseline in body weight (kg). MD: Mean difference; CI: Confidence interval.
Figure 5 Long description
The forest plot consists of two panels: (a) Hypothetical meta-analytical estimand on the left and (b) Treatment policy meta-analytical estimand on the right. Both panels share the same X-axis ranging from minus 6 to 6, representing the Mean Difference in change from baseline in body weight in k g. A vertical line of no effect is at 0.
Each panel contains five comparison blocks:
1. Semaglutide 2.0 m g vs other: Shows negative M D for all comparators (Semaglutide 1.0 m g, Dulaglutide 4.5 m g, 3.0 m g, and 1.5 m g), with values ranging from minus 0.93 to minus 4.48 in panel a, and minus 0.77 to minus 3.82 in panel b.
2. Semaglutide 1.0 m g vs other: Shows a positive M D against Semaglutide 2.0 m g and negative M D against all Dulaglutide doses.
3. Dulaglutide 4.5 m g vs other: Shows positive M D against both Semaglutide doses and negative M D against lower Dulaglutide doses.
4. Dulaglutide 3.0 m g vs other: Shows positive M D against both Semaglutide doses and Dulaglutide 4.5 m g, but negative M D against Dulaglutide 1.5 m g.
5. Dulaglutide 1.5 m g vs other: Shows positive M D against all other treatments, with values reaching up to 4.48 in panel a and 3.82 in panel b.
Data points are represented by squares with horizontal error bars indicating the 95% C I. Numerical values for M D and C I are listed to the right of each plot.
For change from baseline in HbA1c, estimated mean differences for semaglutide 2.0 mg versus dulaglutide 3.0 mg and 4.5 mg are
$-0.47$
(95% confidence interval [CI]
$-0.70$
to
$-0.24$
) and
$-0.30$
(95% CI
$-0.53$
to
$-0.07$
) percentage points, respectively, for the hypothetical strategy; and
$-0.42$
(95% CI
$-0.67$
to
$-0.16$
) and
$-0.28$
(95% CI
$-0.54$
to
$-0.03$
) percentage points, respectively, for the treatment policy strategy. For change from baseline in body weight, estimated mean differences for semaglutide 2.0 mg versus dulaglutide 3.0 mg and 4.5 mg are
$-3.58$
(95% CI
$-4.76$
to
$-2.40$
) kg and
$-2.88$
(95% CI
$-4.06$
to
$-1.70$
) kg, respectively, for the hypothetical strategy; and
$-2.92$
(95% CI
$-4.15$
to
$-1.69$
) and
$-2.22$
(95% CI
$-3.47$
to
$-0.97$
) kg, respectively, for the treatment policy strategy.
At the RCT level, and as indicated by the results of the individual trials,Reference Pratley, Aroda and Lingvay 63 , Reference Frías, Auerbach and Bajaj 65 , Reference Frias, Bonora and Nevarez Ruiz 66 treatment effects under a treatment policy strategy are generally more attenuated than those under a hypothetical strategy. Because the effects of intercurrent events tend to be common across arms, subjects experiencing an intercurrent event tend to become more similar.Reference Ratitch, Bell and Mallinckrodt 80 In diabetes GLP-1 RA clinical trials, including measurements that are collected after treatment discontinuation makes the actual treatment received, on average, more alike between the experimental and the control arms.Reference Aroda, Rosenstock and Terauchi 81 The initiation of anti-diabetic rescue medication is, typically, only allowed if patients do not maintain acceptable glycemic control. Because patients are less likely to maintain adequate glycemic control under the less effective treatment, the initiation of rescue medication is expected to occur more commonly among participants receiving such treatment.Reference Aroda, Rosenstock and Terauchi 81 Hence, including measurements collected after the intercurrent event as part of a treatment policy strategy generally leads to a smaller difference between the treatment arms of a trial.Reference Aroda, Rosenstock and Terauchi 81
The magnitude of the hypothetical and treatment policy pooled estimates is similar for change from baseline in HbA1c at the meta-analytical level (Figure 4), with the treatment policy estimate slightly closer to the null. For change from baseline in body weight, the pull toward the null by the treatment policy strategy is more noticeable (Figure 5), more in line with what would be expected in an individual RCT.
4.3.2 Applicability of meta-analytical estimands
HTA concerns the evaluation of relative effectiveness (added treatment benefit in conditions encountered in “real-world” clinical practice), as opposed to clinical efficacy (added treatment benefit under ideal, controlled, circumstances).Reference Nordon, Karcher and Groenwold 82 The choice of target meta-analytical estimand, and its intercurrent event strategy, should account for external validity or applicability to the decision-making context; in the HTA case, whether treatment effects can be reasonably applied to patients in routine clinical practice.
If treatment discontinuation and the initiation of anti-diabetic rescue medication are occurring in real-world clinical practice, the treatment policy meta-analytical estimand is likely of greater interest to HTA decision-makers. Nevertheless, decision-makers must consider that treatment discontinuation rates and rescue medication can vary over time, settings, and jurisdictions. This may threaten the applicability of treatment policy results, which depends on each intercurrent event and its distribution being stable. There are also concerns about informative missingness if post-intercurrent event outcomes are not collected. For the RCTs in our case study, treatment discontinuation rates at week 36 or 40 were 7%–17% for the treatments in the scope, which is roughly consistent with the 20% discontinuation rate observed after one year for T2D populations using GLP-1 RAs in European real-world settings,Reference Palanca, Ampudia-Blasco and Calderón 83 , Reference Lassen, Johansen and Modin 84 but considerably lower than the 50% observed after one year in the United States and Israel.Reference Weiss, Carr and Pal 85 , Reference Kassem, Khalaila, Stein, Saliba and Zaina 86
The hypothetical meta-analytical estimand targets the treatment effect if patients adhere to treatment as intended, without treatment discontinuation or rescue medication.Reference Aroda, Rosenstock and Terauchi 81 It might still be of interest to HTA decision-makers if the reasons for intercurrent events are not related to clinical effectiveness (including safety), or if there is a clear manner or incentive to prevent the occurrence of intercurrent events in future practice. This is challenging for treatment discontinuation here. While some T2D patients in routine clinical practice discontinue subcutaneous GLP-1 RAs for cost- or preference-related reasons, for example, preferring oral medication over injections, many discontinue treatment due to inadequate glycemic control, nausea, or gastrointestinal events.Reference Sikirica, Martin, Wood, Leith, Piercy and Higgins 87
Ultimately, T2D patients do prematurely discontinue GLP-1 RAs in routine clinical practice, at rates generally higher than those observed in RCTs, and for efficacy or safety reasons that cannot be prevented trivially.Reference Lim, Pasternak, Eliasson and Ueda 88 As such, an estimand that envisages a hypothetical scenario where patients do not discontinue treatment likely has limited applicability to HTA decision-making. Such estimand would have been targeted at the meta-analytical level by analysts naïvely pooling the primary estimands of the RCTs. This is something that probably happens in current meta-analytical practice, where different intercurrent event strategies are typically ignored.Reference Metcalfe, Vuong and Yan 89
Interestingly, an European Medicines Agency guideline for the use of estimands in diabetes trials 90 recommends a treatment policy strategy for premature treatment discontinuation and the hypothetical strategy for initiation of anti-diabetic rescue medication.Footnote 2 Namely, the target estimand is the treatment effect if anti-diabetic rescue medication is not initiated, regardless of premature treatment discontinuation. Such estimand is relevant because patients under the standard-of-care in routine clinical practice may discontinue treatment, and while they may initiate anti-diabetic rescue therapy, this is often medication within the same drug class (not permitted in comparative GLP-1 RA trials).Reference Lim, Pasternak, Eliasson and Ueda 88 In our case study, targeting the “hybrid” estimand, both at the trial and at the meta-analytical level, would have required full access to subject-level data because the individual RCTs did not plan or report results for the hybrid intercurrent event strategy. We shall discuss the benefits brought by subject-level data availability in Section 5.
5 Discussion
5.1 Subject-level data availability
Estimands can be helpful to communicate limitations of the evidence base where subject-level data are unavailable. Our case study is in a therapeutic area, T2D, which stimulated the regulatory discussions influencing ICH E9 (R1), and where recent RCT publications typically report both a treatment policy and a hypothetical estimand for premature treatment discontinuation or initiation of anti-diabetic rescue medication. One is unlikely to encounter such standardized reporting and consistency in estimands across many other therapeutic areas, but this is likely to change over time, as we expect most disease-specific regulatory guidance to be updated with recommended estimands. The harmonization of estimands within therapeutic areas has been identified as a priority for future regulatory–HTA body collaborations and should allow for more reliable evidence syntheses. 91
In any case, it may be challenging in many cases to align all estimand attributes—particularly, intercurrent event strategies—without full access to subject-level data. For instance, Table 2 in Section 3.1 highlighted the primary estimands for a thyroid cancer RCT, which included eight and nine different intercurrent events for each of the two objectives. Aligning on every single trial-level intercurrent event strategy for this indication, especially under subject-level data limitations, will be very challenging. An informed discussion with clinicians will be required to determine which intercurrent events are more relevant, both in terms of frequency and clinical importance. In T2D, premature treatment discontinuation and the initiation of anti-diabetic rescue medication are well-established intercurrent events that should be accounted for, even in aggregate-level meta-analyses. In oncology trials, treatment switching is a major intercurrent event that should be considered.Reference Metcalfe, Remiro-Azócar and Vuong 57 , Reference Metcalfe, Vuong and Yan 89
Ades et al. outline how quantitative heterogeneity for “outcomes reported in different ways” can be addressed through multivariate normal random effects meta-analysis.Reference Ades, Welton, Dias, Phillippo and Caldwell 11 Similar analytical methodologiesReference Bujkiewicz, Jackson and Thompson 92 could be applied where trials report different sets of intercurrent strategies, for example, if a relevant intercurrent event strategy is reported in some trials but not in others, multivariate (network) meta-analyses could be used to borrow information from trials in which the relevant estimands are reported, thereby accounting for the totality of evidence.Reference Remiro-Azócar and Gorst-Rasmussen 12 Clinical trial reporting guidelines should continue to be updated to include information on intercurrent events, so there is an incentive to more easily study them in aggregate-level data meta-analyses. In any case, the conduct of analyses based on subject-level data, as opposed to aggregate-level data, should be encouraged.Reference Riley, Dias and Donegan 93 It will likely become more feasible in the future, as pharmaceutical companies are increasingly allowing access to subject-level data.Reference Modi, Kichenadasse and Hoffmann 94
In our case study, the use of subject-level data would have allowed for targeting the hybrid estimand recommended by the European Medicines Agency, described in Section 4.3.2, at the trial and at the meta-analytical level. It would have also allowed us to pursue greater alignment between the timing of outcome assessments in the SUSTAIN trials (week 40) and AWARD-11 (week 36). Nevertheless, even with full access to subject-level data, it can happen that the relevant data are not collected for all trials. For instance, in our case study, no outcome data were collected at week 36 for the SUSTAIN trials and at week 40 for AWARD-11, as there were no visits during those weeks.
Lingvay et al. attempted to derive results for week 36 from the SUSTAIN 7 and SUSTAIN FORTE subject-level data via linear interpolation between week 28 and week 40, which provided results consistent with those of the original analysis.Reference Lingvay, Bauer, Baker-Knight, Lawson and Pratley 22 Alternatively, one could have used subject-level data from AWARD-11 to derive results by linear interpolation between the week 36 and week 52 visits, or attempted to model time-course relationships by synthesizing outcomes at multiple visits.Reference Pedder, Dias, Bennetts, Boucher and Welton 95 , Reference Pedder, Boucher, Dias, Bennetts and Welton 96 Notably, our case study has focused on efficacy endpoints but additional difficulties may arise with safety endpoints due to cross-trial differences in the assessment of adverse events.Reference Unkel, Amiri and Benda 97
It is worth noting that adopting common intercurrent event strategies across trials may not always be feasible, even with full access to subject-level data. For instance, alignment with respect to a treatment policy strategy would require all trials to collect outcome data after the occurrence of the intercurrent event. Nevertheless, trials designed with other intercurrent event handling strategies in mind may not collect post-intercurrent event outcome data. A trial prioritizing a composite variable strategy—which incorporates the occurrence of the intercurrent event into the endpoint definition—might not have even reported whether it is the primary outcome of interest or the intercurrent event that has occurred for a given subject.
Consider the design of an RCT for which a treatment policy strategy is not of primary interest to the sponsor, but in a therapeutic area where other trials report treatment policy estimands. We encourage trialists to collect post-intercurrent event outcome data to align with a treatment policy strategy for meta-analysis, with the caveat that the meaning and frequency of intercurrent events might change over time. The value of this alignment likely depends on the decision-making relevance of the meta-analytical treatment policy estimand, the type and frequency of intercurrent events being stable across trials, and the feasibility of aligning other intercurrent event strategies instead (if these are relevant to the research question). We also encourage trialists to report information on the intercurrent event distribution to assess the comparability of treatment policy estimands across trials.
5.2 Meta-analysis and meta-regression methodology
Another limitation of our case study is the slight differences in the trials’ inclusion criteria, which imply that the RCT populations are not identical, even though the high-level population attribute in the estimand definitions is virtually the same. It is worth noting that, even if the trials’ selection criteria were to be the same, differences in the distribution of baseline characteristics can arise between “analysis sets”, due to non-random sampling.
Lingvay et al. report summary statistics for the baseline characteristics of enrolled patients in SUSTAIN 7, SUSTAIN FORTE, and AWARD-11.Reference Lingvay, Bauer, Baker-Knight, Lawson and Pratley 22 These are generally similar, but some minor differences exist.Reference Lingvay, Bauer, Baker-Knight, Lawson and Pratley 22 Based on subgroup analyses of SUSTAIN 7, there is little evidence to suggest that differences are “effect-modifying”, with the caveat that the subgroup analyses are post-hoc and underpowered to test for effect modification.Reference Pratley, Aroda and Catarig 98 Again, an informed discussion with clinicians is generally required to determine the importance of cross-trial differences in the distribution of baseline characteristics.
Cross-trial differences in the distribution of baseline characteristics can give rise to treatment effect heterogeneity, which compromises the validity of (network) meta-analyses.Reference Phillippo, Ades, Dias, Palmer, Abrams and Welton 56 , Reference Phillippo, Ades, Dias, Palmer, Abrams and Welton 99 Traditional statistical tests used to quantify heterogeneity, such as Cochran’s Q test and the I-squared statistic, do not explain heterogeneity (and are often underpowered). We propose the use of the estimand framework to explicitly describe the different sources that may be inducing such heterogeneity. When unexplained heterogeneity remains, random effects models have traditionally been used to pool treatment effects.Reference DerSimonian and Kacker 100 , Reference Stijnen, Hamza and Özdemir 101 However, these could not be applied to our case study, which included only one trial per pairwise treatment comparison.
Both fixed and random effects approaches have been criticized for not explaining heterogeneity and not explicitly producing directly interpretable estimates in any well-defined target population.Reference Sobel, Madigan and Wang 14 , Reference Dahabreh, Petito, Robertson, Hernán and Steingrimsson 102 Conventional methods weigh trial-specific estimates according to their precision rather than their relevance to a specific target population and produce ambiguous estimates with limited external validity.Reference Manski 13 , Reference Hasegawa, Claggett, Tian, Solomon, Pfeffer and Wei 103 This criticism can be mitigated by narrowly defining the population in the scope,Reference Ades, Welton, Dias, Phillippo and Caldwell 11 , Reference Remiro-Azócar and Gorst-Rasmussen 12 such that quantitative heterogeneity between trials is reduced and common effects are more plausible.
If there is no quantitative heterogeneity across trials, both fixed and random effects analyses target a single underlying estimand, common to all studies, for each pairwise treatment comparison. Conversely, under the presence of cross-trial quantitative heterogeneity, for example, due to differences in patient populations, endpoint timing, etc., there are different trial-specific estimands. Both fixed and random effects models average over the residual heterogeneity, thereby targeting an average effect in the overall population (“super-population”) of included trials compatible with the broader overarching PICO or meta-analytical estimand.
The difference between fixed and random effects analyses is that the latter can quantify the residual heterogeneity and incorporate it through a predictive distribution.Reference Higgins, Thompson and Spiegelhalter 104 Arguably, this implies that treatment effects cannot be interpreted without a description of the full distribution of estimated random effects, not just its center (e.g., average) value.Reference Hasegawa, Claggett, Tian, Solomon, Pfeffer and Wei 103 Previous research has implied that fixed effects analyses consider a single estimand at the individual trial and meta-analytical levels, for each pairwise treatment comparison.Reference Metcalfe, Remiro-Azócar and Vuong 57 Nevertheless, this interpretation only holds if there is no heterogeneity across trials, such that the collection of trial-specific estimands is homogeneous, and in which case the interpretation also holds for a random effects analysis.
Cross-trial discrepancies in populations could have been resolved through innovative meta-regression techniques for covariate adjustment, such as multilevel network meta-regression,Reference Lingvay, Bauer, Baker-Knight, Lawson and Pratley 22 , Reference Phillippo, Dias and Ades 105 – Reference Phillippo, Dias and Ades 107 which would require access to some subject-level data or the use of federated systems.Reference Han, Hou, Cho, Duan and Cai 108 Meta-regression approaches seem relevant to address the meta-analytical challenges discussed in Section 1, which motivated our proposal for the use of estimands, specifically with respect to the population attribute. They can explain heterogeneity and explicitly mitigate it, by adjusting for cross-trial differences in the distribution of effect-modifying baseline characteristics. We note that—even if heterogeneity is minimized—the “population” attribute of all trial-specific estimands may not match the target population for decision-making. Meta-regression approaches also hold promise because they can explicitly produce treatment effects in any target population of substantive interest, not necessarily that of one of the trials in the evidence base, thereby enhancing the external validity that is necessary for HTA decision-making.
5.3 Population-level summary measure
Our exposition has focused on one of the estimand attributes that is not considered by PICO, the intercurrent event strategy, and on its alignment across trials. There is another estimand attribute not included in PICO, the population-level summary measure. Conflicts in the marginal and conditional summary measures that are pooled in meta-analyses are a cause for concern, particularly for analyses based on aggregate-level data, as trials may report different summary measuresReference Remiro-Azócar, Phillippo and Welton 109 and naïvely combining them may produce bias.Reference Remiro-Azócar, Heath and Baio 49 For non-collapsible summary measures, such as (log) odds ratios and (log) hazard ratios, marginal and conditional estimands—and conditional estimands conditioned on different covariate sets—do not coincide, even in the absence of effect modification by the covariates.Reference Daniel, Zhang and Farewell 110 In the presence of effect modification, this is equally problematic for some measures, those that are not directly collapsible.Reference Remiro-Azócar 111
Fortunately, the population-level summary measure for all RCTs in our case study is a mean difference from change from baseline, which is directly collapsible.Reference Remiro-Azócar 111 Table 5 illustrates the trial-level estimation methods that were employed for each RCT. The mixed model for repeated measures (MMRM) estimates the difference in “least squares” means between treatment groups, which corresponds to targeting a conditional treatment effect at the covariate means. The modeling approaches based on analysis of covariance (ANCOVA) also target a conditional treatment effect, assumed homogeneous across covariate values and therefore equivalent to the conditional treatment effect at the covariate means.Reference Remiro-Azócar, Phillippo and Welton 109 Again, full access to subject-level data would be beneficial to address summary measure incompatibilities, by allowing for the estimation of any desired marginal or conditional summary measure for each of the trials, or by permitting for a subject-level (network) meta-regression.
Aligning the target summary measure at the meta-analytical level with a relevant research question for decision-making is also crucial. Marginal and conditional summary measures correspond to different decision questions, with discussion over which is more suitable for healthcare decision-making,Reference Phillippo, Remiro-Azócar and Heath 112 and some arguing that marginal summary measures are of greater interest for population-level decision-making.Reference Remiro-Azócar, Phillippo and Welton 109 In the context of our case study, conditional mean differences at the covariate means and marginal mean differences are expected to coincide due to direct collapsibility, as long as conditional effects are constant across covariate values (assumed by the trial-level analytical approaches) or vary linearly with the covariates.Reference Remiro-Azócar, Phillippo and Welton 109 , Reference Remiro-Azócar 111
6 Concluding remarks
Pairwise and network meta-analyses are crucial to inform healthcare policy, but threatened by quantitative heterogeneity between trials and by the poor applicability of pooled estimates to decision-making contexts. We have applied the estimand framework to a case study of semaglutide versus dulaglutide for T2D in order to explicitly identify potential misalignments in the evidence base, mitigate key sources of quantitative heterogeneity between trials, and to enhance the applicability of meta-analytical results with respect to a specific research question. We performed a network meta-analysis pooling exclusively hypothetical estimates, and a network meta-analysis pooling exclusively treatment policy estimates. This implied the specification of a hypothetical and a treatment policy target estimand at the meta-analytical level, allowing us to be transparent and explicit about the source of heterogeneity—the intercurrent event strategy—driving the difference between research questions.
There were more notable divergences in results between intercurrent event strategies for one endpoint (change from baseline in body weight) than for the other (change from baseline in HbA1c). It is worth noting that we exclusively pooled estimates using the same intercurrent event strategy, such that the treatment policy attenuation of trial-level estimates may have averaged out across the network. In practice, the naïve pooling of estimates targeting different intercurrent event strategies, especially where the intercurrent event occurs frequently, may shift efficacy and (cost-)effectiveness results in arbitrary directions while producing estimates of ambiguous interpretability.Reference Metcalfe, Remiro-Azócar and Vuong 57
Concerns have been raised about the applicability of estimands to evidence synthesis and (network) meta-analyses, by statisticians working in clinical development and the pre-market authorization (regulatory) setting,Reference Hedman, Kordzakhia, Li and Nyström 60 , Reference Russek-Cohen 113 but also by HTA bodies.Reference Morga, Latimer, Scott, Hawkins, Schlichting and Wang 31 We recognize that many cautions remain, particularly regarding operational complexity and practical feasibility, for example, in relation to HTA scoping or the analysis of secondary data of previously conducted trials, typically with limited input on primary data collection. As such, our proposal for the use of estimands is pragmatic in nature and does not necessarily align with the “estimand thinking process” of ICH E9 (R1). Our proposal also departs from the original idea by Remiro-Azócar of replacing the PICO framework with ‘PICOSI”, incorporating all estimand attributes.Reference Remiro-Azócar 54
Our proposal could be integrated within a broader roadmap of evidence synthesis activities in HTA as follows (Figure 6): (1) conduct a systematic literature review, with protocol and search terms based on the planned assessment scope PICO; (2) assess cross-trial heterogeneity using the trial-level estimands; (3) with stakeholder input, identify a suitable target meta-analytical estimand for healthcare decision-making; (4) perform a feasibility assessment for the quantitative evidence synthesis, based on Steps 2 and 3; and (5) conduct pairwise or network meta-analyses, perhaps with sensitivity analyses allowing for varying degrees of misalignment between the trial-level estimands.
Proposed roadmap for HTA evidence synthesis planning and execution, incorporating PICOs and estimands.
Figure 6 Long description
A large gray downward-pointing arrow on the left serves as a timeline anchor, with colored horizontal bars corresponding to each step.
Step 1 (Orange): Conduct a systematic literature review based on the planned assessment scope P I C O.
Step 2 (Green): Perform assessment of cross-trial heterogeneity using trial-level estimands.
Step 3 (Purple): Identify "ideal" target meta-analytical estimand for healthcare decision-making.
Step 4 (Light Blue): Assess feasibility for quantitative evidence synthesis by incorporating Steps 2 and 3.
Step 5 (Red): Conduct main pairwise or network meta-analysis along with sensitivity analyses.
Our proposal maintains the use of PICOs for qualitative assessments, while using estimands to assess the feasibility and relevance of quantitative analyses. Existing templates for meta-analytical feasibility assessmentsReference Cope, Zhang, Saletan, Smiechowski, Jansen and Schmid 114 may require updating to reflect ICH E9 (R1), estimands and intercurrent event strategies. Importantly, trial-level estimands must be consistently recorded so that those performing systematic literature reviews can retrieve the relevant information from publications. Widespread adoption of estimands will expose the need for a more structured approach to the reporting and dissemination of trial results, for example, via meta-data on treatment effect estimates and their associated estimand attributes on ClinicalTrials.gov.
We acknowledge that the relative imprecision of PICOs is attractive to describe the broad “totality” of evidence in qualitative evidence syntheses, and to define the original scope of HTA decision problems. Nevertheless, this should not detract from the value of estimands in quantitative evidence syntheses. We advocate for the explicit integration of estimands into the planning of pairwise and network meta-analyses. One can formulate a PICO for scoping, then define different meta-analytical estimands. These can be used to restrict the original evidence base and perform more targeted (network) meta-analyses, precisely addressing the specific research question that might be of interest to healthcare decision-makers. The meta-analytical estimand can serve as a basis for cross-disciplinary discussion, to strengthen communication between different stakeholders about what a (network) meta-analysis aims to demonstrate, and to facilitate the interpretation of its results.Reference Lanius, Glocker and Lösch 75
Moreover, the estimand framework provides a shared language to align trial design with both regulatory and HTA objectives. 91 Insofar, HTA considerations have played a limited role in estimand selection, focused primarily on the perspective of key regulatory agencies. While these perspectives are important, HTA bodies may require different estimands for decision-making and views may differ between HTA agencies based on the context in which they make their decisions. Confirmatory clinical trials should be planned with both regulatory- and HTA-relevant estimands in mind.Reference Schiel 115 Beyond traditional clinical trials, real-world data (RWD) are increasingly used to support HTA decision-making. Constructing appropriate estimands for the analysis of RWD—and incorporating them into meta-analyses—will involve specific considerations beyond those encountered for traditional clinical trials and remains a research priority.
We encourage discussion and consensus-building among stakeholders within different therapeutic areas about which intercurrent events are most relevant, and which strategies are appropriate to handle them in specific decision-making contexts. In T2D, premature treatment discontinuation and the initiation of anti-diabetic rescue medication have been identified as key intercurrent events by regulatory stakeholders, with the European Medicines Agency publishing a therapeutic guideline (CPMP/EWP/1080/00 Rev.2) recommending a treatment policy strategy for handling premature treatment discontinuation and a hypothetical strategy for initiation of anti-diabetic rescue medication. 90 Other therapeutic areas are yet to build consensus about intercurrent event strategies. In oncology, treatment switching is considered to be a major intercurrent event,Reference Morga, Latimer, Scott, Hawkins, Schlichting and Wang 31 but current reporting practices to address treatment switching are inadequate, both in trial-level analyses and in meta-analyses.Reference Metcalfe, Vuong and Yan 89 Similar to the development of core patient characteristic sets and core outcome sets, defining important prognostic factors and outcomes to be measured and reported among specific therapeutic and disease settings, we would encourage stakeholder initiatives to define core intercurrent events and intercurrent event strategies.
Finally, we have focused on the role of intercurrent events and intercurrent event strategies in contextualizing the external validity of individual trial and meta-analytical results. Nevertheless, accounting for other estimand attributes is no less important when judging the transferability of results to any given clinical setting. For instance, target populations in RCTs are typically less heterogeneous and more narrowly defined than actual patient populations in routine clinical practice, partly to maximize statistical efficiency and power for hypothesis testing.Reference Greenhouse, Kaizar, Kelleher, Seltman and Gardner 116 The external validity of RCTs can also be undermined if their treatment implementations differ from those in routine clinical practice, for example, due to stringent prohibitions on the use of non-trial treatments, or if their endpoints have limited clinical relevance.Reference Rothwell 117 There remains a need to incentivize the use of estimands that prioritize decision-making relevance and can provide the external validity that is necessary for actionable healthcare decision-making.
Acknowledgements
This article results from the work of the EIWG sub-team in HTA and RWE. We gratefully acknowledge the contributions of current and former members Amel Besseghir, Arthur Allignol, Barbara Rosettani, Frank Kleinjung and, especially, Anders Gorst-Rasmussen, who was instrumental in conceptualizing the article. Finally, we thank the Evidence Synthesis Scientific Working Group and the Estimands Scientific Working Group at Novo Nordisk for their very useful feedback.
Author contributions
A.R.-A, P.P., J.J.H.P., and A.M. originally conceptualized the ideas behind the manuscript. A.R.-A, E.B., A.G., L.L.N.H., K.R.R., and T.V. performed the initial investigations. A.R.-A developed the methodology, with critical input from D.M.P. and H.L. A.R.-A and R.B. curated the data and performed the formal analysis for the example case study, with critical input from H.L. A.R.-A wrote the programming code and produced the visualizations. A.R.-A, P.P., L.L.N.H., and A.M. administered the project, and A.R.-A, P.P., L.L.N.H., K.R.R., and T.V. supervised the project. A.R.-A wrote an initial draft of the manuscript, with critical input from P.P., E.B., and A.M., and the manuscript was revised and edited by all authors. All authors approved the final version of the article.
Competing interest statement
A.R.-A is an employee of Novo Nordisk Pharma. P.P. is an employee of Novo Nordisk Pharma and a minor shareholder in Eli Lilly. E.B. is an employee of Merck Healthcare KGaA and has minor stock ownership in Merck. A.G. is an employee of Daiichi Sankyo Italia. K.R.R., D.M.P., and J.J.H.P. declare no competing interests for this work. L.L.N.H., H.L., and R.B. are employees of Novo Nordisk A/S and hold shares in Novo Nordisk. T.V. is an employee of Sanofi-Aventis Deutschland GmbH and a minor shareholder in Bayer AG. A.M. is an employee of Astellas Pharma Inc.
Data Availability Statement
The network meta-analyses are based on aggregate-level data, available in the SUSTAIN 7,Reference Pratley, Aroda and Lingvay 63 SUSTAIN FORTE,Reference Frías, Auerbach and Bajaj 65 and AWARD-11Reference Frias, Bonora and Nevarez Ruiz 66 publications. Code implementing the network meta-analyses is available at https://github.com/antonio-ra/sema-estimands-NMA and is archived on Zenodo at https://doi.org/10.5281/zenodo.20745699.
Funding statement
D.M.P. is supported by the UK Medical Research Council (Grant Nos. MR/R025223/1 and MR/W016648/1).
Open access
