Introduction
In England, National Institute for Health and Care Excellence (NICE) plays a role in appraising new medicines that have received regulatory approval, with a view to make recommendations regarding their cost-effective use in the National Health Service (NHS). In the single technology appraisal (STA) process, which covers a single medicine for a single indication, NICE develops a final scope (FS) after topic selection, which defines diseases, patients, and medicines covered by the appraisal. The submitting company discusses with NICE on how the decision problem will be addressed, and then, submits evidence. An evidence review group (ERG), which is independent of NICE and the submitting company, is in charge of evaluating the manufacturer's evidence submission to identify its strengths and weaknesses. Based on the evidence submitted by the manufacturer and the evaluation of the ERG, one or more appraisal committee meetings are held to develop a final appraisal determination (FAD) document. Finally, NICE publishes technology appraisal guidance (1).
According to the standard timeline, NICE spends approximately 40 weeks to make a recommendation, which is generally after marketing authorization (MA) (1;Reference Stoykova, Drummond, Barbieri and Kleijnen2). This slow approach to the publication of guidance was criticized, especially considering medicines for life-threatening diseases or diseases with low treatment satisfaction, because patients have limited access to new medicines during the time gap between MA and guidance publication (Reference Mayor3;Reference Haycox4). Following the criticism, NICE took several measures to accelerate the process of guidance publication. For example, it established the STA process that reduced the time for guidance publication (Reference Kaltenthaler, Tappenden, Booth and Akehurst5). Since 2015, more than 300 guidance documents have been published, and more than 80 percent of them were achieved via the STA process (6;Reference Kaltenthaler, Papaioannou, Boland and Dickson7). Furthermore, the recent appraisal process, which came into effect in 2018, established a policy that enabled cancer medicines to have shorter timelines for FS development (8).
Previous studies have shown that introducing the STA process contributed to speeding up the process of guidance publication (Reference Casson, Ruiz and Miners9–Reference Barham11). Moreover, one study suggested that cancer medicines prolonged the appraisal process post-FS development (Reference Casson, Ruiz and Miners9). To the best of our knowledge, there are no studies on the various appraised medicine characteristics that affect the speed of guidance publication by NICE. We believe that the time gap between MA and guidance publication, rather than the total time NICE spends during the appraisal processes, is noteworthy, for the following two reasons. Firstly, the importance of implementing more and faster NICE appraisals for new medicines and delivering a faster adoption of the most clinically and cost-effective medicines is emphasized in the 2019 voluntary scheme for branded medicines pricing and access (VPAS), which announced that all new medicines will undergo an appropriate NICE appraisal by April 2020 (12). Secondly, the time gap between MA and guidance publication better reflects the accessibility to new medicines; however, there is limited evidence regarding this time gap (Reference Ford, Waugh, Sharma, Sculpher and Walker10;Reference Barham11;Reference Connolly, O'Donnell, Lamrock, Tilson and Barry13). The aim of the present study was to investigate various appraisals or medicine characteristics that affect the time gap between MA and guidance publication by NICE, and explore the factors influencing this time gap by focusing on the cost-effectiveness analyses included in these appraisals.
Methods
Data Sources
We used publicly available documents from the Web sites developed by NICE, the European Medicines Agency (EMA), or the European Commission (EC). These included the FS, the first appraisal consultation document, or the FAD issued by NICE, the annual reports issued by the EMA, and the Union Register of medicinal products developed by the EC.
Inclusion and Exclusion Criteria
We considered all technology appraisals (TAs) designated as STAs and completed by July 2020 for inclusion. However, we excluded appraisals completed before August 2016, as they followed a different process from the recent appraisal process. We also excluded appraisals if they were among the following: (i) terminated appraisals, (ii) appraisals for medical devices, (iii) appraisals that reviewed previous appraisals, and (iv) appraisals that had been replaced by subsequent reviews. Additionally, we excluded appraisals if the MA had been granted before any scope documents were published by NICE because such cases follow a considerably different process as compared to the standard process (8).
Key Dates and Periods
From each TA report, we extracted the dates of the following: (i) FS publication, (ii) FAD publication, (iii) validation of MA application by the EMA, and (iv) MA. Based on the dates, we calculated the following time periods: (i) the total number of months between MA and FAD publication (MA to FAD), (ii) the total number of months between validation of MA application and FS publication (VAL to FS), and (iii) the total number of months between FS publication and FAD publication (FS to FAD). The relationship among these three time periods are summarized in Figure 1. We assumed that the MA to FAD period represented the period wherein patients had limited access to new medicines. After identifying medicine characteristics that were associated with longer or shorter MA to FAD period, we subsequently evaluated their influence on the VAL to FS and FS to FAD periods to explore the reason for this association, while focusing on the appraisal processes. We assumed that the VAL to FS period reflected how quickly NICE undertook topic selection and finalized the FS and that the FS to FAD period represented how long NICE spent time on discussions before FAD publication. VAL was used as an alternative indicator of the start date of the topic selection stage by NICE because the start date itself is not publicly available. We understand that it is possible to evaluate medicine characteristics affecting the length of the topic selection and scope development stage based on the period from VAL to FS.
Key periods associated with the evaluation processes in MA and HTA. FAD, final appraisal determination; FS, final scope; HTA, health technology assessment; MA, marketing authorization; VAL, validation of marketing authorization application.
Appraised Medicine Characteristics Associated with Key Periods
We identified the appraised medicine characteristics associated with key periods—MA to FAD, VAL to FS, and FS to FAD—based on literature review and considering the underlying appraisal processes. For example, it is logical that fewer appraisal experiences in a certain disease category could prolong the time required to provide the FS and FAD. Several studies have suggested that the appraisals of cancer medicines are complex and uncertain, given that they tend to be associated with a high incremental cost-effectiveness ratio (ICER) gap between manufacturers and ERGs. Thus, such appraisals could take longer than usual to be completed (Reference Sabry-Grant, Malottki and Diamantopoulos14;Reference Morrell, Wordsworth, Schuh, Middleton, Rees and Barker15). Therefore, we included the following variables: (i) year of appraisal completion, defined as the publication year of the FAD (2016–18, 2019–20); (ii) application type (initial application, extension application); (iii) the total number of previous appraisals in the same disease category; (iv) cancer medicines (no, yes); (v) orphan medicinal products (OMPs) designated by the EMA (no, yes); and (vi) accelerated assessment (AA) granted by the EMA (no, yes). Disease categories were based on the “conditions and diseases” classification formulated by NICE (16).
Factors Influencing the Period from FS to FAD
Once we had identified medicine characteristics that were associated with longer or shorter periods from MA to FAD, and their association with longer or shorter VAL to FS or FS to FAD periods, we used nine detailed variables regarding cost-effectiveness analyses to explore the reason why the identified characteristics were related to longer or shorter VAL to FS or FS to FAD periods. The variables regarding cost-effectiveness analyses were as follows: (i) medicine cost (10,000 pounds/month or less, more than 10,000 pounds/month); (ii) the total number of comparators specified in the FS (two or less, more than two); (iii) ICER gap between the manufacturers’ ICER bid in their initial evidence submission and the ERG's ICER described in its initial report, both of which were based on the list price of appraised medicines (20,000 pounds/quality-adjusted life-year [QALY] or less, more than 20,000 pounds/QALY); and (iv) innovative technology acknowledged in the TA (non-innovative, innovative). Furthermore, factors regarding clinical trials included in the cost-effectiveness analyses were as follows: (v) the number of subjects (500 or less, more than 500); (vi) time from MA validation to approval, which represents the speed of the regulatory approval process (300 days or less, more than 300 days); (vii) phase (others, phase 3); (viii) double-blinded randomized control trial (DBRCT) (no, yes); and (ix) comparators in the clinical trials (not specified in the FS as a comparator in the cost-effective analyses, specified in the FS).
Statistical Analysis
Multiple linear regression analysis was used to examine the association between the characteristics of appraised medicines and the MA to FAD period. In the analysis, we used Cook's distance (Di) to test for highly influential observations, which were defined as having a Di larger than .5, and conducted a sensitivity analysis by excluding these data. Subsequently, we performed the same analysis to examine the association between these characteristics and the VAL to FS and FS to FAD periods. In addition, linear regression analyses were used to identify associations between the factors in the cost-effectiveness analyses and the FS to FAD period. We conducted an univariable analysis to narrow down the variables to be used in the multivariable analysis. As there were a few preliminary findings of the association, baseline variables (p < .10) in the univariable analysis were included in the multivariable analysis. For each analysis, we calculated the unstandardized partial regression coefficient (B) and 95 percent confidence interval (CI). We chose a complete case analysis because the proportion of missing data was low. Variance inflation factors were calculated to assess multicollinearity between variables; factors greater than ten were considered to represent multicollinearity. All statistical analyses were conducted using StatsDirect ver. 3.3.3 (StatsDirect Ltd., Cheshire, UK). Values were considered statistically significant at p < .05.
Results
Overview of the Investigated STAs
One hundred and sixteen appraisals met the criteria for analysis, and an overview is presented in Table 1. Of the appraisals, 64 percent (74/116) were completed between 2016 and 2018, and 60 percent (69/116) were derived from the initial MA application. NICE had experienced less than ten appraisals in the same “conditions and diseases” for approximately 39 percent (45/116) of the appraisals and more than twenty appraisals for 30 percent (35/116). Furthermore, 62 percent (72/116) of the appraisals were cancer medicines and 22 percent (25/116) were OMPs; only 5 percent (6/116) were granted AA. Table 1 summarizes the factors influencing the FS to FAD period. In case of factors in the cost-effectiveness analyses, the monthly medicine cost was more than 10,000 pounds in 21 percent (24/116) of the appraised medicines. The total number of comparators specified in the FS was more than three in 58 percent (67/116) of appraisals. Moreover, the ICER gap between the manufacturer and ERG was more than 20,000 pounds/QALY in 40 percent (37/93) of appraisals, whereas only 14 percent (16/116) of medicines were referred to as innovative, and their values were not fully captured by their ICER. In terms of the factors regarding clinical trials included in the cost-effectiveness analyses, the total number of subjects was more than 500 in 52 percent (60/116) of the appraised medicines, the time from validation of MA application to approval was more than 300 days in 50 percent (53/106) of appraisals, 85 percent (99/116) were of phase 3 trials and 54 percent (63/116) were of DBRCTs, and the comparators of 43 percent (50/116) of the trials were specified in the FS as comparators of the cost-effectiveness analyses.
Summary of 116 single technology appraisals investigated
ERG, Evidence Review Group; FAD, Final Appraisal Determination Document; FS, Final Scope; ICER, Incremental Cost-Effectiveness Ratio; OMP, Orphan Medicinal Product; QALY, Quality-Adjusted Life-Year.
a Percentages may not total 100 because of rounding.
b Previous appraisal means the total number of previous ones in the same “condition and diseases” categories (Reference Morrell, Wordsworth, Schuh, Middleton, Rees and Barker15).
c Ninety-three or 106 appraisals were available because of some missing entries.
Association of the Appraised Medicine Characteristics with Key Periods
The median value of the MA to FAD period was 5.5 months (Figure 2). The periods from VAL to FS and FS to FAD among the appraised medicine characteristics are shown in Supplementary Figures 1 and 2, respectively. There were positive associations between OMPs and the MA to FAD period (B = 3.042, 95 percent CI = 1.100–4.984). This means that OMPs were associated with a 3.042-month increase in the MA to FAD period. Cancer medicines and extension applications were negatively associated with the VAL to FS period (B = −3.405, 95 percent CI = −6.343 to −.467 and B = −5.908, 95 percent CI = −8.564 to −3.252, respectively). There were positive associations between cancer medicines or OMPs and the FS to FAD period (B = 2.366, 95 percent CI = .464–4.268 and B = 2.833, 95 percent CI = .727–4.940, respectively) (Table 2). There was no multicollinearity between variables.
Box-and-whisker plot of the MA to FAD period among apprised medicine characteristics. The upper and lower whiskers are the upper or lower quartiles plus 1.5 times the interquartile distance. The horizontal lines that split the boxes in two represent median values, which are also expressed as the black diamonds on the boxes. The white and black circles denote outliers of 1.5 and 3 times the interquartile range, respectively. AA, accelerated assessment; FAD, final appraisal determination; MA, marketing authorization; OMP, orphan medicinal product.
Multivariable analysis of appraised medicine characteristics associated with key periods
CI, Confidence Interval; FAD, Final Appraisal Determination; FS, Final Scope; MA, Marketing Authorization; OMP, Orphan Medicinal Product; VAL, Validation of Marketing Authorization Application.
a 109 appraisals were available, because there were some missing entries in the European Medicines Agency's validation date. Among the 109 appraisals, the 67 were completed from 2016 to 2018, the 69 were initial application, the 68 were cancer medicines, the 25 were OMPs, and the 6 were granted accelerated assessment.
We identified TA429, which was in scope development for more than one year, as a highly influential observation (17). We conducted a sensitivity analysis excluding this observation and obtained a result consistent with the main findings (Supplementary Table 1).
Association of Factors in the Cost-Effectiveness Analysis with the FS to FAD Period
In the preceding analysis, we found a positive association between OMPs and the MA to FAD period and that OMPs were also associated with a longer FS to FAD period compared to non-OMPs, while there was no association with longer or shorter VAL to FS periods. Based on the results, in the subsequent analysis, we focused on the association of detailed factors regarding cost-effectiveness analysis with the length of the FS to FAD period to explore the reason why OMPs were related to a longer FS to FAD period. The univariable analysis revealed that the ICER gap between manufacturers and ERGs and the time from MA to approval were associated with an increase in the FS to FAD period. The FS to FAD period negatively associated with the following attributes: the total number of comparators specified in the FS, DBRCT, and clinical trials in which the comparators were specified in the FS. Among them, the ICER gap was not used in the multivariable analysis, because it had a large number of missing entries (Supplementary Figure 3). The multivariable analysis showed that independent factors associated with a shorter FS to FAD period were DBRCT and clinical trials in which the comparators were specified in the FS (B = −2.183, 95 percent CI = −3.733 to −.633 and B = −2.637, 95 percent CI = −4.267 to −1.006, respectively) (Table 3). There was no multicollinearity between variables. The multivariable analysis, including the ICER gap, is shown in Supplementary Table 2, in which similar trends to those observed in the main findings were obtained. For OMPs, the time between the validation of MA application and approval is shown in Supplementary Table 3 and their clinical trial designs are shown in Supplementary Table 4.
Univariable and multivariable analyses of factors influencing the FS to FAD periods
ERG, Evidence Review Group; FAD, Final Appraisal Determination; FS, Final Scope; ICER; Incremental Cost-Effectiveness Ratio; QALY, Quality-Adjusted Life-Year.
a ICER gap was not used in the multivariate analysis because it had large number of missing entries.
Discussion
In the present study, we assessed the appraised medicine characteristics that affect the speed of guidance publications by NICE. Among the 116 STAs, 25 had OMP designations. The OMPs were associated with a longer MA to FAD period than non-OMPs, and they had a positive association with the FS to FAD period. In terms of clinical trial-related factors included in each cost-effectiveness analysis, non-DBRCTs and comparators of trials not specified in the FS as the comparator in the cost-effectiveness analyses were associated with a prolonged FS to FAD period.
Earlier studies that assessed the association between the appraisal processes and the time to guidance publication by NICE showed that introducing the STA process or no appeals from manufacturers improved the speed of guidance publication (Reference Casson, Ruiz and Miners9;Reference Ford, Waugh, Sharma, Sculpher and Walker10). However, in these studies, little attention was paid to the appraised medicine characteristics that contributed to this. In addition, in these studies, only one of the periods was considered—the time to guidance publication by NICE or the time gap between MA and guidance publication. In contrast, in the present study, we included characteristics of both appraisal processes and appraised medicines. Considering a broad range of characteristics, we evaluated the NICE appraisal processes (represented by the VAL to FS and FS to FAD periods) along with the time gap between MA and guidance publication (represented by the MA to FAD period).
The present study showed a positive association between OMPs and the MA to FAD period. This may be explained by the association of longer FS to FAD period with OMPs than with non-OMPs, because there was no association between OMPs and other processes related to health technology assessment and MA, including the period between VAL and FS or between validation of MA application and approval. This longer FS to FAD period with OMPs would have partially been caused by their clinical trial designs, which were included in the cost-effectiveness analyses; 64 percent (16/25) of the trials with OMPs were non-DBRCTs, and 68 percent (17/25) used comparators not specified in the FS as a comparator in the analyses.
The prolonged period from FS to FAD in non-DBRCTs was probably due to the complicated evaluation of the added health benefits or potential biases. Of the non-DBRCTs, 26 percent (14/53) were nonrandomized and were single-arm trials, which made it difficult to estimate the additional effectiveness of the appraised medicines. In most cases, published data were referred to; however, manufacturers and ERGs were required to carefully evaluate the heterogeneity between study populations and generalizability of the results to patients in the NHS (Reference Hettle, Corbett, Hinde, Hodgson, Jones-Diette and Woolacott18–Reference Walton, Sharif, Simmonds, Claxton and Hodgson20).
In non-DBRCTs, biases derived from subjective outcome evaluations are inevitable because of their non-blinded setting. Utility data, which are essential for cost–utility analysis, are usually collected via questionnaires from patients, which would lead to a bias (Reference Kahan, Cro, Doré, Bratton, Rehal and Maskell21;Reference Cox, Wade, Peron, Dietz, Eastwood and Palmer22). To confirm their reliability, manufacturers and ERGs also should refer to other data sources, leading to a long FS to FAD period.
For the cost-effectiveness analyses, no information was available on comparators when comparators not specified in the FS were used in the clinical trials. In such cases, external references are necessary to evaluate the added health benefits of appraised medicines. For example, when only placebo-controlled trials are available, manufacturers usually conduct network meta-analyses, connecting the appraised medicines and comparators via placebo data (Reference Thorlund, Zafari, Druyts, Mills and Sadatsafavi23;Reference Rinciog, Watkins, Chang, Maher, LeReun and Esser24). In the appraisal of ustekinumab for the treatment of ulcerative colitis, the manufacturer extracted relevant information from more than ten trials to evaluate the added health benefits of ustekinumab with five comparators. In this appraisal, the FS to FAD period was 12 months, which is longer than the median value (10 months) of 116 STAs. These processes supposedly require time, because the NICE carefully discusses imbalances among the extracted trials; this may lead to a long FS to FAD period (Reference Ramaekers, Wolff, Pouwels, Oosterhoff, Van Giessen and Worthy25).
Both OMPs and cancer medicines were associated with a longer FS to FAD period than other medicines; this was consistent with a previous study finding, that is, cancer topics prolonged the time to guidance publication (Reference Casson, Ruiz and Miners9), whereas cancer medicines had no association with long MA to FAD periods. This was probably because the VAL to FS period was approximately 3.4 months shorter with cancer medicines than with non-cancer medicines. This can partially be explained by the new appraisal process, which came into effect in 2018, and showed that cancer medicines had a shorter topic selection stage, than non-cancer medicines (8). Unlike cancer medicines, OMPs are not considered to shorten the VAL to FS period.
This study had some limitations. We could not consider the discounts offered by manufacturers, because discount information was not publicly available. These discounts were assumed to improve the ICER of the appraised medicines, probably leading to a shortened FS to FAD period. We could not take into account whether budget impact tests were implemented in parallel with each appraisal. Commercial discussions between manufacturers and the NHS England after such tests will allow the NICE to plan potential changes to the timelines of appraisals, probably leading to a lengthened FS to FAD period. We did not deal with the quality of the cost-effectiveness analyses submitted by the manufacturers; this might affect the length of the FS to FAD period as well. It was impossible to quantify their quality, but we confirmed that almost all manufacturers conducted analyses using the submission template formulated by NICE. We could not take into consideration the scientific advice offered by NICE because the process is not disclosed. Such scientific advice would help manufacturers to understand the perspective of decision makers, which might shorten the STA process.
In summary, the findings of the present study suggest that OMPs are associated with a longer time between MA and guidance publication by NICE than non-OMPs; this may be attributed to the prolonged FS to FAD period. Limited access to new medicines for orphan diseases, along with low treatment satisfaction, negatively affects patients. To address this issue, increased efforts are needed to shorten the NICE appraisal process for OMPs, which will help the VPAS commitments to assess all medicines as rapidly as possible and achieve their fast adoption.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462321001677.
Acknowledgments
We thank Editage for English language editing.
Funding
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Ethical Approval
Ethical approval was not required as the authors did not collect any personal information but only used aggregate secondary data.
Authors’ Contributions
ST was the leading author who designed the study, gathered the data, conducted the statistical analyses and interpreted the data, and wrote the manuscript. MN participated in designing the study, interpreting the data, and reviewing the manuscript.
Availability of Data and Material
The data sets generated and/or analyzed during this study are not publicly available, but are available from the corresponding author upon reasonable request.
Conflicts of Interest
ST is an employee of Daiichi-Sankyo Co., Ltd. MN does not have any conflict of interest related to this study.
