As the need for health technology assessments (HTA) increases with either higher health care budget constraints or a greater threshold of evidence for new therapies, the number of agencies involved in such assessments is also growing (Reference Banta5). In 2005, Poland established an HTA agency—Agencja Oceny Technologii Medycznych (AHTAPoL), which quickly became an important stakeholder in the decision-making process of authorization and introduction of health technologies (1). With AHTAPol being a relatively new institution, a comparison with a more established counterpart could offer insights into AHTAPol's appraisal process. Such a comparison also provides an opportunity to elicit recommendations for HTA methodological guideline changes, a document through which desired elements of HTA methodology are communicated. In cases where HTA process depends on data provided by manufactures, HTA methodological guidelines are an essential communication tool (Reference Hill, Mitchell and Henry10;Reference Taylor, Hutton and Culyer21). Even though there are compelling reasons why local HTA methodological guidelines must exhibit some variations, it has been suggested that certain methodological aspects are non-jurisdiction specific and should be approached in similar manner (Reference Sculpher and Drummond20).
The main objective of this study was to provide insight into HTA process in Poland and elicit recommendations for future development of Polish HTA methodological guidelines. It was assumed that there is a direct relationship between HTA methodological guidelines, the HTA submissions, and consequently the quality of HTA recommendations. HTA methodological guidelines were recognized as an important educational tool for manufactures in jurisdictions with limited experience in HTA methodology (such as Poland).
Scottish Medical Consortium (SMC) was chosen as the relevant comparison institution. The AHTAPoL frequently refers to the Scottish recommendations as a benchmark for their guidance (1). SMC serves as a point of reference for other jurisdictions as well (14). Prior comparisons of SMC with other jurisdictions provided additional rationale for choosing it as the benchmark for a Polish agency (Reference Cairns8;Reference Karia, Gani, Perard and Cann12;Reference Watts22). Finally, both Polish and Scottish institutions are modeled after the National Institute for Clinical Excellence and Health's (NICE) and there are instances where both agencies review NICE's assessments of relevant therapies before making their own decision (2;3;16;17).
The HTA process is similarly organized in both jurisdictions. It is based mainly on information provided by the manufacturer. Both SMC and AHTAPoL require HTA submission to consist of clinical and economic evaluation. Economic evaluation includes cost-effectiveness and budget-impact analysis in Poland and cost-effectiveness analysis only in Scotland. The HTA is separated from the reimbursement decision-making process in both countries (Figure 1).
Health technology assessment (HTA) process in Poland and Scotland.
The remainder of this study is structured as follows. First, the approach to the comparison of HTA methodological guidelines and recommendations are presented. The result section follows. The last part of the study highlights key findings and provides recommendations for future improvement of HTA methodological guidelines in Poland.
METHODS
The comparative analysis between Polish and Scottish agencies included two parts:
I. A comparison of HTA methodological guidelines
II. A comparison of HTA recommendations with the following approach:
1. HTA recommendations concerning drug technologies issued by AHTAPol in the period January 1 to December 31 2008 were identified (“Polish set”) (1).
2. Among HTA recommendations published by SMC online until 21 May 2009, guidance regarding the same drug submitted for the same indication as the Polish set were extracted (“Scottish set”) (18).
3. HTA recommendations in both sets were classified following Raftery's approach (Reference Raftery15). In the result of preliminary review of HTA recommendations published in studied jurisdictions, an extended classification was further developed to meet the needs of local decision-making process.
4. If HTA agency does not recommend financing the drug from public funds, HTA recommendation was labeled as negative guidance. Otherwise it was labeled as a positive guidance.
5. Positive recommendations were further divided into guidance with major, minor, and without restrictions. Several restrictions could be imposed simultaneously.
A positive recommendation was classified as “with major restrictions” if at least one of the following was recommended: (a) use only as second or subsequent line treatment, (b) use only if intolerant to other treatment, (c) continue only if response, (d) improve CE results, (e) resubmission required after certain time, (f) used restricted to specific subpopulation.
A positive recommendation was classified as “with minor restrictions” if none of major restriction criteria applied and at least one of the following was mentioned: (a) use at lower price, (b) use by specialist only.
6. Negative recommendations were classified as clinical or economic, depending on the reason. The non-clinical label was assigned only if there were no negative comments concerning clinical issues. Several restrictions could be imposed simultaneously.
Clinical reasons: (a) inappropriate comparator, (b) poor quality data, (c) poor efficacy, and (d) poor safety.
Economic reasons: (a) poor economic data, (b) unacceptable budget impact, and (c) insufficient justification of the treatment's cost in relation to its benefit
7. The comparison of HTA recommendations by jurisdiction was preformed. HTA recommendations were divided into three groups; positive recommendations with major restrictions, positive recommendations with minor restrictions and negative recommendations. The comparison of the total number of HTA recommendation in each group was preformed.
8. The comparison of HTA recommendations by drug technology was preformed. HTA recommendations were divided into two groups: those were different decisions were issued (i.e., SMC negative versus AHTAPoL positive and SMC positive versus AHTAPoL negative) and those were the same decisions were reached by both jurisdictions. The comparison of reasons for differences in recommendations between Poland and Scotland was preformed.
RESULTS
A Comparison of HTA Methodological Guidelines
The comparison of HTA methodological guidelines between SMC and AHTAPoL revealed some differences (4;19).
With regard to clinical evaluations, both agencies pay attention to different aspects of HTA submissions. While the Polish HTA agency instruct in a detailed manner how to conduct search strategy, the Scottish methodological guidelines provides a checklist with details to be presented for each clinical trial. In contrast to AHTAPoL, SMC focuses on comparative effectiveness and comparative safety and requires manufactures to provide arguments for applicability and relevance of studied results to routine clinical practice.
For economic evaluations, AHTAPoL allows using different types of analysis (with the exception of cost-benefit analysis which is not allowed) and does not state a preference for primary outcome measures. This differs from SMC, which considers cost utility analysis as the most appropriate approach and prefers health effects to be expressed in terms of quality-adjusted life years. It recognizes though that other outcome measures may be more appropriate in certain circumstances. If a non-QALY approach was adopted, manufactures must justify the choice.
In addition, AHTAPoL allows for a societal perspective if the implementation of a given technology might have impact on patients and other members of the society. SMC requires a payer's perspective; however, it allows the inclusion of a wider set of costs or outcomes in sensitivity analyses. SMC also recommends conducting relevant subgroup analyses. It emphasizes the importance of clinical rationale and plausible explanation of a differential effect. In justification of criteria for subgroup analysis, SMC highlights the need for an appropriate choice of clinical outcome and the analysis of parameter uncertainty.
There are not many differences in the approach adopted by both HTA agencies with regard to the estimation of budget impact. It should be mentioned though that while SMC concentrates more on calculation of a target population, AHTAPoL provides additional guidance regarding scenario analysis. The recommended time horizon is five and two years for SMC and AHTAPoL, respectively.
Review of HTA Recommendations
Sixty-eight HTA recommendations were issued by AHTAPoL in the period January 1 to December 31 2008 (accessed on May 21, 2009). There were three recommendations concerning non-drug technology and two appraisals concerning multiple drug technologies. Consequently, seventy-three drug technologies were available for the purpose of this analysis. Five recommendations required a resubmission. In those cases, only the most recent guidance was taken into account. In total, sixty-eight HTA outcomes were studied. The SMC evaluated forty-eight of sixty-eight drug technologies appraised by AHTAPoL; therefore, the overall sample size for this analysis was forty-eight.
Review of HTA Recommendations by Jurisdiction
Negative recommendations constituted 40 percent (19 of 48) and 23 percent (11 of 48) of all HTA recommendations issued by AHTAPol and SMC, respectively. While clinical reasons for rejection dominated in Poland, economic aspects were the most often stated reason for negative guidance in Scotland (Table 1).
Comparison of HTA Recommendations for Drug Technologies Issued by AHTAPoL and SMC During January 1-December 31 2008
In terms of clinical reasons for rejection, inappropriate comparator was mentioned in all cases in Scotland. In Poland, on the other hand, safety concerns were the most often stated clinical reason for rejection (Figure 2).
Reasons for negative HTA recommendation issued by AHTAPoL and SMC in studied period-(several reasons given simultaneously)
In cases of rejection on economic grounds, insufficient justification of the treatment's cost in relation to its benefit was the main reason for negative guidance in Scotland. At the same time poor economic data (poor quality or model unadjusted to Polish settings) and unacceptable budget impact were mentioned the most frequently in Poland (Figure 2).
There were twenty-nine and thirty-seven positive recommendations issued by AHTAPol and SMC, respectively. Major restrictions were common in both Poland and Scotland. SMC issued a greater number of positive guidance with no restrictions compared with AHTAPoL (Table 1).
For positive decisions but with major restrictions, use restricted to a specific subgroup was recommended most frequently by both agencies. It was mentioned in sixteen of twenty-two and twenty-two of twenty-five cases issued by AHTAPol and SMC respectively (Figure 3).
Restrictions for positive HTA recommendations issued by AHTAPoL and SMC in studied period -(several restrictions imposed simultaneously)
SMC frequently recommended use only if patient was refractory to other treatment; there were only a few decisions with such a restriction in Poland (Figure 3).
There were six positive recommendations with minor restrictions in Poland. The requirement to lower the price was mentioned in all of them. It was the most frequent restriction mentioned by the AHTAPoL among all positive HTA recommendations (nineteen cases) (Figure 3). SMC recommended as many as twelve drug technologies without any restrictions. There was only one such guidance in Poland (Table 1).
Review of HTA Recommendations by Drug Technology
There were thirty drug technologies that received similar guidance in both countries and eighteen with differences in HTA recommendations. Among the first group, fifteen drug technologies had the same guidance and another fifteen had differences in terms of type of restrictions for positive HTA recommendation or reason for negative HTA recommendations.
SMC Negative versus AHTAPoL Positive (Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc2012011)
SMC negatively appraised eleven drug technologies, of which five received a different appraisal by AHTAPoL. In those cases, AHTAPoL recommended use with major restrictions at four occasions and minor restriction in one case. SMC did not recommend these medicines both on economic (three cases) and clinical (two cases) grounds.
SMC Negative versus AHTAPoL Negative
Both agencies issued negative recommendation regarding six drug technologies. Clinical and economic reasons were given at the same time at both agencies once. At four occasions, SMC mentioned economic and AHTAPol clinical issues in their negative decisions respectively.
SMC Positive versus AHTAPoL Negative (Supplementary Table 2, which can be viewed online at www.journals.cambridge.org/thc2012011)
SMC positively appraised thirty-seven drug technologies, of which thirteen received negative guidance by AHTAPoL. Among those cases, AHTAPoL gave negative guidance based on clinical concerns (nine cases) and economic issues (four cases). SMC recommended major restrictions in ten cases and use without any restriction for three submissions.
SMC Positive versus AHTAPoL Positive
In the group of drug technologies with a positive recommendation, there were twelve medicines with major restrictions assigned by both agencies. In addition, one drug was granted a recommendation with no restrictions in both Poland and Scotland. AHTAPoL was less restrictive than SMC in three cases and more restricted than its Scottish counterpart in eight cases.
DISCUSSION
This study offers insight into the appraisal process in Poland by comparing the AHTAPoL recommendations to those of its Scottish counterpart, SMC. To our knowledge, there have been no attempts to analyze HTA guidance in Poland by benchmarking it against another jurisdiction. The use of Scottish HTA recommendations provides the ability to contrast the Polish HTA agency decision-making with that of the institution that is viewed as more established, yet still not as one that is setting the standards (i.e., NICE or IQWiG).
Based on the review of HTA recommendations, the approach of the Appraisal Body at AHTAPoL can be interpreted as more restrictive than SMC, with more negative recommendations issued during the study period in Poland. Guidance without additional restrictions was provided more often in Scotland. This variation in HTA outcomes to some extent can be attributed to differences in HTA methodological guidelines. Another reason for this difference could be that countries with higher willingness to pay will have better access to innovative treatments (Reference Barbieri, Drummond and Willke6). Although AHTAPoL has a lower CE threshold, it was impossible to assess whether difference between jurisdictions in this respect influenced the findings.
The Polish HTA agency focused more on clinical than economic issues. While comments about efficacy or safety were commonplace among reasons for negative recommendations in Poland, insufficient justification of treatment's cost in relation to benefits was the most often cited reason for rejection in Scotland. In its decisions AHTAPoL discussed budgetary consequences of the technology implementation more often than its cost-effectiveness. The need to lower price and concerns about budget impact were the most often stated reasons among economic arguments in Poland. Although SMC does not consider budget impact in its decisions, equivalence can be made between “unacceptable budget impact” and “insufficient justification of treatment's cost in relation to benefits,” as they both stem from similar concerns.
A review of decisions with different guidance revealed that SMC tended to recommend restricted use to specific sub-populations for several drug technologies negatively appraised by AHTAPoL. Limited methodological experience of Polish manufactures or specific local characteristics such as organization of the health care system in Poland could have contributed to this observation. Evaluation of the decisions made in the future could shed light if this difference persists or as experience increases, the use of specific subgroups in recommendations increases.
As any study, ours is not free of limitations. It is common that countries using the same assessment methods reach different conclusion regarding particular technology, as HTA bodies serve different populations (Reference Hutton11). Financing and organization of health care systems might also have a significant impact on the decision-makers’ behavior. Furthermore, the set of values, beliefs and preferences might be influenced by specific local factors as well. Because we studied only reasons for HTA recommendations, we have not taken these or other factors into account in the comparison.
The study was limited to HTA recommendations issued in Poland during the period January–December 2008. We identified only 48 cases where the same drug technology was submitted for the same indication by Scottish HTA agency; thus, the study should be regarded as an initial exploration. With the HTA process in Poland continuing to develop, similar research performed in the future might lead to different results.
Despite those limitations, several recommendations for future development of HTA methodological guidelines in Poland can be drawn.
The results indicate that further improvement of HTA guidelines with respect to a conduct of subgroup analysis may be required. The findings indicated that SMC recommended mainly restricted use within a specific subpopulation for drug technologies that received negative appraisal by AHTAPoL. This may suggest that SMC recognizes that there are specific groups of patients that may benefit from treatment. To ensure that the identification of an appropriate subgroup in routine practice is feasible, AHTAPoL ought to provide a specific guidance on both the methods for identification of relevant patient subgroups and methods for quantifying uncertainty in CEA (Reference Briggs7). Furthermore, manufactures of new medicines should follow modeling guidelines and incorporate population heterogeneity into their economic evaluations.
The second area of potential improvement of the Polish HTA methodological guidelines is related to clinical effectiveness issues. Poor safety and efficacy were only mentioned among negative HTA recommendations by AHTAPoL. To ensure better quality of presented clinical evidence, more transparent methodological guidelines should be produced. In case of SMC, manufactures are required to submit a detailed description of included studies and data sources used for comparative safety consideration. Although the impact of SMC regulation in this regard on the decision-making process is difficult to quantify, HTA methodological guidelines typically facilitate the preparation of higher-quality HTA submissions. Therefore, more explicit guidelines with respect to the presentation of clinical evidence are likely to enhance better quality of HTA dossiers.
Finally, to ensure better quality of HTA submissions in Poland, more transparent guidelines about local adaptation of global models to the Polish setting should be provided. The comparative review of Polish and Scottish HTA recommendations revealed that poor quality of economic evaluation and inappropriate model adjustment to the Polish settings were mentioned exclusively by AHTAPoL among reasons for negative HTA recommendations. There are several different local factors that might affect cost-effectiveness of a given health technology (e.g., demographic characteristics of patient population, epidemiology, health care organization) (Reference Manca and Willan13). According to ISPOR Task Force recommendations, economic analysis should be made relevant to a local context (Reference Drummond9). To ensure better applicability of delivered economic evaluation to the Polish setting, AHTAPoL could follow SMC's approach. In Scotland, manufactures are urged to describe factors which may influence the applicability of study results to patients in routine practice. SMC also discourages use of treatment pattern data from other countries (19).
CONCLUSIONS
We conclude that AHTAPol issues more negative HTA recommendations than SMC. Scottish HTA methodological guidelines provide more direction to guide manufacturers when preparing their submissions. Further development of the Polish HTA methodological guidelines might increase the likelihood of positive HTA recommendations being issued more frequently. The following areas need to receive further attention: methodology for subgroup analysis, presentation of clinical evidence as well as methodology for adaptation of economic models to local settings.
SUPPLEMENTARY MATERIAL
Supplementary Table 1
Supplementary Table 2 www.journals.cambridge.org/thc2012011
CONFLICT OF INTEREST
Katarzyna Kolasa has received funding from Bristol Myers Squibb. The other author reports having no potential conflicts of interest.
CONTACT INFORMATION
Katarzyna Kolasa, PhD, Adjunct Department of Pharmacoeconomics, Warsaw Medical University, Warsaw, Poland
Radek Wasiak, PhD, EU Director & Senior Research Scientist, Health Economics, United BioSource Corporation, London, United Kingdom
