Background
A health technology assessment (HTA) is the systematic evaluation of the properties and effects of a health technology, addressing the direct and intended effects of this technology, as well as its indirect and unintended consequences, and aimed mainly at informing decision making regarding health technologies (1). This information can then be used to advise governments on the value and feasibility of public reimbursement for a specific drug or intervention in publicly funded health care systems. This process has been dominated by clinical and economic evidence in the past, but many HTA bodies have advocated for and are adopting approaches to ensure patient involvement (PI) as well (Reference Abelson, Wagner, DeJean, Boseveld, Gauvin and Bean2–Reference Messina and Grainger4).
It is generally agreed that PI, defined herein as the involvement of patients or those who represent them through consultation, direct involvement in HTA advisory committees, and patient experience submissions, is important to the HTA process (Reference Rozmovits, Mai, Chambers and Chan5;Reference Abelson, Forest, Eyles, Smith, Martin and Gauvin6). Consultation is defined herein as a dialogue between HTA committees and patients resulting in the passing of information regarding the relevant health technology onto the committee to help inform their opinion (7). Direct involvement is defined as patients taking part in deliberations and decision-making regarding a health intervention on HTA committees. Patient experience submissions are written testimonials provided by patients to HTA committees containing information regarding the effect the treatment had on their lives, whether positive or negative. Patient experience submissions are considered to be patient input, defined as “information provided by patients, their caregivers and patient groups to any deliberative process” (Reference Facey, Single, Facey, Hansen and Single8); since this provides a route for patients to communicate their experience with HTA officials we believe that it should constitute as involvement. Through these personal experiences, patients can provide valuable information that may be lacking in the literature and/or relevant clinical trial results (Reference Rozmovits, Mai, Chambers and Chan5). For example, while a clinical trial may report efficacy and safety of a new cancer drug, it is important to consider whether the outcomes are meaningful to patients and whether its side-effect profile is perceived to be acceptable by individuals affected by the disease. Further, many policy makers agree that there is an ethical obligation to involve patients in the evaluation of a drug or intervention that may affect them as the “end-users” (Reference Esmail, Moore and Rein9). For this reason, PI has been incorporated into HTA processes and studied in many jurisdictions around the world including Canada, the European Union, the United Kingdom, and Australia (Reference Abelson, Wagner, DeJean, Boseveld, Gauvin and Bean2;Reference Abelson, Forest, Eyles, Smith, Martin and Gauvin6;Reference Facey, Single, Facey, Hansen and Single8–Reference Lopes, Carter and Street13).
While several agencies incorporate patient input into the HTA process, the methods of PI vary, ranging from scoping topics, to interpreting evidence, and even drafting recommendations (Reference Weeks, Polisena, Scott, AP, Staniszewska and Facey14). Despite the varying approaches to PI in HTA, there is a lack of literature to compare and evaluate the success of any individual PI strategy in terms of its impact on the assessment.
Due to the paucity of literature regarding the evaluation of PI impact in HTA, we have undertaken a scoping review to identify evaluative tools that are currently utilized by HTA agencies. In particular, we focus on those used to evaluate the impact of PI on HTA recommendations, which has been consistently recognized in the literature as constituting a meaningful form of impact (Reference Abelson, Bombard, Gauvin, Simeonov and Boesveld15–Reference Dipankui, Gagnon, Desmartis, Légaré, Piron and Gagnon17).
In this way, we hope to determine any gaps in the literature regarding PI evaluation and contribute to the development of novel evaluative methods.
Methods
The protocol used for this scoping review was developed using the framework proposed by the Joanna Briggs Institute (JBI) (Reference Peters, Godfrey, Khalil, McInerney, Parker and Soares18). This methodology draws from the framework put forward by Arksey and O'Malley (Reference Arksey and O'Malley19), as well as Levac et al. (Reference Levac, Colquhoun and O'Brien20). We used the refined JBI framework, which consists of the following general approach: (1) Identify the research question, (2) Identify relevant studies while balancing feasibility with the breadth and comprehensiveness of the scoping process, (3) Select studies using an iterative team approach to study selection and data extraction, (4) Summarize results numerically and qualitatively, (5) Report results such that implications of the findings for policy, practice and/or research are highlighted, and (6) Consult stakeholders.
As described by Arksey and O'Malley (Reference Arksey and O'Malley19), a scoping review involves a comprehensive search of the literature, which is akin to a systemic review for a broadly defined (as opposed to highly focused) research question. Hence, we undertook a scoping review using an established protocol to ensure that the compiling and analysis of studies was both unbiased and highly comprehensive (Reference Arksey and O'Malley19).
Inclusion and Exclusion Criteria
Articles from any year were included if they met the following criteria: (1) written in English, (2) about the evaluation or assessment of the impact of the involvement (including consultation, direct involvement in HTA advisory committees, and patient experience submissions) of patients (including patient groups, organizations, patients' families, and patients themselves) in the HTA process. Articles were excluded if they were opinion pieces or were not relevant to the topic of this study. Articles related to the involvement of patients solely through dissemination of results to them were also excluded because this does not comply with our pre-established definition of involvement. The term “involvement” is recognized as a wide spectrum of possible activities, ranging from receiving patient experience submissions to directly involving them on the assessment committee. In this paper, involvement is defined as direct inclusion of patients on an HTA advisory committee, consultation of patients during a HTA, and use of patient experience submissions on the drug under review.
Additionally, articles that focused on the involvement of the public as opposed to patients were excluded. In the context of this paper, the term “patient” is defined as either a direct end-user of a health technology, or an advocate of the patient that acts for his/her benefit including caregivers and family members.
Sources of Data and Search Strategy
An expert librarian comprehensively searched the following databases: MEDLINE, EMBASE, the Cochrane Central Register of Controlled Trials (CCRCT), and the Cochrane Database of Systematic Reviews (CDSR) on 15 August 2017 (Supplement 1). An updated search of the previously stated databases as well as the Cumulative Index to Nursing and Allied Health Literature (CINAHL), PsycINFO, Scopus, Social Sciences Abstracts, and Business Source Premier was conducted on 29 November 2017 (Supplement 2). A third literature search using the same search terms and databases as the second was conducted on 7 June 2019. We also searched for grey literature by searching the websites of 26 organizations related to the conduct, practice or research of HTA (Supplement 3) with the following search terms in various combinations: “health technology”, “assessment”, “appraisal”, “patient”, “involvement”, “engagement”, and “participation”, and “evaluation”. In addition, Google.com was searched with the same search terms used in the HTA website searches. The reference sections of included articles were also reviewed for any additional potentially relevant articles.
Study Selection: Screening
The titles and abstracts of the citations that were identified through database searching were reviewed for relevance to the topic by two team members (MQ and RM) independently using the same selection requirements detailed in the inclusion criteria section of this report. After screening was completed, the inter-rater agreement kappa statistic for selection of studies was calculated and found to be 0.829, which lies above the threshold of 0.75 that is generally considered excellent agreement (Reference Fleiss, Levin and Paik21).
Data Characterization
Articles that were still considered relevant after screening were procured and read in full by a previously mentioned team member (RM). The following characteristics were extracted from the articles: title, year of publication, aim of study, summary of findings, and methods used to evaluate PI impact in HTA.
Synthesis
Data was synthesized by qualitatively analyzing and describing the aim, methods, and results of each of the records selected for further review based on their content. Percentages were used to describe numerical data based on the proportion of records that were sorted into the records' publication year, location, and topic.
Consultation
In accordance with the Arksey and O'Malley (Reference Arksey and O'Malley19) methodology, a copy of this scoping review was sent to key stakeholders who are highly knowledgeable regarding PI in HTA. Our study team worked closely with these stakeholders [Canadian Agency for Drugs and Technologies in Health (CADTH) pan – Canadian Oncology Drug Review (pCODR) program and the Canadian Cancer Action Network (CCAN)] throughout the study. Feedback was received through discussions with these stakeholders, and their suggested insights were incorporated into this study.
Results
Literature Search
In a search conducted on 15 August 2017, 220 potentially relevant citations were identified and on 29 November 2017 an updated search was conducted resulting in 1,033 citations. The literature search was further updated on 7 June 2019, resulting in 215 citations. Once duplicates were removed from these searches, a total of 1,248 unique citations remained. After initial screening, 37 citations were selected for data analysis. Among them, three were excluded from further analysis (one was not about PI in HTA and two were opinion pieces). In addition, a search for grey literature resulted in 22 potentially relevant articles and 7 of them were selected for further analysis. Hence, a total of 44 articles were ultimately selected for data analysis. The flow of this process is outlined in Figure 1.
PRISMA flowchart of the record selection process.
Many of the citations initially found were excluded from this paper (96%) after screening their titles and abstracts. This is due in part to a comprehensive set of keywords used in the search to ensure a broad search and thereby limit the risk of missing relevant articles. For example, searching “health technology assessment” resulted in a number of citations that related to HTA, without any reference to PI. This search also identified two opinion pieces, which were excluded. One paper was excluded on the basis of relevance because the involvement of patient organizations in a health care decision was mentioned but was not elaborated upon (Reference Mavris and Le Cam22).
Study Characteristics
The included studies were conducted in Europe (4) and Canada (2) and all were published from 2013 onwards. The characteristics of the selected records are summarized in Table 1, and the methods used to evaluate PI in each record are detailed in Table 2.
Characteristics of Included Records
Methods Used to Evaluate PI Impact in Included Records
CADTH, Canadian Agency for Drugs and Technologies in Health; HTA, Health Technology Assessment; NICE, National Institute for Health and Care Excellence; EPF, European Patients' Forum; PI, Patient Involvement.
How is the Impact of PI in HTA Evaluated??
Understanding the methods used to evaluate PI impact, and the reasons for selecting a particular methodology, will allow researchers to design future studies regarding the impact of PI on HTA. The methodology used within studies that were included in our scoping review is outlined below.
The methods used by researchers to evaluate the impact of PI on HTA included review of HTA reports, qualitative interviews, and quantitative analyses. For example, Berglas et al. (Reference Berglas, Jutai, MacKean and Weeks23) reviewed 30 reports completed by the Common Drug Review arm of CADTH to determine whether PI had any impact on the final report or recommendation. Staley et al. (Reference Staley and Doherty25) explored the influence of PI on the recommendations made by appraisal committee members at the National Institute for Health and Care Excellence (NICE); in this study, the researchers conducted semi-structured interviews with committee members, which were then thematically analyzed. Dipankui et al. (Reference Dipankui, Gagnon, Desmartis, Légaré, Piron and Gagnon17) interviewed personnel in Quebec (including patient groups) that had been involved in a HTA and reviewed the final HTA report; the focus of this study was to obtain insight about how the patient perspective influenced the content of the report and/or final recommendations.
In contrast, both Hamilton et al. (Reference Hamilton, Griffiths and Hanman24) and Chang et al. (Reference Chang, Versoza, Jaksa and Ho26) assessed PI impact as a purely quantitative variable. In both of these studies, a sample of final recommendations from HTA agencies was analyzed to determine if PI had occurred, and if the health technology under review had received a positive funding decision (Reference Hamilton, Griffiths and Hanman24;Reference Chang, Versoza, Jaksa and Ho26). This numerical comparison was then used to determine whether PI had an impact on the HTA process.
Finally, the European Patients Forum (EPF) surveyed patient groups, HTA agencies, as well as healthcare decision-makers to conduct a broad needs assessment and determine ways to improve PI (27). Sections of the survey asked these three groups about the perceived impact of PI on HTA in a broad sense, ranging from how PI impacted the cost of a HTA to its ability to improve committee members' understanding of the technology under review.
What are the Impacts of PI on HTA?
Among studies that aimed to evaluate the influence of PI on HTA recommendations, it was found that PI provided insights into the technology under review that was not otherwise available. This increased the reviewers' understanding of the technology and allowed for a more informed final decision to be made (Reference Dipankui, Gagnon, Desmartis, Légaré, Piron and Gagnon17;Reference Berglas, Jutai, MacKean and Weeks23;Reference Staley and Doherty25). The studies that utilized quantitative comparisons found that there was no significant association between the presence versus absence of PI and a positive funding recommendation (Reference Hamilton, Griffiths and Hanman24;Reference Chang, Versoza, Jaksa and Ho26). Providing a summary of the responses to the general impact assessment undertaken by the EPF is beyond the scope of this study. However, it is notable that while HTA agencies and decision-makers cited a large impact of PI on the HTA process, patient groups themselves questioned whether their contributions were meaningful (27).
Discussion
We conducted a comprehensive search of databases that resulted in six citations related to evaluation of PI impact in HTA. These six records were all published after 2012; while this is a small number over an 8-year time period, an increased awareness about the importance of evaluating PI in the HTA process is apparent. The differences in methodology used to evaluate PI, ranging from questionnaires to qualitative interviews and the review of completed HTA reports, is likely related to the unique objective of each study.
It is arguable that the impact of PI on the final HTA recommendation is the most important, but among all evaluative methods in this scoping review, the quantitative association between PI involvement with re-imbursement recommendation or approval was the least rigorous. As the respective authors noted, many variables that affect the final decision of an HTA committee regarding the approval of a health technology (Reference Hamilton, Griffiths and Hanman24), such as its efficacy and cost, were unaccounted for. We believe that qualitative interviews of patient groups, experts, and HTA officials who are involved in the assessment of a health technology is a more effective method for evaluating the impact of PI on HTA (Reference Dipankui, Gagnon, Desmartis, Légaré, Piron and Gagnon17;Reference Staley and Doherty25). Even though qualitative interviews are expensive and often time consuming to conduct, they may be very informative and allow for full opinions of stakeholders to be expressed. The use of questionnaires like the 2013 EPF survey is less costly and can be completed quickly, but the depth of information obtained is consequently sacrificed (27).
Given advantages and disadvantages of any single approach to PI evaluation, more than one method may be required to tackle this complex area of research in HTA. In particular, the approach to evaluation must be selected in accordance with how the researcher defines “impact” in their own studies. While we have focused our definition to the impact of PI on HTA recommendations and reports, other researchers have used narrower, and broader definitions. Chang et al. and Hamilton et al. both considered impact exclusively as the ability for the PI to result in a positive funding decision, rather than a negative funding decision (24;26). This straightforward definition allowed for the researchers to easily measure the impact of PI using a purely quantitative methodology. Specifically, the numerical comparison of positive to negative funding decisions was used in both of these studies. In contrast, Staley et al. selected a broader definition of impact, defining it as the degree to which PI affected the decision-making process (Reference Staley and Doherty25). This definition required more in-depth and qualitative methodology to evaluate the impact of PI. Therefore, the methodology used to evaluate the degree of PI impact must be tailored to the specific definition of “impact” in that particular study.
A further direction for research would be to investigate so-called mixed methods approaches that “combine elements from qualitative and quantitative research approaches for the broad purposes of breadth and depth of understanding and corroboration” (Reference Johnson, Onwuegbuzie and Turner28). This methodology has proven to be effective in a variety of health care settings including HTA. For example, mixed method approaches have been used to investigate the perceptions of health care providers on PI using a combination of interviews, observation, and a survey (Reference Joensson and Baker29). Additionally, the impact of the HTA programme of the National Health Service in the UK was assessed using a mixed methods approach of literature searches, surveys, and investigation of case studies (Reference Hanney, Buxton, Green, Coulson and Raftery30). By using a combination of approaches, the strengths of each of them can be capitalized upon, but their respective weaknesses must still be recognized.
We believe that the combination of surveys and interviews would be effective in improving PI as it can provide a wide range of information directly from both patients and HTA organizations. Although surveys alone (as utilized in the EPF study) may be more time- and cost-effective, limitations of survey-based approaches must be considered, including low response rates, bias of opinions toward individuals/groups that choose to respond, as well as inability to comprehensively probe insights of participants. Future research into the impact of PI should ideally account for variables other than PI (e.g. cost and efficacy of the proposed technology) that may affect funding decisions. This is quite challenging due to significant variability in the types of technologies being assessed. Recently, Skegdel et al. analyzed the funding decisions of pCODR to determine which aspect of a drug under review most strongly influenced their final recommendation (Reference Skedgel, Wranik and Hu31). This was accomplished via an initial qualitative analysis of publicly available reports on the recommendations made by pCODR. Attributes that contributed to funding decisions were compiled and analyzed via logistic regression to determine statistically significant attributes that contributed to final funding recommendations (Reference Skedgel, Wranik and Hu31). A similar quantitative methodology could be employed in future research to determine the relative impact of different methods of PI on other measures of success, such as the amount of PI and patient satisfaction; this is because the impact of PI on funding recommendations is insufficient to wholly judge the valuable influence of PI in the HTA process. However, quantitative methods are unable to capture the level and form of PI utilized in the HTA. Since some approaches can be more effective than others, the impact of PI may be underestimated if the type of PI is not considered.
An additional metric that has not yet been considered relates to how meaningful PI is in the context of HTA. This was recently investigated by Rozmovits et al. (Reference Rozmovits, Mai, Chambers and Chan5) through 24 qualitative interviews about the content of patient group submissions with members of the pCODR drug review process. Researchers found that meaningful PI occurred when the information presented in submissions could not be found anywhere else, thereby positively contributing to deliberations. Conversely, it was found that emotionally charged pleas and a lack of transparency about how information was gathered detracted from the meaningfulness of a submission (Reference Rozmovits, Mai, Chambers and Chan5). The editorial that accompanied this publication stated that only a “holistic and multidisciplinary approach” to PI in HTA can result in gaining meaning from it (Reference Abelson32). Hence, we believe that future frameworks for evaluation of PI should consider quality and meaningfulness, rather than its mere presence and method of elicitation.
To our knowledge, this is the first scoping review of its kind, though evaluation of PI in HTA is a growing field of study with new contributions constantly arising (Reference Gagnon, Dipankui, DeJean, Facey, Hansen and Single33). Given the advantages and limitations on any individual evaluative approach, we propose that mixed methods may be ideal to obtain a full spectrum of data regarding PI in HTA. We anticipate that the results of this review will be useful to researchers with an interest in HTA and other health-related fields that involve PI.
Limitations
A primary limitation of this study is that comprehensive searches of the grey literature may be limited despite authors' best efforts due to intrinsic difficulties associated with accessing such data. However, every action was taken to ensure that the maximum amount of grey literature was found. Further, only records in English were considered.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462320000239.
Acknowledgments
Mahin Iqbal Qureshi supported the development of our search strategy and with screening the records that were found.
Financial Support
This study was supported by the Canadian Agency for Drugs and Technologies in Health (CADTH) pan – Canadian Oncology Drug Review (pCODR), and the Canadian Cancer Action Network (CCAN). This study was funded and supported by the Canadian Centre for Applied Research in Cancer Control (ARCC), which is funded by the by the Canadian Cancer Society Research Institute grant #2015-703549.
