The Canadian Agency for Drugs and Technologies in Health (CADTH, www.cadth.ca) is Canada's pan-Canadian health technology assessment (HTA) agency. CADTH assesses many kinds of technologies, including drugs, diagnostic tests, and medical, dental, and surgical devices and procedures to inform healthcare decision making. Healthcare decision making is distributed in Canada, so the responsibility for decisions regarding the delivery, organization, and allocation of health resources falls to the individual provinces and territories as well as to some federal departments. Hence, decision makers at health ministries, health authorities, and hospitals turn to CADTH for independent, objective assessments of a wide variety of health technologies.
Requests to assess and appraise complex nondrug technologies, such as interventions to reduce the spread of respiratory viruses, bariatric surgery, and point of care testing, have challenged members of CADTH's review teams and recommendation committees. It has been challenging for reviewers to understand how they can best approach research questions that are answered using different research methods, while at the same time ensuring the committee gets the results of a rigorous and integrated assessment. For committee members, the challenge has been how to amalgamate, in a meaningful and robust way, the variety of information made available to them for their deliberation.
In this article, we describe a first attempt by CADTH reviewers to apply some of the INTEGRATE-HTA guidance (from here on “the guidance”) (Reference Whalster, Brereton and Burns1) in the conduct of an HTA on dialysis for people with end-stage kidney disease. An initial opportunity to describe our experience with the guidance was presented during an INAHTA-sponsored webinar on INTEGRATE-HTA in September 2016 (2). This study is an expansion and elaboration of that material. We will describe the way CADTH reviewers drew on three chapters from the guidance, specifically those that inform the development and use of logic models (Reference Rowhwer, Booth and Pfadenhauer3), assessing ethical aspects (Reference Lysdahl, Mozygemba, Burns, Chilcott, Bronneke and Hofmann4), and assessing context and implementation (Reference Pfadenhauer, Rohwer and Burns5). For each section, we provide a very brief description of the guidance, followed by how it was used in this HTA. We then discuss some challenges we faced, which ultimately led us to not implement the guidance in full. We close the article with some reflections and suggestions as to how HTA organizations, including CADTH, might make better use of the guidance in future.
OPPORTUNITY TO USE THE INTEGRATE-HTA GUIDANCE
CADTH's Health Technology Expert Review Panel (HTERP) is an advisory body to CADTH that develops recommendations on the use of nondrug health technologies. The panel consists of six core members: a health economist, ethicist, public member, and three clinical experts. Up to five additional expert members may be invited to participate on the panel for specific projects.
HTERP uses a multi-criteria framework (6) to appraise technologies and formulate its recommendations. It consists of ten domains that define the information the committee requires from an HTA to inform their deliberations (Table 1).
CADTH Health Technology Expert Review Panel Deliberative Framework(6)
To date, CADTH's approach to the assessment of complex technologies follows what is referred to as a concurrent or parallel HTA process in contrast to the integrated process that INTEGRATE-HTA aims to facilitate. In the parallel process, CADTH research and knowledge mobilization staff address the need for information in the deliberative framework by developing distinct research questions for each domain and by having separate research teams develop a study protocol and carry out research to address each question. Regular team meetings held throughout the development of the HTA help to ensure reviewers on each team are kept apprised of emerging results, challenges, and the overall direction by reviewers working on other teams. Reports from each team are compiled into one overall HTA document and a single discussion and conclusion section attempts to summarize and synthesize the diverse information in relation to the policy question that motivated the HTA.
In early 2016, CADTH began an Optimal Use project (i.e. an HTA with recommendations from HTERP) on dialysis modalities for adults with end-stage kidney disease (ESKD) (7). This project sought to inform recommendations about the optimal use, including appropriate patient selection, of self-care or assisted home dialysis (hemodialysis [HD] or peritoneal dialysis [PD]) and self-care in-center HD strategies. Assessing and making recommendations about dialysis modalities for ESKD involves assessing technologies that are “complex,” in the sense intended by the guidance: there are a large number of interacting components influencing access to and delivery of the interventions (patient/provider conversations and decisions; behaviors of those delivering and receiving the intervention, including home support, training, and comfort with procedures); there are numerous and variable outcomes (quality of life, mortality, morbidities, social, and environmental consequences); and a degree of flexibility both between and among individual dialysis modalities (specific prescriptions, setting of care, assistance with administration) (Reference Whalster, Brereton and Burns1).
CADTH's approaches to the assessment of research questions related to clinical benefit and harms, cost-effectiveness, and patients’ experiences and perspectives are relatively well developed. In contrast, approaches to assess research questions in other domains of HTERP's multi-criteria framework are currently under development at CADTH. In particular, for the dialysis HTA, we did not have well developed methods for addressing questions related to implementation considerations and strategies, and we were likewise looking to refine our approach to the assessment of ethical considerations. Hence, CADTH's methods group saw the assessment of dialysis modalities as an opportunity to explore the extent to which the guidance could provide useful direction on how to approach, further develop, and integrate these aspects of the assessment.
USING THE INTEGRATE-HTA GUIDANCE
An important consideration in using the guidance is that the INTEGRATE-HTA model views “modifying factors,” for example patient characteristics or policy context, as connected to and influencing the “aspects” of HTA, which include effectiveness, socio-cultural, economic, ethical, and legal aspects. Through its parallel HTA process, CADTH has traditionally considered “modifying factors” as distinct information domains or “aspects” on their own, without formal consideration of how they may influence other aspects. As we will discuss below, this approach of developing sections of the HTA in parallel presented challenges in using the guidance in the way in which it is intended, and may signal a need to reconceptualise HTA.
We will not comment directly on the integration of stakeholder views, an important part of the guidance. Instead, we simply note here that CADTH involves stakeholders (HTERP members, customers, patients, clinical experts) in several ways during the assessment of nondrug technologies: by gathering information that is important to topic prioritization and scoping (for example, interviewing patients to gather information on relevant outcomes and context of use) and by seeking feedback on the HTA protocol, list of included studies, draft report, and draft recommendations. This feedback is used to further clarify the methods, findings and potential limitations or information gaps in the HTA.
Logic Models and Context and Implementation
Recent requests from CADTH's HTA customers focus on their need for information to support implementation of recommendations about the use of health technologies following expert committee deliberations. For this reason, “implementation issues” is a distinct domain in HTERP's deliberative framework (6). The dialysis project included research questions related to the identification of strategies and processes to implement home-based and self-care dialysis modalities, and contextual factors that could contribute to successful implementation. Given that CADTH reviewers had little prior experience assessing implementation related issues, the team looked to the guidance and in particular drew on two chapters to develop methods to address these research questions: Logic Models (Reference Rowhwer, Booth and Pfadenhauer3) and Context and Implementation (Reference Pfadenhauer, Rohwer and Burns5).
The guidance (Reference Rowhwer, Booth and Pfadenhauer3) proposes that a logic model be developed and used to serve three related functions over the course of the assessment. First, the model provides a graphic representation and structured overview of the interactions among participants, interventions, comparators, and outcomes. The logic model should identify and formally incorporate patients’ preferences and other treatment moderators, as well as context and implementation issues that may influence whether or not an intervention is delivered and, if so, the potential for effectiveness in a specific circumstance. In doing so, the logic model is intended to help the assessment team and stakeholders to identify and articulate the evidence that is relevant for the assessment.
Second, the model provides a conceptual framework to guide the evidence assessment of individual aspects. Third, the logic model is extended to assist decision making by graphically representing the relationships and interactions among assessment results within the decision context. In the case of assessing the use of dialysis modalities in Canada, HTERP members would need to consider the various jurisdictional contexts for which they are developing recommendations.
CADTH Experience
An immediate challenge the team faced for this aspect of HTA was how to characterize context and implementation issues. In response, the team used the definitions for context and implementation provided in the guidance, as they appeared to be applicable prima facie. For the developers of INTEGRATE-HTA, context refers to a set of characteristics and circumstances that consist of active and unique factors that interact, influence, and modify, facilitate or constrain the intervention and its implementation (Reference Whalster, Brereton and Burns1). Likewise, implementation refers to an “actively planned and deliberately initiated effort with the intention to bring a given object into policy and/or practice.” (Reference Whalster, Brereton and Burns1). These definitions were used to guide the population of the logic model, as described below, and to characterize the data that would be collected and analyzed to respond to the research questions within the “implementation issues” domain.
Data collection strategies included a literature review to identify implementation issues that have been explicitly discussed in the scientific literature, as well as two national surveys: one of nephrologists, and one of other clinicians and administrators involved in the delivery of dialysis care, to identify the range of strategies that have been used to establish or increase the uptake of home-based and self-care in-center dialysis programs in Canada. The definitions for context and implementation in the guidance were used to select terms for the literature search strategy and for structuring questions used within the survey. Ultimately, these definitions and the four implementation domains (i.e. “provider,” “organization and structure,” “policy,” and “funding”) as well as an additional domain of “patient” also provided the coding structure through which data collected from both the literature search and survey were analyzed.
While the guidance suggests that a logic model should inform all dimensions of the HTA, in this case, we drew on the guidance related to logic models to assess the implementation questions only. Historically, CADTH has not consistently used an explicit framework or model to structure thinking about the interaction of the components of its HTAs. The dialysis HTA provided a unique opportunity to begin using this approach; however, given this was our first experience using the guidance, the decision was made to proceed with standard CADTH methods for other aspects of the HTA (e.g., clinical and cost-effectiveness).
We decided on an iterative logic model, primarily for practical reasons: due to limited time available for protocol development, we were not confident we could develop a reasonably comprehensive a priori logic model in the allotted time. Instead, we wanted the opportunity to refine the logic model as we learned more about the technology, patients’ perspectives, and the policy context, and had the opportunity to speak with clinical experts and hear from other stakeholders.
The broad literature search helped to identify articles with information to support the preliminary development of a logic model, which was structured initially around the 12 domains of context and implementation and their relationship to each dialysis modality, and the intended outcomes. We also took the opportunity to ask questions of clinical experts engaged in the HTA about the interactions among patients, setting, technology, providers, and outcomes.
Through this process, we were able to identify countless modifying factors that seem to influence whether or not people are able to access specific dialysis modalities, and if so, the potential for effectiveness of those modalities in their specific circumstance. For example, patient factors including age, clinical status, comorbidities and cognitive ability appeared to affect both the ability of people to access interventions and their effectiveness. Likewise contextual factors, such as access to clean water, distance to a medical facility, and caregiver support, were identified as potentially affecting people's experience with dialysis modalities.
Ultimately, because of time and resource pressures, a logic model was not completed nor included in the HTA. A working draft was developed, but it was not completed in such a way that we could be confident it identified the most important elements that lie on a causal pathway between dialysis use and effectiveness outcomes; nor did it specify, in sufficient detail, the relevant interactions between the various aspects. In this sense, our first attempt at developing a logic model was constrained by the implicit view that each aspect of HTA can be assessed by an isolated piece of research.
While we were unable to develop a logic model, working through the steps was useful to start conceptualizing links between different aspects of an intervention and its delivery and impact. The model proved helpful in understanding the connections between the components of the technology, the range of variables influencing delivery and patient outcomes, and subsequently in identifying factors that could affect successful implementation. What we learned by going through this process was applied when analyzing data obtained through the surveys, and ultimately in responding to the implementation related research questions.
Specifically, the implementation issues we identified included: (i) Policy and funding level: lack of policies for promoting home-based modalities and lack of funding for establishing the requirements of home-based programs; (ii) Organization/health institution level: lack of health administration and clinical staff support and appropriate infrastructure, education, and training; (iii) Healthcare provider level: nephrologist preference for in-center HD; (iv) Patient level: lack of education, increased utility costs, housing issues, burden on family, lack of interest in home treatment and a preference for in-center HD.
Working through the process of developing a logic model also helped to identify subgroup analyses to be pursued in the clinical review, including analyses based on place of residence, training provided for patients and caregivers, and caregiver relationship to patient. Finally, we also used the categories from the draft logic model to help guide the final summary and discussion in the HTA report. While returning to the logic model to map the evidence at the end of the HTA is an important use of the logic model for INTEGRATE-HTA, we have not yet done this.
Ethical Issues
The guidance (Reference Pfadenhauer, Rohwer and Burns5) identifies five characteristics of complex technologies that are particularly germane to a rich discussion of the ethical challenges posed by these technologies: multiple and changing perspectives, indeterminate phenomena, uncertain causality, unpredictable outcomes, and ethical complexity. The guidance also provides information on potential approaches that may be used to identify and address ethical issues in HTA and provides a helpful summary of the various meanings of “integration.” The core of the guidance on ethics is a procedural framework that specifies the steps involved in conducting an ethics analysis: (i) assess the complexity of the technology, (ii) select an approach with which to identify and analyze ethical issues, (iii) confirm and modify the approach, (iv) apply the approach, (v) the outcome of approach. The process emphasizes the need for the other aspects of the HTA and modifiers (clinical effectiveness, economic, implementation, patients’ preferences, etc.) to provide information for ethical reflection and for the ethics analysis to reflect on those other aspects.
CADTH Experience
For this project, CADTH identified ethical issues through a systematic review of explicit ethical issues and an analysis of other ethical issues identified by an ethicist. The approach was, broadly speaking, principalist in orientation, but included a more wide ranging value inquiry by considering questions regarding the social status of ESKD and other socio-cultural and legal issues. The results of the ethics analysis were presented as a separate section within the final HTA report, and organized around key issues at the micro, meso, and macro levels.
The ethics analysis identified the following issues as being particularly relevant to dialysis modalities in Canada: (i) Public versus private funding of health care (an on-going question) and fee-for-service versus salaried physicians in the provision of ESKD care; (ii) Reallocation of resources from institutional to community/home based care; (iii) The justification for an allocation policy, PD (or home HD) as the default and in-center HD as the exception; (iv) The importance of physician and patient education and informed choice; (v) Patient rights versus broader public good (i.e. should patients eligible for PD be required to use it).
Identifying and addressing ethical issues arising in the other sections of the HTA is an important function of the ethics analysis. From a project standpoint, meaningful integration requires the involvement of an ethicist in project team discussions, including discussions about the logic model. Without this early involvement, it is much more difficult to identify and discuss important value judgements made in the conduct of the HTA. In the case of dialysis, one such issue was patient-borne costs for in-center as compared to at-home dialysis, and the circumstances under which such costs might be alleviated by productivity gains.
Ultimately, identifying the issue provided two benefits: the economic evaluation could identify and discuss these assumptions and the ethicist could also discuss the ethical implications of aspects of the analysis that could not be considered explicitly in the economic analysis. Nevertheless, because a parallel approach had been used, the issue was not identified until the HTA evidence was presented at an expert committee meeting, resulting in additional work to address the issue and incorporate the relevant discussion into the final HTA report.
DISCUSSION
In our first attempt at using the guidance, we focused on those sections that related to aspects for which CADTH's methodological approach is currently under development rather than attempting to modify our approach for aspects for which there were already well defined methods. Part of this choice was opportunistic: our methods team was eager to become familiar with and apply the guidance for these new aspects. The choice was also necessitated by current process, timelines, and available resources, which were fixed. While we ultimately were not able to implement the guidance in its entirety, working through the guidance allowed the team to begin thinking about complexity and integration, and how these could or should relate to HTAs conducted at CADTH.
Complexity and integration are both relatively new concepts to most HTA researchers, and there is a steep learning curve to understand the theory before applying the concepts in an appropriate and meaningful way. The guidance provided us with helpful and necessary direction for thinking through the potential aspects that ought to be considered in an HTA. In the context of the dialysis HTA, it provided conceptual clarity and a reminder that several modifying factors will influence whether and how an intervention is delivered, and ultimately whether an intervention will be effective, or not.
Considering the guidance in this dialysis HTA was a step toward reconceptualizing our HTA methods, in particular, how we may begin to more meaningfully integrate the broad range of evidence relevant to HTA. It allowed us to think critically about our process, and helped us to realize that we need to discuss more carefully what integration means for our organization and what processes may need to be adjusted to realize the kind of integration that the guidance is attempting to achieve. While the complete integration of all components is not CADTH's aim, we would hope to reach the level of integration described by INTEGRATE-HTA as “coordination.”
Perhaps the most challenging issue in our experience working with the guidance was our attempt to apply it while working within CADTH's standard “parallel” HTA process, in which each information domain is treated in isolation. It quickly became apparent that it is impossible to assess different, interacting aspects independently and then attempt to complete the integration afterward (Reference Whalster, Brereton and Burns1). For example, in producing the report, it was believed that there was some redundant information as a result of overlap among the aspects of assessment and modifiers being addressed in the various sections. The authors of the report tried to deal with the redundancy by identifying and locating similar information within themes in the Discussion section of the report. While this served the function of acknowledging the perceived redundancy, it was an admittedly ad hoc approach and ultimately did not address the source of the problem. In any case, the INTEGRATE-HTA model does not provide guidance about the relevant notion of redundancy or how to handle redundancies of this kind.
Relatedly, thinking through the guidance highlighted for us the fact that, while the deliberative framework for the HTERP recommendation committee asks for information regarding ten distinct domains, it indicates neither how the domains are related nor how the information might be amalgamated for decision making. Further thinking and conceptual understanding of what is meant by “integration” is needed. While information from each of the ten domains is informally integrated during expert committee deliberations and recommendation development, meaningful integration could be hampered because the interactions between the information domains are not explicitly recognized or explored. A set of common characteristics of complexity that could be used across the different aspects may help to better integrate the domains of the HTA.
Process Challenges: Applying Guidance Requires Changes
Our experience using the guidance also highlighted the need for process changes to be able to implement the guidance as intended. Specifically, there appears to be value in a more staged approach to gathering and analyzing evidence, in contrast to the parallel model. Based on our experience at attempting to develop a logic model, we see a valuable role for logic models earlier in the HTA process; they would help the team to identify various issues that may then be acknowledged in, or that may inform, subsequent aspects of the HTA. To include this step requires more time before the final HTA protocol is written and also requires engaging with relevant stakeholders earlier in the process.
Moving forward, a challenge from an agency perspective will be to achieve a level of detail in the logic model that is proportionate to the intended purpose and audience of the report. Agencies like CADTH, for reasons of credibility, consistency, and efficiency, prefer to have a relatively narrow set of options for conducting various aspects of their HTA. The same pressures will apply to the use of logic models.
Similarly, for some “modifying factors” in the INTEGRATE-HTA model, CADTH is considering gathering relevant data earlier in the process to inform a subsequent assessment of each aspect of the HTA. For example, evidence on patients’ experiences and perspectives has typically been analyzed by an independent research team in parallel to the collection of evidence for clinical and cost effectiveness, and results are provided in distinct chapters in a final HTA report. Based on the guidance, it appears that it would be more profitable to collect and analyze these data at an earlier stage so that they can inform other aspects of the HTA, for example to help identify important clinical outcomes, and identify and challenge or confirm assumptions about the ways in which patients value and interact with the technology.
Finally, the guidance also recommends involving ethics expertise when the HTA is initiated, to help make explicit the moral dimensions of the technology and the assessment itself. If CADTH were able to implement this guidance, we expect earlier identification would help to consider explicitly the importance and implications of contextual factors that may be of relevance to CADTH customers. To implement any or all of these process changes would require changes to the way in which HTA projects at CADTH are resourced, schedules are managed, and how protocols and reports are drafted, reviewed, and presented. None of these changes are insurmountable; however, substantial effort would be required to ensure processes within a large agency are aligned.
Finally, our experience suggests to us that implementing the guidance at any similar agency would require focused and perhaps ongoing training. Training would be required on the guidance in general and the proposed assessment framework, and also on specific topics such as logic models, complexity theory, and implementation science. Even with an existing understanding of HTA and practical experience assessing a range of technologies, there appears to be a steep learning curve for reviewers that can challenge implementing the guidance, especially within a tight HTA timeline. HTA researchers, generally, are not familiar with the broad range of disciplinary methods that are proposed in a wide-ranging assessment nor using the same “data” (e.g., perspectives of patients on the acceptability of a technology) to assess different aspects of a health technology. To facilitate broader understanding of these issues internally, CADTH has initiated an INTEGRATE-HTA study group, with individual project team members responsible for knowing their chapter of the guidance in detail, and for working through the other chapters with other members to understand how the model is intended to work. As the guidance notes, integration needs to begin at the start of the project, and we expect that having a study group will facilitate implementation of all aspects of the guidance in a coherent manner.
CONCLUSION
INTEGRATE-HTA provides detailed and helpful guidance for truly integrating the broad range of issues involved in assessing complex health technologies. We have described the first experience of our agency, CADTH, in implementing some of the INTEGRATE-HTA guidance in the context of a review of dialysis modalities for end-stage kidney disease. We were challenged by a steep learning curve for research team members, tight project timelines, and a misalignment of current HTA processes with those required to implement the guidance as intended. Despite these drawbacks, considering the guidance in the context of this HTA initiated a dialogue about what might be needed to assess complex interventions and the potential process changes that could facilitate the completion of more integrated assessments.
CONFLICTS OF INTEREST
The authors have nothing to disclose.
