Calls for citizens and patients to be “at the center” of healthcare systems worldwide have permeated the health policy arena where public and patient engagement in a wide range of activities is increasingly encouraged (Reference Dipankui, Gagnon, Desmartis, Legare, Piron and Gagnon1–Reference Wortley, Street, Lipworth and Howard3). By “public and patient engagement,” we mean the incorporation of the views and perspectives of those who use or are affected by technologies into the assessment of these technologies. In health technology assessment (HTA), public and patient engagement can take many forms, including inviting patients to join expert panels, to provide evidence mediated by an interviewer or survey, or to provide written submissions about the technology or condition being considered (Reference Wortley, Street, Lipworth and Howard3;Reference Abelson, Wagner, DeJean, Boesveld, Gauvin and Bean4).
Public and patient engagement initiatives are commonly seen (and sometimes strongly advocated by government bodies) in health-related settings like hospitals and quality improvement initiatives (5), clinical guidelines groups (Reference Hameen-Anttila, Komulainen, Enlund, Makela, Makinen and Rannanheimo6), health research (Reference Shippee, Domecq Garces, Prutsky Lopez, Wang, Elraiyah and Nabhan7), and health technology organizations (Reference Dipankui, Gagnon, Desmartis, Legare, Piron and Gagnon1–Reference Wortley, Street, Lipworth and Howard3).
Although the practice of engaging the public and patients (PPE) in HTA has spread worldwide, gaps in knowledge about how best to organize and implement these efforts remain (Reference Single, Facey, Livingstone and Silva8–Reference Vanstone, Abelson, Bidonde, Bond, Burgess, Canfield, Schwartz and Tripp10). We sought to support these efforts by casting a critical eye on the current state of PPE in a single HTA agency—(the Canadian Agency for Drugs and Technology in Health or CADTH)—Canada's largest HTA organization. The agency was chosen for its pioneering work in this area (Reference MacPhail and Shea11) and its willingness to work collaboratively with us to strengthen its PPE work through an evaluation of existing practices.
Study Context and Public and Patient Engagement at CADTH
To understandCADTH's role in HTA and related policy making in Canada, it is important to recognize some of the nuances of the country's healthcare system. CADTH's mission is to “deliver credible scientific evidence and management strategies that enable the appropriate use of health technologies” (12) to help Canada's decentralized system of thirteen federal, provincial, and territorial healthcare plans make decisions informed by rigorous scientific assessment (13). Although CADTH is the largest producer of HTAs in the country, HTA activity is also undertaken at the provincial, hospital, and regional health authority level across the country to suit local needs (Reference Martin, Polisena, Dendukuri, Rhainds and Sampietro-Colom14).
CADTH initiated its PPE activities at a time where there were no PPE standards to follow; for this pioneering role, it has gained international recognition as a leader in the area of PPE. At the time that this study was undertaken (September 2017–December 2018), CADTH offered a broad range of activities in which it consulted and engaged with patients, caregivers, and members of the public. The rationales for these initiatives were (i) to promote fairness and inclusion of a range of stakeholders, (ii) to enhance public or patient support of decisions, (iii) to help ensure decisions reflect public and patient values, and (iv) to enhance public and patient understanding of HTA (Reference Wortley, Street, Lipworth and Howard3). CADTH engages in a range of PPE activities and this study focused on activities initiated by CADTH, where members of the public, individual patients, and patient advocacy groups participate in governance and advisory roles, providing input and their perspectives or participating in events organized by the agency. CADTH also includes patient perspectives in its work through syntheses of the literature on patient experiences, opinions, and beliefs. The synthesis of secondary evidence was not the central focus of the PPE activity examined in this study.
Our objectives were to (i) develop a deep understanding of CADTH's foundational PPE principles (e.g., mandate, governance, goals) and (ii) describe staff and stakeholder experiences with PPE (e.g., expectations, tensions, issues, culture of engagement) to allow CADTH to identify areas of strength and potential improvement.
Methods
We used the qualitative research methodology of Institutional Ethnography (IE). As a research strategy, IE emphasizes exploration and discovery to answer questions about everyday life and experiences and how the everyday life of people is organized and coordinated by text in organizations (Reference Campbell15;Reference Tummons16). IE puts emphasis on material text as the connection between people's lived experiences and the institutional relations that organize and regulate those experiences (Reference Campbell and Gregor17).
IE takes people's experiences as a “point of entry” (Reference Devault18) into understanding organizational processes. Power dynamics are important in IE, and the researcher is required to adopt the standpoint (or “take the side”) of groups with lower levels of power, termed “those being ruled.” Through social inquiry, the research explores and exposes the organizational processes and makes explicit how “ruling” or management and governance practices affect people. Qualitative interviews allow the researcher to learn about the interviewee's social location in the organizational hierarchy (Reference Walby19).
Fieldwork
This study anchored itself on what was happening to people involved in PPE activities at CADTH. Our standpoint is the experiences of those closely involved in and supporting CADTH's PPE portfolio of activities including staff, patient advocacy groups, and committee members (hereafter “informants”). As well, we sought to understand and articulate the standpoint of subjects “who disappear” in the social relationships (Reference Smith20), for example, patients not involved with advocacy groups. We interviewed fifty-three participants from patient advocacy groups (n = 18), patient liaison forum members (n = 5), staff (n = 15), committee members (n = 14), and a small number of industry participants (n = 1) to understand the work processes and how they were organized.
Data Collection
Data were collected in several ways: interviews with staff and others involved in CADTH PPE; observational field notes made by the first author when spending time at CADTH during a fellowship; and the institutional and external texts referenced by participants or the organization. Through these multiple sources of data, we gathered understandings about the intersections between the work that was required of informants to be a part of CADTH's processes and the work of a wide range of people variously involved in formal and informal PPE ways in CADTH. The observational field notes paid close ethnographic attention to people's practices within the institution, providing valuable data about the social organization of the broader institution.
Informants
We recruited informants through the following means: (i) e-mail invitations sent by CADTH's personnel, (ii) e-mail invitations directly sent by the first author inviting past advisory committees’ members, and (iii) snowballing/word of mouth.
Interviews
The first author conducted all interviews between April and September 2018. Interviews were in English, lasted between 30 and 60 minutes, and were audiotaped and transcribed verbatim by a professional transcriptionist. Given that CADTH staff and stakeholders live and work in different locations from where the first author's office was located, most interviews were conducted over the phone, whereas some were in person; this was determined by logistics and participant preference. Interviews were open-ended and semistructured. The interview guide was revised iteratively to reflect new insights learned and to inform and change the subsequent interview (Reference Devault, McCoy, Gubrium and Holstein21). Participants were identified in the manuscript through the letter P and a participant ID number. All informants signed a consent form and researchers ensured their confidentiality.
Direct Observation
As a CADTH Health System Impact Fellow, the first author was fully engaged and committed to this project. As an embedded researcher (Reference McGinity and Salokangas22), she observed advisory committees and gathered field notes during her research activities within CADTH. She also participated in national/international conferences organized by CADTH and webinars and events (social and academic) offered throughout the study period.
Text
In IE, text means “materially replicable words or images” (Reference Smith and Turner23). Textual review was ongoing; information on participants’ interviews and first author observations led the selection of text used in this study. Texts mentioned by informants were documents (e.g., Web sites, terms of references, videos, scientific publications, prior agency evaluations), government-produced documents (e.g., standing committee reports, Health Acts, the Local Health Systems Integration Act 2006), international organizations’ documents (e.g., NICE, Scottish Medicine Consortium, European Medical Association) as well as books or journal publications (e.g., The Illness Narratives, Forest 2018).
Analysis
Analysis in IE “is reflexive, iterative, political and relentlessly empirical, and develops as one thinks and write” (Reference Rankin24). Analysis in IE seeks to locate individual experience within the organization; it uses individual narratives to tell a broader story about relations and organizations (Reference Walby19). The analysis was carried out in concurrent phases as follows: (i) listening to audio recordings, (ii) reading and rereading transcriptions and notes; at this point, (iii) we shared preliminary results with CADTH personnel to review initial findings and discuss ongoing analyses, (iv) mapping interconnections (i.e., identifying all references to text from the interviews and reading them, linking what people said to the text to identify the work they do, the text, and others), (v) organizing the data considering the ideological practices of the institution, and the text, and (vi) analyzing and writing about the data organized as described. Through the analysis, we noticed taken-for-granted practices, contradictions and tensions, and clues about social relations organizing people's practices. We looked for what people said about the work they do that connects them to the work of what others are doing elsewhere (Reference Smith20).
Reflexivity
The team expertise in HTA and PPE was important for this project and provided the forum for critical reflection. Expertise ranged from including an agency leader, three independent researchers with expertise in HTA, ethics and PPE, and someone with experience in international HTA organizations. We used several techniques to examine our own position in the research. First, the first author kept notes following interviews and committees’ observations. Following (or in between) interviews, the first author shared critical reflections on the interviews that facilitated the discussion of different perspectives. We took a collaborative approach to the analysis (bimonthly check-ins) to draw on the different disciplinary and interpretive skills of each researcher. We discussed each team member's relationship with the data and acknowledged its role in guiding research decisions.
Results
Our results are organized around three lines of argument: CADTH PPE work (what is happening on the ground) as an entry point to PPE practices, how PPE is socially organized, and the experience of PPE.
CADTH PPE Work (What Is Happening on the Ground)
CADTH's PPE activities have grown in the past decade. Figure 1 presents a summary of the involvement of the public or patients in CADTH's activities during the study period:
CADTH patient and public activities. CDEC, Canadian drug expert committee; CDR, common drug review; pERC, pan-Canadian expert review committee; CODR, pan-Canadian oncology drug review; HTERP, health technology expert review panel.

The Board of Directors sets the strategic direction of the agency (vision, mission, and values), and following a standard governance structure, interacts with the Chief Executive Officer (CEO). The agency is armed with state-of-the-art scientific methods and practices and highly qualified individuals from disciplines traditionally linked to HTA (e.g., medicine, epidemiology, health economics, pharmacy) to conduct its work. Agency staff describe feeling pressure to produce timely results and to maximize efficiency in all areas. The work occurs within very specialized teams, which informants described as limiting communication across the entire organization.
The expert committees work in close relationship with these specialized teams. There are three committees (common drug committee or CDEC, pan-Canadian oncological expert review committee or pERC, and health technology expert review panel or HTERP), whose members are scientific leaders in the country. Members use assessments of clinical and cost-effectiveness generated by the agency to make recommendations that influence the funding and delivery of Canadian health care. For historical, political, as well as pragmatic and strategic reasons, these expert committees carry out their work using different frameworks and processes. Committee approaches to patient input and engagement have not been (consciously or explicitly) aligned. Figure 2 details how the different committees gather input from patients, how they carry out their review and recommendation development processes, and how they undertake their follow-up activities.
Committees' approaches to patient input. CDEC, Canadian drug expert committee; CDR, common drug review; pERC, pan-Canadian expert review committee; CODR, pan-Canadian oncology drug review; HTERP, health technology expert review panel.

Patient input is gathered in three different ways: through patient submissions (i.e., electronic questionnaire response to open calls for patient input), through interviews with individual patients, and through syntheses of published literature focused on eliciting patient values and experiences. One-on-one interviews are sometimes conducted in parallel to the synthesis or to inform its development.
The Scientific Advice (SA) program is a fee-based consultancy service offered to companies developing pharmaceuticals typically at early stages of the development (i.e., prephase III) since 2015. Patients’ lived experiences have been incorporated into every project; patients are recruited as individuals or from patient advocacy groups (25). In addition to interviews, published literature or relevant information reviewed by the common drug and oncological committees is sometimes included. The program Web site states about patient involvement: “At this point in time, best practices for involving patients in early SA programs are not established. The agency's process may evolve as more experience is gained” (25).
How CADTH's PPE Work Is Socially Organized
CADTH's approach to public and patient engagement includes some clear strengths, for example, small groups of dedicated staff, coordination and communication with patient organizations, valuing and recognizing of patient participation, and the multidisciplinary nature of committees. Nevertheless, we identified some important challenges related to engagement practices in the organization, and how patient engagement input is treated.
Engagement Practices in the Organization
Who Are the Public and Patients Involved?
Currently, an estimated 150 patient advocacy groups, coalitions, or networks and a few individual patients respond to calls from CADTH to provide input when it is requested. CADTH's relationship with these advocacy groups has matured and evolved through the years. They have found ways to learn from each other and nurture each other's needs through active listening and mutual respect.
However, the mechanisms for soliciting input from patient organizations have certain limitations: the form-based patient submission template represents a preconceptualized view of patients’ perspectives and experiences. Also, the current structure for patient input allows certain voices to dominate the process; this leads to the involvement of a narrow group of already active and engaged members of the patient community and, according to some, leaves behind “whole swaths of the patient population” (P11), particularly those not associated with disease advocacy groups. Without clear rules for its integration with other forms of evidence, patient input may not be seen as equally valuable as other important considerations such as efficacy and cost factors.
What Are the Processes?
Patient input processes followed the format and instructions determined by the organization. These often suggested a preference for quantified measures of patient experience, which we identified as a ruling interest at CADTH. Patient input was used in a number of different CADTH activities. First, patient input contributes to the institutional production of knowledge; with this approach, patient perspectives address an institutional need and produce data in form and quality, which serve the decision makers. Second, the input process presupposes (without explicitly mentioning) certain expectations from patient groups, such as their knowledge of CADTH's work and the related HTA process, an understanding of the technicalities of a drug or medical device approval process, the ability to accommodate CADTH's timelines, and the human resources required to engage in the process and to devote to patient input data collection and analysis. These presumptions may overestimate the capacity of patient organizations to fulfill these responsibilities. Informants highlighted many difficulties in fulfilling the current roles entrusted to them, in particular, demands for time, human, and financial resources. With increasing demands for patient organization participation in other health sectors (e.g., health research), the future of patient input in HTA derived from patient advocacy groups as currently implemented is uncertain.
The instructions for patient input provided by the agency reveal a unidirectional relationship with no explicit returns to match the effort required for this activity.
Is Patient Input Advocacy or Access to the Decision-Making Process?
CADTH perceives patient input as a form of promoting fairness and inclusion and a way to reflect patient values in decision making (Reference Weeks, Polisena, Scott, Holtorf, Staniszewska and Facey26); yet there is a delicate balance of remaining patient centered and attending to a fair or equitable process. One of the major reasons patient advocacy groups engage with the agency is the hope of gaining access to the decision-making process and the technology. An analysis of texts (Supplementary Table S1) indicates that an unclear understanding of the purpose of the patient submission process and its opacity has left patient groups doing what they were established to do—advocating for access to drugs and technologies for the people they serve. It is not clear that they have gained access to the decision-making process, as was envisioned by the Standing Committee on Health (Reference Smith27). We identify a ruling relation here, in that access to the decision-making process is determined by CADTH and not by, for example, patients wishing to engage.
How Patient Input Is Treated (in Relation to Other Sources of Evidence)
The process (i.e., generation, utilization, and reporting) of patient experiences generated visible and obvious challenges for informants. HTA is an evidence-based decision-making process in which the concepts of evidence include appraisal and verification through scrutiny (Reference Merlin, Weston and Tooher28). Evidence-based medicine (EBM) does not give explicit consideration to the value of subjective experiences in the HTA context. Patient narratives, or accounts of lived experiences, are not typically considered within the realm of EBM practice and are difficult to manage. HTA principles about hierarchies of evidence rank patient input (anecdotal or narrative type of evidence) as low level. In documents and on its Web site, the agency describes itself as “(scientific) evidence driven” (12), which makes reference to its epistemic traditions and understanding of knowledge—another way of ruling. This creates a tension with the explicit value it places on anecdotal or narrative types of patient evidence. The paradoxical relationship of scientific evidence and anecdotal information from patients is experienced across PPE activities at the agency and shared by informants.
The structuring of the process asks patient advocacy groups to produce their accounts of lived experiences in forms as close to the other types of evidence they consider as possible. In practice, this places the onus on advocacy groups to engage in research-like practices to collect information directly from patients. They lead surveys, focus groups, or phone interviews to prepare and submit patients’ experiences to the agency. Yet, there are multiple, layered interpretive lenses between what patients are experiencing and the decision making that arises from the committee work that they are contributing to. What these lenses do is digest the lived experiences prepared by lay (public and patient) members for consumption by expert committees, who have little dedicated patient engagement expertise.
Patient advocacy groups (as other members participating in the committees) are screened for their financial relationships with industry and funders. These relationships, driven by a range of factors, routinely generate questions about the quality of patient input submissions and the overall credibility of the patient advocacy groups. The judgments placed on patient advocacy groups echo common EBM practices related to the identification of bias in the production of knowledge and raise additional challenges to the PPE process at the agency.
Core Membership and Roles (or the Lack of)
An important point of discussion was the expert groups’ (drugs and nondrug) membership and roles. As stated previously, the oncological drug group includes patients (or caregivers) with cancer, whereas both common drug and nondrug committees include public members. At the time of conducting this study, the roles of most members were not described or described only vaguely, for example, “to bring a lay perspective” (common drug committee TOR). In these vague roles and the lack of rationale or clear description, we identify another ruling practice. Researchers’ observations reveal committees’ expectation for the public member(s) “to be” the voice and bring the perspective of the patient/patient group (in addition to the lay/taxpayer perspective). Regardless of the number of drugs considered by the committee in a given month, the common drug committee's public members are the only ones carrying the responsibility to process and deliver the patient input submissions during the committee's monthly meeting.
CADTH's patient and public member recruitment and selection is careful and purposive. Candidates often have long-lasting experiences with the healthcare system, have nonmedical backgrounds, and as mentioned above in the case of the common drug committee, have the responsibility to read, process, and present patient groups’ input summary to the expert committee (advertisement for new members). As such, public members have unofficial duties (i) to be agents for patients/patient advocacy groups, (ii) to represent the interests of taxpayer and “citizens more broadly”—meaning the public interest, and (iii) to bring awareness to certain dimensions of citizens' (public or patient) experiences of the healthcare system (e.g., issues of rural or remote vs. urban).
The current social arrangements about the importance and roles of public and patient members give rise to a range of tensions and paradoxes. The lack of clarity of roles (and the organizational “appetite” to explain them P13) and differences in committee membership reflects—to those whose standpoint we are exploring—the values informing CADTH's practices. In this particular context, institutional history may outflank informants’ concerns about fairness and transparency on committee membership.
Effective decision making is difficult if roles are not clear (e.g., are you a citizen, taxpayer, patient, all of them?). Unclear roles, as a result, drive members to engage with all kinds of things—advocating for themselves, their diseases/conditions or treatments they would like to see funded, or to silence them. The lack of a patient member in two of the advisory groups and the use of a public member as a proxy for a patient in another are challenges. They highlight differences in how the same agency using seemingly equal processes (input–synthesis–deliberation–recommendation) interacts differently with, and considers, the patient experience.
Discussion
CADTH's commitment to PPE is acknowledged by those within and outside the organization. Given this commitment, our results can help other HTA organizations learn by reflecting on the challenges described.
Our findings show that even with a demonstrated commitment to PPE, organizational history, structures, and practices can impede a unified approach. Some of CADTH's committee membership positions them better than others to capture the range and diversity of stakeholder values. A narrow, disciplinary committee approach reduces opportunities for balanced, challenging, and innovative decision making. Involving CADTH's stakeholders (including patients, citizens, or taxpayers) with pluralistic perspectives could enhance decision making in HTAs.
A lack of clear goals and processes, roles, and differential evidence treatment in PPE can minimize the relevance of patient experiences and displace their validity. Although clarity of roles may not be easy to establish, and with the acknowledgment that flexibility is needed, work in this area is warranted. Current ambiguity in public members’ roles requires these individuals to often play multiple roles to bring a public/social values lens while also ensuring that patients’ perspectives are understood. Improved clarity and precision of their roles and expectations will help address conflicting expectations for these members.
Clearer articulations of the roles of patient advocacy groups and of individual patients are also needed. Patient advocacy groups have been given the heavy burden of gathering material to prepare their submissions, including developing qualitative methods expertise, which are all viewed as useful to the HTA process. Conversely, these groups have been criticized for being biased, funded by the pharmaceutical industry (Reference Fabbri, Parker and Colombo29), and producing poor quality submissions. As a result, patient advocacy groups continue advocating for access to new technologies, rather than participating in a meaningful way in the HTA decision-making process. As in other HTA agencies, individual patients are invited to participate in some programs (i.e., nondrug committee, optimal use) (Reference Kleme, Pohjanoksa-Mäntylä, Airaksinen, Enlund, Kastarinen, Peura and Hämeen-Anttila30), but in CADTH, their roles are unclear, which is followed by unsystematic documentation and reporting of their involvement, both key components of transparency and reproducibility (Reference Mercer, Chambers, Mai, McDonald, McMahon and Chan31).
Biomedical intellectual traditions drive most HTA agencies (Reference Hofmann32;Reference Facey, Boivin, Gracia, Hansen, Lo Scalzo, Mossman and Single33), and CADTH is no exception. The high degree of value given to formal evidence-based principles at times conflicts with meaningful integration of patient input. An acknowledgment of conflicts between multiple epistemic traditions at work within HTA activities may strengthen HTA agencies’ approaches to PPE and avoid transforming PPE into a commodifiable product (Reference Löblová, Trayanov, Csanádi and Ozierański34). There is a need to recognize, accept, and incorporate other epistemological perspectives for dealing with patient engagement information and to ensure that each of these forms of evidence is enabled to provide valuable and complementary ways of thinking about health and HTA. Patient input submissions include anecdotal human experience, which is assigned to a low level in the hierarchy of evidence used in this tradition. Also, biomedical traditions simplify the disease or health issue to biological factors; a broader (i.e., biopsychosocial) model may help contemplate other dimensions of the disease (i.e., social and environmental). The use of PPE in HTA calls for a paradigm shift, moving from a strictly biomedical paradigm to one that emphasizes the social–political and moral dimensions central to PPE (Reference Löblová, Trayanov, Csanádi and Ozierański34).
Limitations
The results of this study provide insights that HTA agencies such as CADTH can use to strengthen its approaches to, and foster a culture that supports high-quality, PPE activity. The first author (JB) was guided mainly by the interviewees in regard to what text to read and was restricted to documents available to the public. These may have limited our findings and introduced some selection bias as texts may have been weighted more heavily in favor of issues of relevance to our informants. Study informants (patients, public, staff, and stakeholders) were self-selected volunteers who may have had different experiences and stories of those who chose not to participate. What we heard through our data were the perspectives of a selected group, which may not reflect how CADTH perceives what, or how PPE, functions for them.
Conclusion
This report focuses analytic attention on the social relations organizing the practice of CADTH's public and patient engagement. The analysis offered acknowledges people's experiences in the everyday world and the importance of creating embodied, self-critical, and coherent knowledge and practices. It makes visible, and provides, insights into various socially produced problems that are revealed through the work of different actors involved in CADTH's PPE practices.
CADTH has been an early leader in PPE practice but, like other HTA organizations, faces challenges in realizing its commitment in this area. A lack of clear goals and processes, roles, and the treatment of evidence in PPE can contribute to minimizing the importance of patient experiences and displacing their validity. Our findings point to the need for a clear agency-wide direction set by goals, values, and outcomes, a comprehensive framework, and policy and procedures that will help with operationalization. Since the completion of our study, CADTH has implemented some of these measures. This will result in clearer roles for public and patient members and contribute to greater transparency and responsiveness. The acknowledgment and recognition of other epistemic traditions, besides EBM, in addition to leadership and expertise in PPE, will strengthen CADTH's leadership role in this area.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462321000088.
Acknowledgments
We thank study participants for volunteering for this project. We acknowledge the support of Dr. Tammy Clifford, Vice President, Research Programs (Canadian Institutes of Health Research) and Ken Bond, Director of HTA and External Relationships (Institute of Health Economics). We thank Sharon MacNeil, Laura Tripp, and Terry Martens for their ongoing support during the first author's fellowship.
Funding
The study took place during Julia Bidonde's tenure as a Health System Impact Fellow, funded by the Canadian Institutes of Health Research (Grant #HT2-151-626) and CADTH.
Conflict of Interests
Dr. Bidonde reports grants from CIHR and CADTH, during the conduct of the study. Dr. M Vanstone has nothing to disclose. Dr. Schwartz reports grants from CIHR, during the conduct of the study; other: Arnold L Johnson Chair in Health Care Ethics, outside the submitted work; and she was formerly a member of an expert review panel for the organization described, and for which she received honoraria. Dr. Abelson reports grants from CIHR, during the conduct of the study.
Ethical Standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2013. The Hamilton Integrated Research Ethics Board granted ethics approval for this study (#4550).


