Health technology assessment (HTA) is a form of policy research which synthesizes and appraises information, mainly from systematic reviews and cost-effectiveness analyses, to inform a policy question (Reference Krahn, Miller and Bayoumi1). Over recent decades, it has become apparent that decisions about health technologies (adoption, optimal use, disinvestment) are not only scientific questions, but are heavily value-laden. HTA agencies, governments, and researchers have been exploring methods to better integrate the consideration of ethical, legal, and social issues (ELSI) into HTA (Reference Burke, Appelbaum and Dame2–Reference Potter, Avard and Graham10). They have also recognized the importance of patient and public perspectives (Reference Facey, Boivin and Gracia11–Reference Morin13), and embraced methods to integrate them into decision processes (Reference Abelson, Wagner and DeJean14).
However, this is challenging work. While unequivocally important, ELSI is arguably the broadest and most heterogeneous of HTA domains. Ethical issues abound, not only with respect to the use of technologies, but also with the methods and processes by which technology is appraised. Social values include equity, solidarity, population health, resource sufficiency, and more. Legal considerations are complex, and growing as a field of study.
In addition, patients' values and preferences, quality of life, experiences, and satisfaction are essential considerations in the appraisal of health technologies. These heterogeneous constructs underlie the “plus” in our “ELSI+”. Their complexity is compounded by the variety of professionals who study them (bioethicists, lawyers, psychologists, social scientists, health economists, and clinicians) and the various methods they use (bioethical analyses, surveys, quality of life and utility studies, and qualitative research). Finally, different professions adopt different terminologies to refer to very similar constructs (Reference Lehoux and Williams-Jones9). An economist is interested in preferences for health states. A qualitative researcher studies patient experience. A psychologist studies quality of life, function, or satisfaction. While these ideas are not identical, there is surely some congruence and overlap.
We believe there is a fundamental lack of clarity regarding key ELSI+ concepts (Reference Assasi, Schwartz, Tarride, O'Reilly and Goeree15), which lies at the heart of the challenge of integrating ELSI+ into HTA. For example, is there an underlying construct, or set of constructs, for “ELSI+”? What are the relationships between ethics, patient preferences, patient perspectives, quality of life, and social values? The literature characterizing the relationships between these ideas is limited. There have been a few systematic attempts to understand ethical considerations (8;16), and patient preferences (17;18), but little transdisciplinary work that spans the entire ELSI+ domain.
The objective of this study was to identify the key concepts in the ELSI+ domain and their inter-relationships, using concept mapping. A visual and conceptual representation of ELSI+ may provide a foundation for literature search strategies and a conceptual roadmap for HTA researchers and decision makers.
Methods
This study was approved by the Research Ethics Board of the University Health Network, Toronto, Canada.
Concept mapping is an analytic method used to guide evaluation and planning in health and medicine (Reference Burke, O'Campo and Peak19–Reference van Bon-Martens, van de Goor and Holsappel21). It combines qualitative approaches with quantitative analytical tools to visually represent ideas and their relationships in a picture or map (Reference Burke, O'Campo and Peak19), and thus is useful for exploring and understanding complex phenomena (Reference van Bon-Martens, van de Goor and Holsappel21). The steps in a typical concept mapping process include (22;23):
(1) Preparation—outlining the research goals and developing the focus of the study; identifying participants;
(2) Generation of statements—gathering knowledge and opinions to create a set of ideas;
(3) Structuring of statements—engaging participants to sort the statements into categories, and rate them on one or more attributes; combining the individual sorting and rating results into matrices based on how many participants grouped statements together and how they rated the statements;
(4) Concept mapping analysis—analyzing the statement relationships using computer software for concept mapping;
(5) Interpretation of maps—conducting group sessions to discuss and interpret the conceptual structure of the maps.
We followed these steps in our study, as described below.
(1) Preparation
Our goal was to understand the range and inter-relationships of ELSI+ concepts. We aimed to recruit participants working or studying in social science, bioethics, law, patient advocacy, health economics, and HTA.
(2) Generation of statements
Scoping Review
We followed the Joanna Briggs methodology (24) for scoping reviews to examine the extent and nature of issues, ideas, and concepts related to ethical, legal, social, and patient values in the HTA literature (Reference Arksey and O'Malley25). The search strategy (Supplementary Table 1), developed by an experienced librarian and peer-reviewed (Reference McGowan, Sampson and Lefebvre26), included terms related to social/societal or legal or ethics or quality of life (utility) or patient preferences/satisfaction or social values/norms. These were combined with terms related to technology or HTA.
We searched the following databases: (1) Ovid MEDLINE®: Epub Ahead of Print, In-Process & Other Non-Indexed Citations, (2) Ovid MEDLINE® Daily, (3) Ovid MEDLINE® 1946-Present, (4) Ovid Embase® 1974-Present. We limited the searches to English language studies published from 1 January 2000 (approximately when ELSI was introduced into HTA) to 2 March 2017.
Results from all searches were combined in a bibliographic database and duplicates deleted. The titles and abstracts of the remaining articles were reviewed independently by members of the study team. Duplicate papers that had not been identified previously were excluded. We retrieved full-text articles if the abstract reported on HTA, or at least one treatment, device, technology, or program; and ethical, legal, or social issues, or compliance, adherence, patient satisfaction, values, or preferences. Four reviewers (KEB, JB, JW, TT; see Acknowledgements) independently reviewed the selected full-text articles, and assessed their content for relevance to ELSI+ in HTA. The following data were extracted from the included articles and entered into an Excel database: first author, year of publication, journal title, topic of paper, and items describing the ELSI+ concepts and ideas. The extracted items were categorized into “ethical,” “legal,” “social,” “patient values and preferences,” “patient adherence/compliance/satisfaction,” “quality of life and utilities,” and “patient priorities.” This classification was used to organize items in the data. Because this was not a meta-analysis, we did not record the frequency with which ideas were mentioned.
Brainstorming Session
We approached ten individuals who, based on the literature and our personal knowledge, had expertise in health economics, health policy, social science, and health outcomes, including from the patient perspective, for participation in a brainstorming session. The purpose of this in-person session, with a video-teleconference option, was to discuss the relevance of the ELSI+ items found in the scoping review and to elicit others. Six academic researchers/professionals and a patient advocate were available to participate. Five representative scenarios from the literature were presented: (1) screening for fetal abnormalities, (2) a drug for terminal cancer, (3) a universal vaccine, (4) telehomecare for chronic illness, and (5) genomics and genetic tests. Participants were asked to think of and discuss ELSI+-related issues for each, and some ideas and issues mentioned in the literature were then shown and discussed. Notes were taken, and the session was audio-recorded with the verbal consent of participants. The recording was later reviewed and the notes were augmented.
Review and Consolidation of Items
We reviewed all items derived from the scoping review and the brainstorming session. We sorted items with similar ideas into groups, selected the most representative terminology, and eliminated duplicate ideas. A maximum of 100 statements is considered manageable for the tasks required for concept mapping (22;23). We aimed for no more than eighty statements in our consolidation, to limit the burden on participants.
(3) Structuring of statements
We used The Concept Systems® Global MAX™ software (27) to conduct and analyze the study. The list of statements was entered into the brainstorming page, and a randomization function shuffled the list and assigned statement numbers. These statements were used for the rest of the study.
The tasks required from online participants were “sorting” and “rating” the statements. The sorting task asked participants to place the statements into groups based on common themes that made sense to them, and then name the groups. Rating involved assigning each statement a number on a five-point Likert scale. Our question for rating was: “How important do you think that each of the issues below are to decisions about whether or not a technology should be adopted for use in Canada? Please select the number that best describes how important you think each of the concepts below should be in Health Technology Assessment.” The scale anchors were 1 = not at all important and 5 = extremely important, with 2, 3, and 4 (unlabeled) in-between. Demographic-related questions (gender, age group, type of work, area of expertise) were included to characterize the participants. Participants signed in anonymously or with their name, and they could save their responses at any time and return later to reduce task burden.
Participants
We searched the websites of thirteen Canadian universities for names and e-mail addresses of academics and researchers working in bioethics, health policy, or social sciences or law related to HTA. We also searched the websites of Canadian HTA agencies to identify individuals involved in HTA research, health economics, or patient advocacy, and scanned the author lists of the articles included in the scoping review to identify additional Canadians involved in ELSI+ endeavors.
We sent e-mails describing the study to 133 potential participants. We inquired about their interest in the study and asked them to forward the e-mail to others in their institution who might be interested. We received one “undeliverable” reply, and could not contact that person. We received automatic “out-of-office” replies from eight people on leave. Two people forwarded the information about the study to their group. All respondents who expressed interest in the study were e-mailed the consent form (to be signed and returned via e-mail) and the link to the online sorting and rating tasks, described above. We sent reminder e-mails to people who did not return the consent form within 10 days.
All forty-eight respondents who returned consent forms began the sorting task, and thirty-nine of them began the rating task. At least sixty of the eighty statements had to be sorted or rated to be considered a complete response for the purpose of analysis. Thirty-eight people completed the sorting task; one person sorted seventy-nine statements, and the others sorted eighty statements. A total of thirty-eight people completed the rating task, although one rated only seventy-three statements and three individuals rated seventy-nine statements. One person who assigned the same rating to all statements was excluded. Therefore, we analyzed sorting data from thirty-eight participants and rating data from thirty-seven participants.
(4) Concept mapping analysis
We used the Concept Systems® Global MAX™ software (27) to create point and cluster maps from the sorting and rating data. A point map locates each statement as a separate point on a two-dimensional (x, y) map, based on the frequency that statements were put in the same pile by the participants (22;23). Hierarchical clustering of the multidimensional scaling (x, y) co-ordinates was used to create clusters from the points, based on their proximity on the point map (Reference Kane and Trochim23). The Concept Systems® Global MAX™ software (27) allows researchers to create several cluster maps, each with a different number of clusters. The cluster maps require interpretation for the selection of a final concept map.
(5) Interpretation of maps
We created cluster maps with as many as sixteen and as few as four clusters. Two clusters were merged as each map with fewer clusters was created. For example, in going from fifteen clusters to fourteen clusters, two clusters (and their statements) were merged to form one cluster. By examining how statements were merged to form progressively fewer clusters, we decided on the number of clusters that best fit the purpose of the study. The goal was to find the number of clusters that retained meaningful differences between the clusters but also kept statements together that seemed to belong together (Reference Kane and Trochim23). The Concept Systems® Global MAX™ software (27) assigned labels to each cluster based on the names the participants gave to their groups of statements in the sorting task. These labels could be revised by the research team.
We used other analytical tools offered by The Concept Systems® Global MAX™ software (27) to analyze the point map and the cluster maps. For example, “bridging” analysis provided a value for each statement that described how well it reflected other statements in its vicinity on the map (Reference Kane and Trochim23). A statement with a low bridging value is an “anchor”—it was sorted with nearby statements by many people. A “bridging” statement has a high bridging value—it was sorted with statements somewhat distant from it on the map. The bridging analysis therefore provided information about the relationship of that statement to statements nearby and far away.
The results of the rating task were used to create point-rating and cluster-rating maps, in which the average ratings for each statement and for each cluster were super-imposed on a point and cluster map, respectively (23;27).
Results
Scoping Review
A total of 26,890 articles were retrieved from Medline (11,155) and Embase (15,735), and 6,882 duplicates were removed to yield 20,008 articles, which were entered into bibliographic software. Another 1,248 duplicates were removed when reviewing article titles, resulting in 18,760 unique articles. After reviewing the titles and abstracts, 546 articles were retrieved for a full review. Information was extracted from 251 full articles (Supplementary Figure 1).
Brainstorming Session
The brainstorming session confirmed that the scoping study was comprehensive. However, the discussion generated some new ideas, such as parents making decisions for their unborn child in the case of screening for fetal abnormalities, the preferences of the family in the case of a drug for terminal cancer, and the consequences of knowing one's risk for disease in the case of genomics or genetic tests.
Review and Consolidation of Items; Generation of Statements
Approximately 300 ideas were compiled from the scoping review and the brainstorming session. These were consolidated into 100 statements that were further reduced to eighty statements (Table 1). Statements were concise and to the point to facilitate understanding and limit ambiguity. Supplementary Table 2 describes examples of how statements were consolidated.
Final List of Eighty Statements for the Sorting and Rating Tasks
Participants
The characteristics of the participants are shown in Supplementary Table 3. Participants were working as researchers (36 percent), and academics (21 percent), and 42 percent self-identified as experts in HTA. Study recruitment and data collection were completed within 2 months.
The Point Map
The point map (Supplemental Figure 2) shows the two-dimensional non-metric multidimensional scaling of the sorting data (Reference Kane and Trochim23). The numbers correspond to the statement numbers (Table 1). The proximity of the points is determined by how likely they were to have been sorted in the same group by the participants. For example, six statements (26, 42, 44, 50, 58, and 74) are very close together. These statements are about sleep, disruption to daily life, mood and emotions, daily activities, and anxiety and stress (Table 1). Five statements (2, 27, 31, 34, and 69) are close to each other but not as close as the other six. These five statements concern universal access, rare diseases, equity, and distribution of health care and benefits (Table 1). Examination over the entire map indicated that the positions of the statements made sense (Reference Kane and Trochim23). The point map was the basis for all other maps, and its structure did not change.
Cluster Maps
We created maps with varying numbers of clusters. Given eighty statements, a map with twenty clusters would have an average of four statements per cluster, while a map with ten clusters would have an average of eight statements per cluster. We aimed to select a map in which the clusters represented different concepts and the statements within each cluster represented the same concept. We began with an overall ELSI+ domain comprised of ethical, legal, and social issues, and patients' values and preferences, so we anticipated that fewer than four clusters would not be practical, nor would more than sixteen.
Two members of the study team (MK, KB) and two of the study participants (VR, TT, see Acknowledgments) reviewed the cluster maps in order to select the optimal number of clusters for the ELSI+ map. We started with sixteen clusters and observed which statements were merged with the formation of fifteen clusters. At this point, we independently asked ourselves if the statements that were merged into one cluster belonged together. We each recorded whether the merge made sense (yes, no, undecided), and why it did or did not make sense (Reference Kane and Trochim23). We then looked at the fourteen-cluster map and examined the two clusters that were merged with its formation. Again, we recorded if we thought that the merged cluster included statements that belonged together. We continued this process until the map had four clusters. Figures 1A and 1B show the maps with sixteen and four clusters, respectively. It is possible to see which clusters were merged in going from sixteen to four clusters.
Cluster map creation. (A) The map with sixteen clusters. (B) The map with four clusters.
The four reviewers unanimously agreed with most of the merges from sixteen to five clusters, with two exceptions that were resolved through discussion. However, when the four-cluster map was created by merging a cluster with statements about patients' financial costs, location of care, and continuum of care with a cluster about incorrect test results, incidental findings, and ease of use, none of the reviewers thought the merge was appropriate. The statements in the first cluster were about operational matters and those in the second cluster were about risks and usability. Therefore, in assessing the merges made in going from sixteen to four clusters, it was decided that the map with five clusters was the most appropriate one. Each of the five clusters described a different concept, while the statements within each cluster described a singular concept. Table 2 describes the structure of the map with five clusters with the cluster labels given by The Concept Systems® Global MAX™ software (27).
The Clusters Comprising the ELSI+ Concept Map with Five Clusters With Statements, Mean Importance Ratings, and Bridging Values
Mean importance and bridging values for each cluster are bolded italics. Statements within each cluster are ordered by bridging value. The three highest importance ratings within each cluster are bolded.
We reviewed the statements in each cluster and the cluster labels (27) (Table 2). After some discussion, we renamed “Preferences” to “Patient preferences and experiences” and renamed “Patient quality of life” to “Patient quality of life and function.” “Fairness” was judged to be appropriate for the third cluster. “Characteristics of the technology” was renamed “Patient burden/harm” and the “Access” cluster was renamed “Organizational.” Figure 2 shows the final ELSI+ concept map with five clusters.
Final cluster map with five clusters for ELSI+ with mean importance ratings.
We analyzed the bridging values and importance ratings for the final ELSI+ concept map. The bridging value for each statement and the mean bridging value for each cluster are shown in Table 2. Statements and clusters with lower bridging values reflect the content in their vicinity well, whereas statements with high bridging values relate to other statements that are more distant from it on the map (Reference Kane and Trochim23). “Patient quality of life and function,” “Fairness,” and “Patient burden/harm” have statements with lower bridging values than the other two clusters (Table 2). “Patient quality of life and function” has the lowest mean bridging value (.09), while “Organizational” has the highest (.73).
We also obtained the mean rating on importance in HTA for each statement and for each cluster (Table 2, Figure 2). The mean importance ratings for all clusters were above 3.5. The concept of “Fairness” was considered more important (mean rating 3.69) than “Patient preferences and experiences” (mean importance rating 3.61), or “Patient quality of life and function” (mean rating 3.65). However, “Patient burden/harm” was rated more important than any of these (3.82), as was the “Organizational” cluster (3.92). On average, the statements rated least important were religious beliefs, desire to fit in with other people, how much life-style or behavior choices cause disease or illness, and the statement about rare diseases (if a few people need a very expensive treatment, we should help them regardless of cost). The statement “quality of life” had the highest mean rating on importance in HTA (4.68), but other highly rated statements were basic human rights (4.56), equity in access to treatment and health care (4.54), receiving accurate and adequate information (4.5), and risk of harm (4.5).
Discussion
This study used qualitative and quantitative research methods to characterize ideas related to ELSI+ in HTA. We extracted eighty statements related to ELSI+ from the literature and a focus group. Although this extraction involved some subjective judgment, all members of the research team agreed on its inclusiveness and appropriateness. Moreover, after reading over 200 articles, saturation was reached. We used concept mapping to visually represent ELSI+ ideas and their relationships (Reference Burke, O'Campo and Peak19). We selected a concept map with five clusters, each of which represented a different concept and, at the same time, the statements within each cluster represented the same concept. The five clusters were (i) Patient preferences and experiences, (ii) Patient quality of life and function, (iii) Fairness, (iv) Patient burden/harm, (v) Organizational.
Our final clusters were not completely congruent with the original “ELSI+” conceptualization. The four moral principles commonly used in biomedical ethics (Reference Beauchamp and Childress28) were found in all five clusters. Statements related to beneficence were in the “Patient preferences and experiences” cluster (patient satisfaction, patient preferences for treatment and outcomes), as well as the “Fairness” (maximum benefits for the most people, we should help the people who need help the most), and “Organizational” (beneficence; do no harm) clusters. Non-maleficence was addressed in almost all of the statements in “Patient burden/harm,” and in “Patient quality of life and function” (anxiety and stress, effects on daily activities, amount of disruption to daily life), and “Organizational” (additional financial costs to patients or family). The principle of autonomy was evident in the “Patient preferences and experiences” cluster (freedom of choice, autonomy/personal choice, patient self-efficacy, patient empowerment, patient involvement in decisions), but also in the “Fairness” (children's rights to make their own choices, receiving accurate and adequate information), and “Organization” clusters (ability to obtain informed consent (ill, disabled)). Most of the statements related to justice were included in “Fairness” (basic human rights, equity in access to treatment and health care, equity in health outcomes, universal access, balance between individual's benefit and societal good, individual choice vs. societal choice), but were also expressed in “Patient preferences and experiences” (respect for patients, desire to be treated as a whole person).
Statements about legal concerns were grouped in the “Organizational” (legal liability and responsibility for malfunctioning equipment, technical problems) and the “Fairness” (discrimination because of illness, treatment, or genetic test results) clusters.
Eleven social values relevant to HTA were identified by Giacomini: (1) Quality; (2) Evidence-informed policy; (3) Effectiveness; (4) Resource stewardship; (5) Resource sufficiency; (6) Equity; (7) Solidarity; (8) Population health; (9) Patient-centered care; (10) Collaboration; and (11) Shared responsibility for health (Reference Krahn, Miller and Bayoumi1). Many of these social values were distributed across our clusters. For example, equity, solidarity, and population health were included in the “Fairness” cluster, while patient-centered care was expressed in “Patient preferences and experiences.” The “Organizational” cluster included ideas related to resource stewardship and resource sufficiency.
With respect to the “+” in our ELSI+, three clusters addressed the patient perspective: preferences and experiences, quality of life and function, and burden/harm. The “Organizational” cluster also included statements that were relevant to patients, such as financial costs, and privacy of information. Although the “Fairness” cluster was more about patients as a group, its statements included discrimination, disability, and receiving information. Our conceptualization of ELSI+, therefore, appears to be very patient-centric.
Limitations
Representation from the public and patient communities would have allowed us to include their voices and assess similarities and differences between them and our participants.
Conclusions
This study offers an innovative approach to guide the integration of ELSI+ into HTA, applicable to any setting in which HTA is conducted. Articles in our scoping review were from Europe, the United Kingdom, the United States, Canada, and Asia.
Rather than a fundamentally profession-centric approach, we conceptualized ELSI+ based on the congruence of the underlying ideas, and in terms of patient preferences and experiences, patient quality of life and function, burden and harm, fairness (individual and societal), and organizational issues. This implies that the professions and academic disciplines that work within the ELSI+ domain (e.g. bioethics, social science, law, economics) cover a heterogeneous, but overlapping, set of concepts, which would benefit from multi-disciplinary and cross-disciplinary study.
Future work should (i) use a larger and more comprehensive sample of respondents, including patients, (ii) identify research methods associated with each concept, and (iii) create a validated search string for identification of the relevant literature.
Supplementary Material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462319000722
Acknowledgments
The study team acknowledges, with many thanks, the help of Joanna Bielecki, Valeria Rac, Teresa Tsui, and Josephine Wong (Toronto Health Economics and Technology Assessment Collaborative) with the scoping study and concept map replay exercise.
Financial Support
This study was funded by a project grant from The Canadian Agency for Drugs and Technology in Health. Murray Krahn is supported by a Tier 1 Canada Research Chair in Health Technology Assessment; Andrea Tricco is funded by a Tier 2 Canada Research Chair in Knowledge Synthesis.
