Qualitative Study

The specific research objectives of this study were to explore the barriers and facilitators that can influence the inclusion and use of PROs in research with patients with rare diseases. This study was approved by the Hamilton Integrated Research Ethics Board.

Through email, we invited international researchers and HTA experts to participate in a telephone interview conducted in English. We identified the researchers through the initial literature review and sought individuals who were first authors on articles in the field of LSDs that included the use of a PRO. We emailed the author of the studies using the contact information provided in the publication. We identified HTA experts in Europe and North America by reviewing the list of presenters and participants at the Health Technology Assessment international (HTAi) meetings in 2013 and 2014.

The research team developed the semi-structured interview guide. Open-ended questions prompted the participants to: (i) describe their experiences using PROs in their research or to inform HTAs, (ii) discuss the importance or the benefits of using PROs, (iii) describe some of the challenges they may have experienced in using PROs, and (iv) make suggestions regarding ways to improve or optimize the use of PROs in research or HTAs. See Supplementary Materials 1 and 2 (HTA experts and researchers, respectively) for interview scripts.

One researcher (P.M.) conducted all telephone interviews; another researcher (B.C.J.) was present for the first four interviews. The interviews, undertaken between October 2013 and February 2014, lasted approximately 1 hour, and were audiotaped. At the start of the interview, we obtained the participant's consent and collected demographic information. Two researchers (P.M., B.C.J.) used qualitative content analysis (Reference Hsieh and Shannon11) to analyze the audiotaped interviews that had been transcribed. For the analysis reported herein we anonymized the data (removed the name of participants and organizations). Qualitative content analysis involves the subjective interpretation of text data through coding and the identification of patterns or themes (Reference Hsieh and Shannon11). We applied conventional content analysis in which there was no application of preconceived categories or theoretical frameworks and the results are based in the actual data (Reference Hsieh and Shannon11). Both researchers independently reviewed the transcripts and identified preliminary codes, categories, and themes. The two researchers then met to come to consensus on the categories and themes, and selected representative quotes to highlight those themes.

We extended invitations to ten researchers and seven agreed to participate. There were three women and four men. Their research focused on Fabry disease (n = 2), Gaucher disease (n = 2), Pompe disease (n = 1), MPSII (n = 1), and both Fabry and Gaucher diseases (n = 1). The researchers resided in the United States (n = 3), Denmark (n = 1), Holland (n = 1), Israel (n = 1), and Italy (n = 1).

We extended twenty-six email invitations to HTA experts and four agreed to participate and five declined. We asked the five experts who declined our invitation to identify another potential interviewee (i.e., snow-ball sampling) (Reference Creswell12). This enabled us to identify two willing participants, totaling six HTA experts. There were two women and four men. The HTA experts resided in USA (n = 2), Germany (n = 2), Canada (n = 1), and Spain (n = 1).

Eligibility Criteria

Articles were selected based on sample, study design, and inclusion of an eligible PRO. Table 1 outlines the eligibility criteria. Eligible PROs of interest were those that tapped into the patient experience, including symptoms experienced, overall sense of well-being or quality of life. We included tested (i.e., those with published psychometric properties) and untested (ad hoc) PROs (i.e., without published evidence of psychometric properties). We excluded quantitative studies that did not address quality of life using PROs (e.g., measuring a patient's response to a noxious stimulus), as well as studies addressing prediction or describing the qualitative experience of patients with any of the LSDs of interest.

Table 1.

Inclusion Criteria for the Systematic Review

Study Selection

Using a pilot-tested, standardized form, pairs of reviewers worked, independently and in duplicate, to screen titles and abstracts of identified citations to identify the full texts of articles that either reviewer judged as potentially eligible. Subsequently, using another pilot-tested, standardized form, pairs of reviewers applied eligibility criteria to the full text of potentially eligible studies. Supplementary Material 4 includes our full-text screening form.

Data Extraction

Subsequent to the screening processes, the reviewers extracted data independently and in duplicate from each eligible study using standardized forms. To improve consistency, abstraction forms were accompanied by a detailed instruction manual. Before beginning data abstraction, we conducted calibration exercises to further ensure consistency between reviewers. Specifically, we assigned each review team the same two articles from which they extracted information on at least two occasions. A third, more experienced reviewer checked the abstractions and provided individualized feedback to each team throughout the data extraction process.

Extracted data included type of study, demographic information, details of the intervention, PROs used (both tested and ad hoc), results from both PROs and other outcomes, and specific details about the study methodology (e.g., cross-sectional, cohort, case-control), and results from articles that addressed the development or testing of PROs in LSDs.

We resolved any disagreements, at all stages of the screening and extraction, by means of discussion or, when disagreement remained after discussion, with the help of a third reviewer.

Data Analysis

We summarized the PROs (generic and disease-specific) and the ad hoc PROs used, as well as summarizing identified studies that reported the psychometric properties of PROs developed or tested among patients with our LSDs of interest. We presented the findings of both the descriptive and interventional studies separately. When possible, we used the minimally important difference (MID) to quantify the burden of disease (descriptive studies) or the responsiveness (interventional studies) of PROs.

Definition and Assessment of PROs Using the MID

The MID provides a measure of the smallest change in the PRO of interest that patients perceive as important, either beneficial or harmful, and which would lead the patient or clinician to consider a change in management (Reference Schunemann and Guyatt14).

We used the MID to quantify the burden of disease and magnitude of change for frequently reported PROs. We defined frequently reported PROs as those reported more than five times in our systematic review of five LSDs.

We used the same instrument-specific MIDs for the descriptive and interventional studies. Two primary approaches for estimating an MID are distribution- and anchor-based methods. Distribution-based methods rely on the distribution around the mean scores of the measure of interest (e.g., standard deviation[SD], standard error) (Reference Guyatt, Osoba, Wu, Wyrwich and Norman15). In the anchor-based approach, investigators examine the relation between the target PRO and an independent measure that is itself interpretable, the anchor. An appropriate anchor will be relevant to patients (e.g., measures of symptoms, disease severity, or response to treatment) (Reference Guyatt, Juniper, Walter, Griffith and Goldstein16). When available, we used published anchor-based MIDs (Reference Guyatt, Osoba, Wu, Wyrwich and Norman15). When anchor-based MIDs were not available, we calculated the distribution-based MID using published normative data (Reference Norman, Sloan and Wyrwich17). We considered normative data as that from a substantial sample of representative individuals in the general population, typically sampled by those who developed the outcome measure of interest (e.g., development of normative standards for the SF-36 by the developers of the SF-36). To calculate the MID, we divided the SD of the normative mean scores by two. When authors provided no normative or control data, we calculated the MID using the largest available sample of patients with the target LSD who were not receiving ERT from any study that used the PRO of interest. We divided the SD of the mean PRO score by two to determine the MID.

Application of the MID: Descriptive Studies

We applied the MID to quantify the disease burden if studies reported mean scores for patients, mean scores for comparator group, and the results demonstrated significant differences in scores between patient and comparator groups; or, failing that if authors did not compare study samples with normative populations, but included a very large sample (i.e., 200 patients).

To quantify the burden of disease in descriptive studies, we first identified the difference in mean PRO scores between the study sample and a comparator group, which could be a normative sample, or a sample of healthy controls (i.e., a sample smaller or less clearly representative than the normative sample). When the authors did not provide comparator data, we used the following approaches, in order of decreasing preference, to identify the comparator group: (i) we searched the literature for a large sample of healthy people for whom PRO scores were available (normative data); (ii) when normative data were not available, we used mean scores from healthy controls reported in a study included in our systematic review; (iii) when we could not identify a healthy control sample for scales measuring disability and pain, we assumed a sample of disability-free or pain-free individuals who would obtain the full score.

We then used the MID to determine the extent of the burden by dividing the difference between the patient and comparator means by the MID. We labeled the result a “decrement.”

In studies in which the authors quantified the difference between the patients’ PRO scores and those of the controls or normative sample, we quantified the burden of disease by calculating the number of MID units for the respective PRO in context of SD units. For example, we determined one SD to be equivalent to two MIDs (Reference Norman, Sloan and Wyrwich17).

Application of the MID: Interventional studies

To determine the magnitude of change or difference in experimental studies, we first identified the statistically significant between-group difference in PRO scores for controlled studies or the statistically significant within-group difference in PRO scores for observational before-after studies using the last reported time-point. In controlled studies, we identified whether the difference was an improvement or deterioration in health status associated with the intervention. In observational before-after studies, we identified whether the change was an improvement or deterioration in health status. Subsequently, we used the MID to determine the magnitude of difference or change by dividing the mean difference by the MID.

We considered a difference that was less than half of the anchor-based or distribution-based MID as a trivial effect; when the difference was between half of one MID and included one MID, we considered this a small effect; when it was greater than one MID but less than two MIDs, a moderate effect. We considered a large effect twice the MID or greater.

PRO-Specific Information

Below is the process we followed for: (i) selection of comparative data when authors of descriptive studies did not present normative or control data; and, (ii) identification of MID for specific PROs.

1. 1. Medical Outcomes Study Short Form (36) Health Survey (SF-36) (Reference Ware18). Comparator: When authors did not include normative or control data for comparison, we used normative values reported by Sullivan and colleagues (Reference Sullivan, Karlsson and Ware19) for the domain scores. This was a sample of 8930 healthy Swedish people, of whom 52 percent were female, and whose mean age was 43 years. For the physical and mental summary scores, we used control data reported by Schermuly and colleagues (Reference Schermuly, Muller and Muller20). This study enrolled a sample of 20 healthy people from the United States, of whom 55 percent were female, and whose mean age was 37 years. MID: We used the anchor-based MID for the eight domain scores and summary scores reported by Kosinski and colleagues (Reference Kosinski, Zhao, Dedhiya, Osterhaus and Jr21). The sample included 693 Americans, of whom 76 percent were female, and 84 percent were over 45 years old, with rheumatoid arthritis (RA).

2. 2. EuroQOL 5 Dimensions (EQ-5D) (Reference Szende, Oppe and Devlin22). (i) EQ-5D utility. Comparator: When authors did not include normative data or control data for comparison, we used normative data reported by Kind and colleagues (Reference Kind, Hardman and Macran23). The authors reported EQ-5D utility data for healthy adults living in the United Kingdom. The study enrolled 3,392 participants, of whom 57 percent were female. The authors did not report the mean age of the study participants, but noted that 41 percent of the women and 34 percent of the men were younger than 60 years. MID: We used the anchor-based MID for the EQ-5D utility score reported by Waters and colleagues (Reference Walters and Brazier24). The mean scores from ten patient groups including those with leg ulcers, back pain, early RA, irritable bowel syndrome, acute myocardial infarction, and COPD were used to calculate the MID (n = 532; no demographic data reported). (ii) EQ-5D Visual analogue scale (VAS). Comparator: When authors did not provide normative data or control data for comparison, we used data reported by Kind and colleagues (Reference Kind, Hardman and Macran23). MID: We calculated the distribution-based MID for the EQ-5D VAS using healthy adults in the United Kingdom. The study enrolled 3,378 participants, of whom 57 percent were female. The authors did not report the mean age of the study, but noted that 41 percent of the women and 34 percent of the men were younger than 60 years (Reference Kind, Hardman and Macran23).

3. 3. Brief Pain Inventory (BPI) (Reference Cleeland and Ryan25). (i) BPI summary scores for pain severity and pain interference. Comparator: When authors did not provide normative or control data, we used the mean scores of the healthy control group (n = 20; 55 percent female; mean age 37 years) reported by Schermuly and colleagues (Reference Schermuly, Muller and Muller20). MID: We calculated the distribution-based MID for the pain severity and the pain interference scores using the healthy control group reported by Schermuly and colleagues (Reference Schermuly, Muller and Muller20). (ii) BPI item scores. Comparator: We were unable to find a normative population or a sample of healthy controls to determine the extent of burden. So, we assumed a sample of pain-free individuals who would obtain the score of 0. MID: According to the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials, the MID for patients with chronic pain on the 11-point numeric rating scale is 1 (Reference Dworkin, Turk and Wyrwich26).

4. 4. Short-Form McGill Pain Questionnaire (SF-MPQ) (Reference Melzack27). Comparator: We assumed a sample of disability-free or pain-free individuals who would obtain the full score, 0. MID: We calculated a distribution-based MID for the total score of SF-MPQ using a sample of enzyme-naïve Chinese patients with Fabry Disease from the same family (n = 16; 50 percent female; mean age 34 years) reported by Ro and colleagues (Reference Ro, Chen and Chang28).

5. 5. Fatigue Severity Scale (Reference Krupp, LaRocca, Muir-Nash and Steinberg29). Comparator: When authors did not include normative or control data, we used the mean scores for the healthy control sample reported in Merkies and colleagues (Reference Merkies, Schmitz, Samijn, van der Meche and van Doorn30). MID: We calculated a distribution-based MID for the total score using 113 healthy adults (48 percent female; mean age 54 years) reported by Merkies and colleagues (Reference Merkies, Schmitz, Samijn, van der Meche and van Doorn30).

6. 6. Child Health Assessment Questionnaire (CHAQ) Disability Index (CHAQ DIS), CHAQ VAS Pain, and CHAQ VAS Well-Being (Reference Singh, Athreya, Fries and Goldsmith31). Comparator: When authors did not include normative or control data, we used the mean scores for the healthy control sample reported by Nugent and colleagues (Reference Nugent, Ruperto and Grainger32). MID: We used an anchor-based MID for the CHAQ DIS reported by Dempster and colleagues (Reference Dempster, Porepa, Young and Feldman33), who enrolled a sample of 131 children with inflammatory diseases (e.g., juvenile arthritis, psoriatic arthritis, reactive arthritis, etc.) (69 percent female, mean age: 10 years). We calculated distribution-based MIDs for the CHAQ VAS pain and VAS well-being scores using a sample of healthy children (n = 221; 50 percent female; mean age 10 years) (Reference Merkies, Schmitz, Samijn, van der Meche and van Doorn30). When authors reported CHAQ DIS scores combined with scores from the Health Assessment Questionnaire (HAQ) (CHAQ/HAQ DIS). When the CHAQ/HAQ DIS scores were reported together, we used an anchor-based MID for the HAQ DIS from Kosinski and colleagues (Reference Kosinski, Zhao, Dedhiya, Osterhaus and Jr21), who enrolled 693 Americans, of whom 52 percent were female, and 84 percent were older than 45 years with RA.

7. 7. Rotterdam Handicap Scale (RHS) (Reference Merkies, Schmitz, Van Der Meche, Samijn and Van Doorn34). Comparator: We assumed a sample of disability-free or pain-free individuals that would obtain the full score, specifically 36. This approach is confirmed by Hagemans and colleagues (Reference Hagemans, Laforet and Hop35), who reports that the RHS score for a healthy person would be 36. MID: We calculated a distribution-based MID for the RHS total score using the sample of enzyme-naïve patients with PD (n = 257; 53 percent female; mean age 48 years) reported by Hagemans and colleagues (Reference Hagemans, Laforet and Hop35).

The systematic review has been completed (Reference Johnston, Miller and Agarwal36).