The United States Food and Drug Administration has defined a patient reported outcome (PRO) as “any report of the status of a patient's health condition that comes directly from the patient without interpretation of the patient's response by a clinician or anyone else” (1). Discussions of PROs abound, and many regard the patients’ perspective as crucial in the evaluation of treatment benefits.
Due to the low prevalence of rare diseases, establishing the impact of available treatments is difficult. Given this problem, there are two key reasons that PROs are critical in addressing the possible benefits of interventions for rare diseases. First, surrogate outcomes (laboratory measures, physiological measures, organ size) may not reflect benefits that patients value. In other words, surrogate measures may suggest benefit, but patients may neither feel better, nor live longer.
Second, PROs are sometimes responsive to modest treatment effects in studies of small sample size (Reference Wiebe, Guyatt, Weaver, Matijevic and Sidwell2). Thus, with choice of the right PROs, it could be possible for randomized controlled trials (RCTs) with sample sizes that are feasible in rare diseases (fewer than 100 patients, for instance) to detect small, or at least moderate sized improvements in patients’ physical and emotional function.
The European Organisation for Rare Diseases (EURORDIS) acknowledges the potential role for PROs in rare diseases by recommending in its position paper 2014–20 the development and validation of tools to support PROs as a desirable research initiative. Further evidence for the potential usefulness of PROs in rare diseases comes from the frequency of PRO claims in orphan medicines: frequency of claims seems to be similar to the frequency of PRO claims in all products approved by the European Medicines Agency (EMA) (Reference Arnould, Perrier and Acuadro3).
Although many PROs have shown satisfactory measurement properties including reliability, validity, and responsiveness, there is great concern about risk of bias, especially in clinical trials. This article addresses the obstacles to use and interpretation of PROS in rare diseases. The authors provide insights from their own specialties, including methodological aspects, the significance of generic questionnaires, the view of health technology assessment (HTA), and the patient perspective. Our aim is to contribute to the discussion regarding PRO use in rare diseases and illustrate the valuable integration of PROs into the research for handling rare diseases. Rather than presenting a single perspective of PRO measurement in rare diseases, we highlight differences in perspectives that may arise, for instance, from clinical, patient advocacy, and HTA views of the world.
METHODOLOGICAL ASPECTS (GORDON GUYATT)
Investigators often address treatment effectiveness through measurement of surrogate outcomes. Surrogates are, however, often not closely related to outcomes of importance to patients. They sometimes show apparent benefits when patient-important outcomes fail to improve, and sometimes fail to show improvement when patient-important outcomes do improve. Thus, surrogates are unsatisfactory for measuring impacts that are important to patients. Alternatives include morbidity and mortality. For many questions in rare conditions, however, morbidity and mortality occur too infrequently, and over too long periods of time, to be helpful. Therefore, direct measures of how patients are feeling offer an attractive alternative for addressing treatment effectiveness.
Such direct measures include generic instruments designed for broad populations. In this area, the Short Form 36 is now the universally adopted standard. Alternative generic measures include utility measures such as the EuroQol and Health Utilities Index. These latter measures produce estimates of effect in quality-adjusted life-years (QALYs), typically on a 0 to 1.0 scale in which 0 is death and 1.0 is full health. Utility measures are particularly suitable for economic assessment.
Disease-specific measures provide an alternative to generic measures. Taking the example of lysosomal diseases, one could potentially have a measure that is specific to lysosomal storage diseases (LSDs) in general or to each specific LSD.
It is possible, however, that there is excessive variability even within a particular LSD, making a measure in which each patient answers the same questions unsuitable. Were that true, one could construct a measure that is specific to individual patients. An approach such as Goal Attainment Scaling in which investigators determine what individual patients want to achieve or what they experience as important and measure patients’ function on those items over time may provide a solution to this problem.
These broad measurement strategies available for patient-reported outcomes are variably suited to the purpose of the measurement. One goal is to document the burden of illness using what we call a discriminative instrument. In LSD, for instance, a discriminative instrument could address how great is the magnitude of the burden, and particularly, who in the population is facing a larger, and who is facing a lesser burden?
For a second and quite separate purpose, which is to measure change over time, which can either be improvement or deterioration, we use evaluative instruments. If we want to measure the impact of treatment, we need an instrument that is able to capture change over time.
Considering the following item: “My health does not allow me to make friends like others.” Will it be realistic that an intervention in LSDs helps people sufficiently over a short period of time to allow them to make friends more easily? Perhaps not, particularly because how well one makes friends is not only a function of health, but also of personality and social setting, which are unlikely to be affected by interventions for LSD. That item might, therefore, discriminate well between patients who are doing better and patients who are doing worse, but is unlikely to detect change over time. Were that was true, the item distinguishes between individuals, but does not capture change over time, it would be good for a discriminative purpose but not very useful for measuring the impact of treatment.
In contrast, consider this item: “I feel tired much of the time” and its associated response options from “strongly agree” to “strongly disagree”. Many even apparently health people would agree at least to some extent with this item. Therefore, it might not be a very good item to discriminate between people with more and less severe LSD. On the other hand, an intervention could result in patients moving from being very tired all the time to somewhat less tired, and thus it would work well as an evaluative item in measuring change.
“Validity” represents one key concept in addressing the extent to which an evaluative instrument is working well. Validity is a technical term for the degree to which an instrument is measuring what it is intended to measure. To address the validity of an evaluative instrument we look at the correlations in change, ideally after making a priori predictions. The independent measures that we use for validation might be global ratings of change, ratings by family members, ratings by healthcare providers, and physiologic measures.
The other key characteristic of evaluative instruments is responsiveness, the ability to detect any important change, even if that change is small. If instruments are not responsive, an intervention may make people feel better, but the instruments will fail to detect the change, and investigators will falsely conclude that the intervention is not having an important effect when in fact it is.
A final issue is interpreting the results. If an intervention improves a patient, or a group of patients, by five points, we need to establish whether this is a trivial effect, a small but important effect, or a large effect. One approach to addressing this issue of interpretability focuses on the minimally important difference (MID), the smallest change that patients consider important. Using this approach, investigators follow patients over time, asking them if they have improved or deteriorated. If patients report a small important improvement, that defines the MID. In several questionnaires in the respiratory and heart failure area using a seven-point scale, we found that 0.5 represents the MID.
Unfortunately, the MID is open to misinterpretation. If a patient or a group of patients improve by 0.6, slightly above the MID it does not necessarily mean that everyone benefits? In particular, if the change in a group is less than the MID, 0.3 for instance, it does not necessarily mean that no one benefits and a treatment is useless. Indeed, it is possible that a third of the patients will change by 1.0, two thirds by zero and that will yield a mean difference of 0.3. Such a situation would mean one needs to treat three patients to result in one patient having an important benefit, which in the general medical treatment is a very satisfactory Number Needed to Treat. One can address this interpretation problem by examining the proportion of patients who have improved their score by the MID or greater.
Returning to alternative measurement approaches, generic instruments such as the SF-36 have the advantage of being extremely well validated and allowing comparisons across conditions. They have, however, the disadvantage of being relatively unresponsive. Similarly, utility measures such as the EuroQol are useful for economic analysis, but again are unlikely to be responsive. Specific instruments, on the other hand, do not allow comparisons across conditions, but may well be more responsive. Thus, in choosing instruments for a particular study, it may be wise to use both a generic and specific instruments and, thus, gain the advantages of comparability across populations, data for an economic analysis, and detecting small but important changes.
DISEASE-SPECIFIC QUESTIONAIRES (DEBORAH ELSTEIN)
Health-related quality of life (HRQoL) should be a universal goal for all people across all life stages and it, therefore, behooves healthcare providers to ascertain that HRQoL is optimized also among those with rare genetic disorders to the degree that conditions allow. Heretofore, generic measures have been used widely for assessing impact of disease status and of disease-specific therapies in the rare and often ultra-rare LSDs as in more common morbidities. However, disease-specific therapies PROs and HRQoL measures have been introduced as efficacy end points in clinical trials of new therapies and in long-term surveillance registries of both treated and untreated patients.
There is certainly justification for the use of PROs in assessing the impact of rare diseases, and of treatments for rare diseases, because in the most fundamental sense, patients can and should evaluate the overall quality of their daily lives. On a more simplistic level, however, patients should evaluate the degree to which their personal expectations of improvement or nondeterioration (relative to the untreated state) have been met when they have been compliant with a disease-specific therapeutic intervention.
However, HRQoL responses may not necessarily be solely a composite of the patient's assessment of disease-specific symptoms and signs but may additionally reflect a myriad of other, indirect, environmental influences, not the least of which may be the context in which their lives unfold. The influences include family support, opportunities for education and socialization, financial constraints, co-morbidities, and hopes for the future. Such concerns are often limiting factors for enrollment in clinical trials with new interventions.
In assessing HRQoL in rare disorders, generic PROs may miss the vital nuances that have variable effects on HRQoL. This limitation is particularly concerning in generic tools for LSDs because of their variable disease-specific characteristics: even the most closely related LSDs, such as the three forms of the mucopolysaccharidosis I (MPS) are dissimilar to each other and differences are even greater in comparison to other MPS such as MPS II or MPS VI. A generic HRQoL tool or even an MPS-generic tool would not reflect the disease-specific symptoms of all these patients.
Moreover, beyond the tremendous phenotypic heterogeneity among different LSDs, there is intra-disease phenotypic variability. Most LSDs have a clinically discernible range of lethal infantile forms, juvenile forms, and more long-lived chronic forms correlating with the progression and severity of neurological features. Disease progression, and consequently therapeutic efficacy, may vary in each of the sub-groups in each of the LSDs.
Therapeutic efficacy is indeed variably effective in the different LSDs. For example, the benefits of disease-specific enzyme replacement therapy (ERT) and pharmacological chaperones for Gaucher disease differ dramatically from benefits of these same treatment modalities for Fabry disease. Additionally, the relative safety of the same modality is unequal across the LSDs: for example, the side effects of substrate reduction therapies may outweigh their benefits as an oral option in type 1 Gaucher disease that is non-neuronopathic but benefits may outweigh adverse effects in type 3 Gaucher disease, Niemann-Pick type C disease, or late-onset Tay-Sachs disease where patients have life-threatening neurological involvement but no other effective therapeutic option.
Because of these differences in diseases, treatments, and treatment impact, formats for PROs for each of the LSDs need to be tailored to the condition and the therapy. It is for this reason that the need for a combination of generic and disease-specific HRQoLs for each of the LSDs was recognized soon after the advent of ERTs for LSDs. Nevertheless, the tradition of using generic PROs as primary measures of cost-effectiveness (because of the extraordinarily high per-unit cost of all of the LSD treatments), may be not only limiting but inappropriate.
Although open-ended questionnaires or the use of the narrative format bring to the fore a personalized evaluation of health-related status, they are cumbersome in routine monitoring of patients and even more so in clinical trials.
Thus, an objective systematic approach for assessment of HRQoL in the LSDs and other rare diseases might be the development of a series of measures that have the same basic formulaic presentation containing some of the traditional features of the generic instruments. The PROs would also include the personal and societal context appropriate for all patients as well as some disease-specific aspects that impact HRQoL including the safety and efficacy of treatment where available and/or rate of disease progression based on disease-specific considerations. A final set of questions would be customized for each disease based on the input of experts in each disorder.
In summary, the continued use in LSDs, as in other rare disorders, of generic questionnaires that were developed for health economics purposes may be a disservice to both subjective and objective assessment of the impact of diseases. There are several cogent reasons that a single generic questionnaire is inadequate specifically for all the LSD: (i) LSDs differ widely in their signs and symptoms because each involves deficiency of a different enzyme; (ii) there is unequal availability of therapeutic options, that is, for Gaucher disease there are three enzymatic therapies and two options of substrate reduction, while for most LSDs there is no known treatment; (iii) even where therapies exist in various LSDs, they are not uniformly safe and/or effective. One solution would be to develop a basic stencil for the PROs for LSDs that would include three sections: first, basic quality of life questions; second, disease-specific questions; and third, a section of social-societal concerns. All these questions would be in a user-friendly format with binary responses (yes/no) that would be adequate for routine monitoring and also for clinical trials. Some of us are currently in the process of validating a PRO along these lines for Gaucher disease with the support of the Shire company (Zug, Switzerland).
HTA ASPECTS (ALRIC RÜTHER)
Especially in the case of pharmaceuticals there are two main pathways for decision making based on HTA in Europe: reimbursement based mainly on cost effectiveness and price negotiation based mainly on clinical benefit. Representative of the first is, for example, the National Institute for Health and Care Excellence (NICE) in UK, of the second Haute Autorité de Santé (HAS) in France or the Institute for Quality and Efficiency in Health care (IQWiG) in Germany.
NICE requires QALYs for their cost-effectiveness analyses. QALYs require HRQoL data. Therefore, NICE takes PROs into account: “the measurement of changes in HRQoL should be reported directly from patients and the utility of these changes should be based on public preferences using a choice-based method. The EQ-5D is the preferred measure of health-related quality of life in adults” (4). NICE has lots of experience, but critics point out a lack of concordance between utility and subjective wellbeing using disease-specific Qol measurement (Reference Richardson, Iezzi, Khan and Maxwell5). Moreover, the mapping exercise NICE often needs for assessing the results of different QoL measurements is not an easy one, and may sometimes be highly problematic (Reference Longworth and Rowen6).
For HAS, HRQoL assessment may be as important as assessment of morbidity and mortality in fatal or very severe diseases (i.e., oncology, asthma) and additionally in nonsevere but HRQoL impacting diseases like arthritis and migraine. HRQoL may also be considered as a secondary end point. When measuring HRQoL, investigators should use a validated instrument and perform agreed standard analyses (e.g., missing data, multiple testing, clinical significance of changes) (Reference Pavlovic7). An HTA-specific approach similar to that of EUnetHTA would use both a generic and a disease specific multidimensional HRQoL questionnaire for rapid relative effectiveness assessment and a generic questionnaire for cost-utility (8).
For IQWiG, PROs can cover a variety of dimensions of potential benefit including disease symptoms. The instruments must, however, be suitable for use in clinical trials. Demonstrating an effect usually follows the same principles as for other outcomes. Following the criteria of evidence-based medicine, this means for example, that RCTs are best suited. PROs are subjective by nature and are potentially at high risk of bias (9).
Regarding small populations typically seen in rare diseases, IQWiG sees no convincing argument to deviate in principle from the hierarchy of evidence levels. Here, the right of the patient to the most reliable information from the optimal study design, RCTs, takes precedence. Whatever approach investigators take should be specified a priori and well justified (9).
In conclusion, HTA organizations take PROs into consideration as valuable information. Uncertainties of evidence must, however, be dealt with clearly and transparently. PROs could provide information for patient relevant end points, but cannot be seen as synonym for them.
PATIENTS PERSPECTIVE (DURHANE WONG-RIEGER)
In many cases, robust RCTs are challenging. Developing validated outcomes measures for many rare diseases may also be problematic.
Consider thalassemia, a rare blood disorder treated by regular transfusions, which results in iron overload. The excess iron can lead to liver and heart failure as well as life-threatening blood clots. Treatment, until fairly recently, was an iron chelator infused by means of small pump over 8 to 12 hours. The burden resulted in high levels of nonadherence.
The introduction of a once-daily oral chelator was celebrated by patients but the high cost led to an unfavorable “incremental cost-effectiveness ratio” and an HTA recommendation that the oral drug be available only for patients for whom infusion was unsuccessful or contraindicated. The impact on patients, however, is not so predictable or equitable.
Consider three patient profiles: Charles, age 24, has suffered a heart attack and liver damage attributed to nonadherence; he is approved for the oral therapy. Gloria, age 27, has experienced reactions to infusions severe enough to lead to her quitting her job as a dental hygienist; she is approved for the oral therapy. Lisa at 18 years of age has just started university; she continues to be highly adherent to her infusion therapy despite the challenges of living in a dormitory and interference with her social life; she is denied the oral drug and complains that “patients are rewarded for noncompliance and punished for adherence.”
One problem is that the standard QoL scales could not justify the higher cost of an oral therapy for those patients adherent to the infusion therapy. In the clinical trials, patients who adhered to the standard infusion therapy had high QoL scores as measured by the traditional SF-36 and EuroQual. A Satisfaction with Chelation Therapy instrument, which measured only chelation, was also nondiscriminatory.
In response, a validated transfusion-related quality of life (TRQL) instrument was developed to measure physical, emotional, social, and role functioning. This has proven to be a very useful tool for healthcare professionals and patients to talk about the impact of alternative chelation therapies (Reference Klaassen, Barrowman and Merelles-Pulcini10).
Unlike thalassemia, LSDs may not have sufficient patients to develop a validated PRO. For instance, Hunter's syndrome is a very rare LSD that presents in two forms: a mild form involving physical symptoms only and a more severe form including neurological symptoms. ERT was approved for Hunter's syndrome based on improvements in physical outcomes only (6-minute walk test and decrease in symptoms). Moreover, because the drug molecule was believed too large to cross the blood-brain barrier, reimbursement was granted by the Ontario Drug Plan only to patients who had no neurological impairment.
Consider Lee, a 7-year-old with Hunter's syndrome. He was diagnosed at age 2 with moderately severe disease and no neurocognitive symptoms, and it was uncertain whether he would develop them. He was started on therapy at age 5. His mobility, energy, and liver size all improved. However, at age 7, concerns were raised about potential cognitive deterioration. The parents were told Lee may have to come off therapy, because it was reimbursed only for patients without the severe form. The mother was devastated. A generic PRO was unable to capture her perspective that, “prior to therapy, Lee was barely able to walk and run but was now riding a bicycle and in school”.
Recently, a Hunter Syndrome-Functional Outcome for Clinical Understanding instrument has been developed, and this may include patient-defined outcomes and values which could be used to determine starting criteria and monitoring response (Reference Raluy-Callado, Chen, Whiteman, Fang and Wiklund11). The challenges are to validate the instrument (despite the small patient population) and to convince HTA bodies and payers to accept the findings.
CONCLUSION
This article presents different perspectives in dealing with and thinking about PROs. PROs are already playing an important role in assessing treatment benefit. Small sample sizes will always be inadequate to allow inferences regarding mortality or major morbid events, and surrogate outcomes will never leave us confident that treatment effects represent important benefit in patients’ lives. PROs can address both issues: they may be responsive to modest treatment effects, and they represent aspects of patients’ lives that they surely value. Nevertheless there remain challenges in the development and use of PROs, with contrasting concerns on the one hand regarding lack of responsiveness of generic measures and limited ability to capture all aspects that patients consider important, and on the other hand concern about optimal study design and risk of bias.
CONFLICTS OF INTEREST
No conflicts of interest appear.
