Many new genomic- and genetic related findings have lately been published. Health policy challenges arise when the promising new technology is in its early development phase and certain stakeholders find reason to speed up implementation in clinical practice. Nowadays, Technology Assessment (TA) is a frequently used evaluation approach to enable decisions on coverage and reimbursement of new technologies (Reference Hutton, Trueman and Henshall9). However, the point at which a new technology should be assessed remains a contentious issue (Reference Mowatt, Bower and Brebner17). Broad clinical implementation and performing a TA for policy decisions may be premature in the absence of prospective data of the actual benefits. However, if we wait to perform a TA, it might very well be that worthwhile technology is withheld from the public (Reference Ioannidis11). Coverage decisions usually have to be made at a time when the data on all the relevant variables and adequate comparisons are not available from high-quality studies. Coverage with Evidence Development (CED) is one of several policy options that have been posited to overcome the problems associated with making coverage decisions under uncertainty (Reference Hutton, Trueman and Henshall9).
In the Netherlands, the Dutch Health Care Insurance Board (DHCIB) has experimented with a program of controlled introduction of promising innovations in an early stage of development from 2004 onward. Our case, the use of the 70-gene signature, was one of the three technologies to be studied. At present, the DHCIB and the ministry of Health Care discussing the most appropriate way of stimulating innovations, for instance through a CED program.
In 2002, researchers at the Netherlands Cancer Institute (NKI, Amsterdam, The Netherlands) identified a new genomic technology: the 70-gene prognosis signature (MammaPrint®), using microarray analysis for lymph node-negative breast cancer patients (Reference van't Veer, Dai and van de Vijver26) (Note: The 70-gene prognosis signature, performed by Agendia, Amsterdam; MammaPrint® Agendia's ‘Mammaprint diagnostic service’ is cleared by the Food and Drug Administration as an IVDMIA medical device and is ISO-17025 accredited, using a custom designed array chip “Mammaprint®”). This signature was presumed to outperform currently used clinical factors in predicting disease outcome and overall survival. A patients’ prognosis is usually based on clinical and pathological factors, such as age, nodal status, tumor diameter, and histological grade. However, these factors do not accurately predict the exact clinical behavior of breast tumors, and, therefore, patients can be undertreated or especially overtreated. It is generally agreed that patients with a poor prognosis or clinical high risk for metastasis will benefit from adjuvant systemic treatment (7). However, because these treatments can have severe side effects, a careful selection of those high-risk patients is very important. Using the 70-gene signature, the selection of patients that will benefit most from adjuvant systemic treatment could be more accurate. The signature has meanwhile been validated in three retrospective patient series (Reference Bueno-de-Mesquita, Linn and Keijzer3;Reference Buyse, Loi and van't Veer5;Reference van de Vijver, He and 27). It would take at least 8–10 years to bring the signature into clinical practice, by means of the usual path of prospective trials. Therefore, it was decided that a controlled introduction would be appropriate to evaluate this technology. The DHCIB sponsored this controlled introduction study, along with a technology assessment to ensure and improve the quality of implementation (Reference Douma, Karsenberg and Hummel6). The MicroarRAy PrognoSTics in Breast CancER (acronym RASTER) study was a clinical, multicenter, prospective observational study (a list of the participants in that study can be found at www.journals.cambridge.org/thc). The main aim was to analyze the differences between adjuvant systemic treatment advice for breast cancer based on the Dutch CBO guidelines (13) and the prognosis signature, taking into account patients’ preferences (Reference Bueno-de-Mesquita, van Harten and Retel4). We chose to support the controlled introduction of the 70-gene signature with a comprehensive technology assessment, which takes technology dynamics into account, and decided to perform a Constructive Technology Assessment (CTA).
CTA is based on the idea that, during the course of technology development, choices are constantly being made about the form, the function, and the use of that technology (Reference Schot23). CTA has developed from assessing the impact of a new technology to a broader approach, including the analysis of design, development, and implementation of that new technology (Reference Schot and Rip22). CTA is related to Health Technology Assessment (HTA), which predominantly suggests a cost-effectiveness analysis (CEA). HTA generally starts after the technology is stabilized and proved to be valid in clinical trials. It commonly presumes a “ceteris paribus” (static) situation, whereas it has become evident that environment and technology are often dynamic and mutually influencing each other. In addition to “studying” changes, “influencing” changes is sometimes necessary to improve effectiveness. During this time many changes in available treatments can occur, which results in that HTA subsequently answers—at least partly—outdated questions. CTA can be used as a complementary approach to HTA, especially for the early and dynamic introduction of new technologies in a controlled way (Reference Douma, Karsenberg and Hummel6). Only limited publications are available describing the application of CTA in health care (Reference Douma, Karsenberg and Hummel6;Reference Retel, Hummel and van Harten20). At different phases of CTA, the focus will shift to the aspects most likely to change during the introduction of these new technologies. In this study the mixed method approach of the CTA covers aspects of quality of care following the Institute of Medicine (IOM) (10) and uses diffusion scenarios to monitor the dynamics. Diffusion scenarios, which are commonly applied in industry to anticipate on their strategies concerning future development, have been adapted in this study.
Our aim was to perform a CTA on the controlled introduction of the 70-gene prognosis signature in the participating community hospitals to anticipate in modern decision- and policy making. The following substudies were performed: (i) Clinical effectiveness: studied in the clinical feasibility study, the MicroarRAy PrognoSTics in Breast CancER (acronym RASTER) study, and more detailed reported by Bueno-de-Mesquita et al., Reference Bueno-de-Mesquita, van Harten and Retel2007 (Reference Bueno-de-Mesquita, van Harten and Retel4). The most important results of the clinical implementation study were: out of 812 accrued patients, 427 prognosis signatures were assessed, 51 percent of the patients (219/427) had a good and 49 percent (208/427) a poor prognosis signature. The prognosis signature was discordant with risk assessment based on the Dutch CBO guidelines in 30 percent of the cases, which resulted in change of treatment in 54 percent of the discordant patients (see Figure 1a–c). Discordant cases are patients who are clinically low risk and according to the signature high risk or clinically high risk and according to the signature low risk.
(a) Adjuvant chemotherapy based either on prognosis signature or clinical risk (based on Dutch CBO guidelines). (b) Adjuvant endocrine treatment based either on prognosis signature or clinical risk (based on Dutch CBO guidelines). (c) Adjuvant systemic treatment (chemotherapy and/or endocrine therapy) based either on prognosis signature or clinical risk (based on Dutch CBO guidelines). RASTER numbers for a–c are from Bueno-de-Mesquita et al. (Reference Bueno-de-Mesquita, van Harten and Retel4).
In this study we report on (ii) Organizational efficiency: What are the changes to the actual care provision processes, logistics, and teamwork, and which organizational aspects influence the implementation? (iii) Patient centeredness: Analyzing understanding, psychological impact of the test results, satisfaction, and decision-making process. (iv)Diffusion scenarios: Are diffusion scenarios, commonly used in industry, applicable for new technologies in health care? And how can we use these diffusion scenarios to guide the implementation process in this study?
METHODS
The CTA study was part of the clinical RASTER study, using the same procedures and thus the same hospital team-members and (part of the) patient population (Reference Bueno-de-Mesquita, van Harten and Retel4). The Institutional Review Board of the Netherlands Cancer Institute approved this side-study.
Organizational Efficiency: Logistics and Teamwork
In the participating hospitals, semistructured baseline and postsurvey interviews were conducted, involving all relevant breast cancer care team members. The postsurvey was conducted at a minimum of 6 months after the first included patient. Information was gathered regarding changes of the total clinical and pathological processes, and processes of multidisciplinary meetings and related patient contacts. Finally, the team members were questioned about their expectations regarding the role this signature would play in future clinical practice.
Patient Centeredness
Based on a pilot series of structured interviews, a questionnaire was constructed and was sent to patients from three of the sixteen participating hospitals at 4 weeks after surgery (Supplementary Table 1, which can be found at www.journals.cambridge.org/thc). At that moment, patients had received the results of the pathological report, the prognosis signature outcome, and the final adjuvant systemic treatment advice. The main topics were as follows: was the information about the prognosis signature and its consequences clear to the women and what was the impact of the prognosis signature outcome on these women? This was measured according to the following parameters. (i) Knowledge questions to assess the insight of the patients in the consequences of the 70-gene prognosis signature; (ii) Perception of satisfaction regarding the whole trajectory, informational process of the prognosis signature, receiving the outcomes, and the treatment decision; (iii) Psychological impact, conducted by a questionnaire (developed by Lynch et al. and adapted for the Dutch population by Bleiker et al.) (Reference Bleiker, Hendriks and Otten1;Reference Lynch, Lemon and Durham15), was used to assess the respondents’ emotional reaction to the test results, also called “negative affects,” and the Cancer Worries-scale developed by Lerman et al. (Reference Lerman, Seay and Balshem14) which assessed the amount of worries the women had after receiving the 70-gene prognosis signature. Calculations were done with SPSS (version 15.0), using univariable analysis, factor analysis, and analysis of variance.
Diffusion Scenarios
Scenarios can be used to monitor the implementation process through the various diffusion phases and can support and identify the need for evaluation or even interfere through formal decision making (Reference Douma, Karsenberg and Hummel6). The method used to describe scenarios is based on the Royal Dutch Shell approach, using a most likely course of development with “There Is No Alternative” (TINA) elements and alternative course projections represented by “what-if” deviations. A baseline description was drafted, regarding the consensus of expert opinions. It was written before the prognosis signature was introduced in the Netherlands (mid-2004), using the timeline of diffusion phases as described by Rogers’ diffusion theory, Reference Bueno-de-Mesquita, van Harten and Retel2003 (Reference Rogers21). In the innovation phase, the prognosis signature technique is developed and the first organizations adopt (introduce) the technology in their daily practice, in this phase the presence of a champion (an opinion leader) is necessary. The early adoption phase describes the implementation a priori in 10–15 hospitals. The early majority phase describes the implementation in other participating hospitals that are relying on opinion leaders and well established logistics. The late majority is conservative and waits until there is no further debate on the validity and clinical value of the test and the logistics are further improved. A second scenario was drafted based on the first experiences (mid-2005).
RESULTS
Organizational Efficiency: Logistics and Teamwork
Baseline and postsurveys were conducted in fifteen of the sixteen participating hospitals in the RASTER study (see Table 1). All hospitals succeeded in implementing the required tumor sampling logistics. The duration of the implementation, measured from consent to participate till first patient inclusion, varied from 0.2 to 9.4 months (median 1.2) (Reference Bueno-de-Mesquita, van Harten and Retel4). The two outliers (4.3 and 9.4 months) especially had start-up problems in the pathology process. The change in routine work-up for tissue handling (fresh frozen tissue versus paraffin embedding) and the onsite availability of the pathologist were most difficult to achieve. However, if those logistics were in place, no other major problems appeared. The time between surgery and start of radiotherapy or adjuvant systemic treatment did not change as a result of the new technology in any of the hospitals. In the beginning, the explanation of both the nature of the prognosis signature and the study design to the patients was time-consuming (reported in thirteen hospitals), but once accustomed to the procedure, consultation times returned to normal. As the results could be either concordant or discordant with existing clinical guidelines, oncologists had to be careful concerning the moment and manner of giving the results of both the tests to the patient. Because of the longer waiting time (approximately 10–14 days for execution of the signature and the nodal status), discordant patients were either discussed twice in the multidisciplinary team, or the medical oncologist took a final decision as soon as both were available. The overall trend was to initially follow the pathology report and to communicate this with the patient, stating that the treatment advice could be changed based on the prognosis signature result. Six hospitals indicated to make the treatment decision based only on the pathology report, because they questioned the value of the prognosis signature considering lack of validation studies available at that time. However, of the total number of discordant patients (n = 128 in the RASTER study), the decision to use adjuvant treatment compared with the CBO guidelines was changed in 54 percent of these patients (Reference Bueno-de-Mesquita, van Harten and Retel4). This resulted in an additional increase of 1 percent of patients who were advised chemotherapy, 9 percent of patients who were advised endocrine treatment and 2 percent of patients who were advised both (Reference Bueno-de-Mesquita, van Harten and Retel4). Clinicians and patients seemed to base their decision on the more unfavorable predictor, regardless whether this was the genomic or clinical (see Figure 1a–c).
All interviewed physicians expect that the signature will eventually become part of future regular diagnostics. Some expect the signature to be performed in all patients; others considered it as complementary parameter especially in difficult cases. In general, the physicians rated the addition of the 70-gene signature as beneficial for patient management; however, several medical oncologists tended to look for more confirmative data concerning the validity of the signature.
Patient Centeredness
In total, twenty-nine interviews and forty-eight questionnaires were analyzed, n = 77 (response rate of the questionnaires was 78 percent). The mean age of the responders was 48 years (range, 27–59 years), which did not differ from the total RASTER population, but the distribution of the risk groups were different (more concordant low-risk patients). The results from the knowledge test are presented in Figure 2 and were not different in the three hospitals. Important issues were the predictive accuracy of the test (87 percent wrong answers) and the consequences of the test (66 percent wrong answers). Significant differences (p = .001) were found between the different risk groups for emotional reactions after receiving the 70-gene signature. Women with discordant clinical low/high risk-signature and clinical high risk/no signature (no signature due to failure in process) had the highest negative affect-scores (n = 77).
Results of the knowledge questions (n = 77).
Remarkably, women with a clinical high/good signature scored almost the same as women with clinical low/good signature (see Figure 3).
Negative effects: respondents’ psychological reaction to the 70-gene prognosis signature results, received after the pathological test results (CBO). Higher scores mean more negative feelings experienced by the patients (n = 74).
The scores of “thought about chances of getting cancer again influencing the mood” on the Cancer Worries scale (n = 77) (Reference Lerman, Seay and Balshem14) were significantly different (p = .01) per risk-group: 43 percent of patients with clinical low/poor signature and 29 percent clinical high/no signature often worried about getting a recurrence, compared with 0 percent of the patients with clinical high/good signature, 20 percent clinical low/no signature, 13 percent clinical high/poor signature and 3 percent clinical low/good signature. This was consistent with the Lynch scale.
The satisfaction about receiving the 70-gene signature per risk-group was 76 percent. Six of seventy patients (8.6 percent) were very dissatisfied, four of those patients had a discordant clinical low/high risk-signature, two (no discordant patients) were dissatisfied about the way the result of the 70-gene signature was communicated. Eleven patients had a neutral opinion. The overall satisfaction regarding the total trajectory, from diagnosis to the time of interviewing, around 2 months after surgery, was 82 percent (n = 77).
Diffusion Scenarios
Two rounds of scenarios were written, taking various socio-dynamic interactions into account. The original scenario was written in 2004 and revised mid-2005, using professional feedback. The initial expectation among the direct involved researchers and professionals was that less adjuvant chemotherapy would be needed compared with guideline based treatment and that the impressive potential of the test would lead to swift diffusion (Reference Goldhirsch, Glick and Gelber8). The current Dutch CBO guidelines, however, proved to be more restrictive in the prescription of adjuvant systemic treatment, compared with the St. Gallen guidelines on which the first analysis was based. It became apparent that the signature in combination with the CBO guidelines (with the physicians tending to follow the highest risk) led to more chemotherapy prescription in the RASTER study, instead of less. Although an unexpected result, it might lead to improved selection of patients and ultimately, an improved survival outcome (Reference Bueno-de-Mesquita, van Harten and Retel4).
A second important issue was the “what-if” deviation that suggested that the complex bio-informatics used to select the relevant genes, was incomprehensible for the average clinician. As a consequence, if a discussion would start concerning the validity an expectative attitude might be the result, leading to a prolonged early adoption phase. Although not considered very likely at the time of starting the study, this proved to be reality especially in Europe (see Figure 4).
Results of scenario description, diffusion stages source Rogers, Reference Rogers2003 (Reference Rogers21).
DISCUSSION
This study evaluates the methodology of CTA as a means to guide the controlled early implementation of a promising technology and its possible use for coverage decisions: the 70-gene prognosis signature in the treatment of node-negative breast cancer patients. An important goal of CTA is to inform policy makers in an early stage about possible advantages or disadvantages of new developments and, ultimately, to aid a decision on usage and coverage.
The logistics necessary for profiling was complex but successfully implemented in all participating hospitals. Changes in the pathology process and multidisciplinary decision making on treatment advice particularly influenced the duration of the implementation (median 1.2 months). However, physicians rated the addition of the 70-gene signature as beneficial for patient management. The patient interviews and questionnaires (n = 77) showed that, regarding the level of knowledge about the (consequences of the) 70-gene signature, there is room for improvement for the patient information. The impact on patients seems to depend on the nature of the test results and the way these were communicated to the patient. Because the women received their results in succession (first the clinical risk assessment, followed by the signature), a “framing effect” could have been realized. The “framing theory” suggests that the way content is presented influences the opinion people develop (Reference Tversky and Kahneman25). The “frame,” a low clinical risk result, followed by a poor signature result causes consequently more negative affects. To reduce a possible framing effect, we recommend that physicians communicate all diagnostic results in one appointment after surgery.
The scenarios, especially the “what-if” deviations proved relevant to picture the possible future developments; in a further round these are expected to be useful to specify parameters in planned cost-effectiveness modeling.
The selection of participating hospitals was not at random. In agreement with the DHCIB, regional/urban and size differences were taken into account when selecting hospitals interested in participating. As a consequence, all were probably early adaptors and willing to put effort in the implementation process, which could have been negatively influenced by random selection. Other diffusion groups might not have a comparable positive attitude toward spending money or efforts in implementing the test. The amount of patient questionnaires was too small to conduct extensive statistical analysis, though it may be large enough to give an exploratory insight of the impact of the prognosis signature, and this will be elaborated in the continuation of the CTA. The distribution per risk-group in this part of the study was not equal to the total RASTER population. Because more questionnaires were returned by concordant low risk patients, these might be more inclined toward responding or the present results might depict a too positive situation. The DHCIB was of the opinion that a CEA was not yet relevant in the very early phase of the study, because the development and diffusion of the signature was not sufficiently advanced. However, the results of the CTA led to a positive decision on performing a CEA and a discussion on the possibility of provisional coverage.
There are several remaining issues for further research. First, patient-related aspects that appeared to be relevant or significant in this study, such as quality of life and knowledge of the 70-gene signature, have to be elaborated. Second, a third round of scenario drafting is planned for mid-2008, in a formal set-up with opinions to be obtained from international acknowledge experts. Third, ethical and juridical aspects will be studied, involving patients’ rights concerning future diagnostic use of banked tissue. Finally, a model based CEA will be performed, using several scenario deviations as input to calculate expected costs and outcomes.
The introduction of the 70-gene signature had and will have several clinical implications. The prognosis signature resulted in 30 percent discordant cases compared with the Dutch CBO guidelines, whereas using the United States-based Adjuvant! Online Software resulted in 38 percent discordance (Reference Olivotto, Bajdik and Ravdin18;Reference Ravdin, Siminoff and Davis19). Thus, the use of this prognosis signature, for example in the United States, could lead to greater reduction of adjuvant systemic treatment compared with the present Dutch situation, where the guidelines were more restrictive in prescription of adjuvant systemic treatment. However, in the concept CBO guidelines of 2008 (12), the criteria for adjuvant systemic treatment will be less restrictive, which can also result in greater reduction of chemotherapy in the Netherlands. In the United States, the 70-gene prognosis signature is meanwhile FDA approved, based on the available validation studies. Although officially accepted in the United States, basing a possible catalogue decisions just on retrospective validation series caused serious debate in the Netherlands. Countries thus can have different implementation and diffusion patterns, possibly related to their attitude toward technology innovation. Consensus among opinion leaders on the value of this type of prognostics appears to be essential for further diffusion. The validity discussion in Europe initiated a prospective randomized phase III clinical trial, the MINDACT-trial (Microarray In Node-negative Disease may Avoid ChemoTherapy) (Reference Bogaerts, Cardoso and Buyse2;Reference Mook, Van't Veer and Rutgers16) The MINDACT-trial has, however, a very complex design and organization, and feasibility and compliance might prove to be issues in its execution. The CTA will be continued alongside the MINDACT-trial as this study produced several aspects which need further attention.
Clinicians have a tendency to prefer traditional “ceteris paribus” HTA designs and to challenge the CTA with its broad approach and acknowledgement of dynamic aspects of technology diffusion. Intensive discussions with clinicians can, therefore, be anticipated. Furthermore, the complexity of a broad CTA using a mixed method design demands a lot of effort, organization, costs and knowledge on different areas such as psychology, economics and medical science (Reference Douma, Karsenberg and Hummel6). To achieve a manageable design, it is important to select the most relevant aspects to be researched, which again demands a thorough discussion. Furthermore, finding a balance between broadness and depth will inevitably play a role in publishing CTA results.
Logistics and Teamwork as an Aspect of Efficiency, per Hospital (n = 15)a
aInclusions of patients/numbers of signatures performed. Duration of implementation in months, calculated from Review Board Approval until the first included patient. Prior tissue handling: tumor tissue storage before start of the RASTER study, based on paraffin (formalin) or fresh frozen (dry). Pathology lab inside or outside the hospital. Number of participating team members in the RASTER study. The result of the 70-gene signature part of the adjuvant treatment advice. Disciplinary eventually decided on adjuvant treatment: MDM, multidisciplinary meetings; onc, medical oncologist; surg, surgeon.
It proved that the CTA method is suitable for evaluation of this type of technology, and we suggest that it can be used as a tool for early stage coverage decisions. Especially in case of a CED-program, due to the comprehensive evaluation with its mixed method approach, CTA can be helpful in decision making (Reference Tunis and Chalkidou24). We, therefore, assume that it is appropriate for evaluation of other complex technologies, especially during the early controlled introduction in a dynamic environment. It can be expected that a score of new (personalized) diagnostic tests based on genomics, proteomics, and/or nanotechnology will be developed. The complex analytical methods, the design of the various elements of technologies and the possible costs make CTA a logical approach in early stages of development and diffusion of new promising techniques.
CONTACT INFORMATION
Valesca P. Retèl, MSc (v.retel@nki.nl), PhD Student, Department of Psychosocial Research and Epidemiology, Jolien M. Bueno-de-Mesquita, MD (j.bueno@nki.nl), PhD Student, Department of Pathology, Netherlands Cancer Institute-Antoni van Leeuwenhoek Hospital, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
Marjan J. M. Hummel, PhD (j.m.hummel@bbt.utwente.nl), Postdoctoral Student, School of Governance and Management, University of Twente, PO Box 217, 7500 AE Enschede, The Netherlands
Marc J. van de Vijver, MD (m.vd.vijver@nki.nl), Pathologist, Department of Pathology, Academic Medical Center, Postbus 22660, 1100 DD Amsterdam, The Netherlands
Kirsten F. L. Douma, MSc (k.douma@nki.nl), PhD Student, Department of Psychosocial Research and Epidemiology, Netherlands Cancer Institute–Antoni van Leeuwenhoek Hospital, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
Kim Karsenberg, MSc (k.karsenberg@descentrum.nl), Department of Communication, Stichting DES-centrum, Wilhelminapark 25, 3581 NE Utrecht, The Netherlands
Frits S. A. M. van Dam, PhD (f.v.dam@nki.nl), Professor, Department of Psychosocial Research and Epidemiology, Netherlands Cancer Institute-Antoni van Leeuwenhoek Hospital, Plesmanlaan 121, 1066 CX Amsterdam, the Netherlands
Cees van Krimpen, MD, PhD (ckrimpen@kg.nl), Pathologist, Department of Pathology, Kennemer Gasthuis Haarlem, Postbus 417, 2000 AK Haarlem, The Netherlands
Frank E. Bellot, MD (f.bellot@spaarneziekenhuis.nl), Pathologist, Department of Pathology, Spaarne Hospital Hoofddorp, Spaarnepoort 1, 2134 TM Hoofddorp, The Netherlands
Rudi M. H. Roumen, MD, PhD (r.roumen@mmc.nl), Surgeon, Department of Surgery, Máxima Medical Centre, PO Box 7777, 5500 MB Veldhoven, The Netherlands
Sabine C. Linn, MD, PhD (s.linn@nki.nl), Department of Medical Oncology, Wim H. van Harten, MD, PhD (w.v.harten@nki.nl), Professor, Department of Psychosocial Research and Epidemiology, Netherlands Cancer Institute–Antoni van Leeuwenhoek Hospital, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
