Disease management focuses on providing well-integrated care that is intended to maximize quality of care while minimizing costs across the continuum of care settings (8;55;58). To assess to what extent these goals are reached, principles of health technology assessment (HTA) are applied. The most important characteristic of HTA is that it focuses on effectiveness and at least one other aspect of health care (for example costs, organizational aspects, or quality of care), with the aim to provide input to decision making in policy and practice (5). This statement suggests that several different types of indicators have to be measured to provide meaningful, precise, and timely information for various stakeholders. The question that is dealt with in this study is, “Are we measuring what matters when assessing disease management programs for purposes of decision-making?”
To answer this question, an assessment must be made first, to see if there actually is a fit between the general aim of a disease management program (e.g., improving quality and/or increasing efficiency of care) and the type(s) of indicators used to evaluate the program. Previous studies already suggested that a mixture of indicator types—as Donabedian stated, structure, process, and outcome indicators (14)—may be more useful in assessing quality of care from various stakeholder perspectives, and that attention should be paid to further conceptualization and operationalization of quality of care indicators (3).
When assessing the quality of care as delivered within a disease management program, one has to realize that a wide variation of programs exists and that these programs have been implemented within differing organizational structures in daily practice. This heterogeneity within disease management programs—even when these programs focus on the same disease—not only hinders the understanding of what disease management actually is, but also hinders the uniformity in evaluation of the programs and, consequently, the comparability of study results (25). The Disease Management Association of America (DMAA) formulated a definition of disease management that is now widely accepted, but they also acknowledge that there are many differences between programs in terms of their comprehensiveness and focus. Therefore, they distinguish between the so-called full-service disease management programs, which comprise all elements of disease management, and the disease management support services, which are less comprehensive (13). Because most disease management programs are introduced gradually, very few full-service disease management programs exist, let alone that they are evaluated rigorously. From the current scientific literature, in general, three types of disease management programs can be distinguished, based on the extent to which they consist of organizational, professional, and/or educational interventions (60). The choice for specific indicators is expected to be influenced by the type of program that is evaluated.
What this literature review adds to the existing knowledge is an overview of indicators currently used to assess disease management programs. Moreover, a framework is provided regarding which sets of indicators could be used when taking the aims and types of disease management programs into account. This framework should lead to a minimum data set for HTA of disease management and is intended to increase generalizability and comparability of disease management programs. Within this framework, attention is being paid to the fact that a wide range of programs is implemented worldwide within different health care systems. Using the framework will improve the comparability of results of HTA of disease management and increases the usefulness of study results for policy and practice. Because asthma, chronic obstructive pulmonary disease (COPD) and diabetes are expected to be among the most common diseases for which disease management programs have been developed, the focus of this review will be on these diseases (8).
METHODS
Systematic Literature Review
We reviewed the scientific literature to identify studies that evaluated disease management programs designed for patients with asthma, COPD, or diabetes. MEDLINE and Cochrane databases were searched for English, Dutch, and German language articles published between January 1987 and September 2004 using the following Medical Subject Headings: patient-care team, patient-care planning, primary nursing care, case management, critical pathways, primary health care, continuity of patient care, guidelines, practice guideline, disease management, comprehensive health care, and ambulatory care. Furthermore, the terms disease state management, disease management, integrated care, coordinated care, and shared care in combination with either asthma, COPD, or diabetes mellitus were searched alphanumerically as title and/or abstract words.
Titles of articles and abstracts extracted by the search were reviewed for relevance, and if found to be so, the full-text article was retrieved. Reference lists of the included articles were also reviewed and provided additional relevant citations. To be included in the review, studies had to contain a description of the disease management program, explicitly mention the aim(s) of the program, and provide a description of indicators used. Moreover, disease management programs should be aimed at asthma, COPD, and/or diabetes mellitus.
Disease management was defined as a system of coordinated health care interventions and communications for people with conditions in which self-care efforts are significant (13). To determine whether a program could be considered as a disease management program, we searched for key words referring to the six disease management components defined by the DMAA. The key words were: population identification process, evidence-based practice guidelines, quality improvement, collaborative practice model, transfer of care, health care teams, patient and provider education, self-management, process and outcomes measurement, management, routine reporting, feedback loops, and benchmarking.
The definition of disease management was very strictly applied to reduce the heterogeneity of the studies as much as possible. Therefore, articles on risk reduction programs, screening programs, and rehabilitation programs were excluded, because these reports do not encompass the whole continuum of care. Likewise, evaluations of single components of disease management (e.g., pharmacological treatment, smoking cessation, nutritional advice, self-management, patient education, rehabilitation programs) were excluded. Also, pediatric programs were excluded, because in the evaluation of interventions tailored to children, specific outcome measures are needed (e.g., from both children and their parents), further increasing heterogeneity in measurement as well as in the programs themselves. Two reviewers (H.J.M.V./L.M.G.S.) independently assessed the methodological quality of studies, using the HTA-DM instrument. This instrument has been developed and validated for the purpose of assessing methodological quality of disease management evaluation studies (49). Methodological quality is scored between 0 and 100 points, with a score <50 points indicating inferior quality and ≥70 points indicating good quality. The HTA-DM instrument proved to have the best clinimetric characteristics when applied to disease management evaluation studies compared with three other commonly used instruments (2;48;54). Methodological quality was not an exclusion criterion, because we did not intend to use the value of the outcomes as such. However, based on methodological quality of studies, process and outcome measures were selected to enter the framework.
Data abstraction was performed independently by the two reviewers, using a standardized form. Data collected are: type of disease management program; general aim(s) of the program; and structure, process, and outcome indicators used to evaluate the program. Any discrepancies between reviewers were resolved by discussion or referred to the fifth author (C.S.).
Classification of the Disease Management Programs
A disease management program was classified as educational, when the interventions were aimed at self-management abilities or disease-specific knowledge of patients; professional, when the interventions focused on changing the performance of care providers or improving their adherence to guidelines; or organizational, when the interventions interfered in the structure of the primary care process or in any way aimed to improve the continuity of care (20;60).
Classification of Indicators
According to the paradigm of Donabedian, structure indicators are defined as the physical and organizational properties of the settings in which care is provided. The term process encompasses all that is done to and for a population of consumers and patients, such as preventive services and measures, diagnosis, treatment, and palliative care. Outcomes are the end results of care, that is, the changes in patient or population health status, life expectancy, quality of life (14), and health care costs.
Effectiveness and efficiency are regarded as attributes of quality in health care. The term effectiveness is defined as the ability to attain the greatest improvements in health care now achievable by the best care; efficiency is defined as the amount of resources needed to attain a certain goal. According to Donabedian, this goal should be to lower the cost of care without diminishing attainable improvements in health (15). For each article included, the type (structure, process, or outcome) and attribute (effectiveness or efficiency) of indicators were registered.
Providing a Taxonomy of Indicators and Establishing the Framework
An overview of indicators currently used in evaluating disease management programs is provided in tabular form in relation to the aim(s) and type of the disease management program. Based on the indicators used in studies that were considered as having good methodological quality and complementary literature concerning HTA and quality assessment, a framework is provided for evaluation of each type of disease management program.
RESULTS
Description of Studies
The search identified 123,104 references published between January 1987 and September 2004, of which we accepted 6,214 for further screening. After reading titles and abstracts, 6,052 papers were excluded for not reporting any form of program evaluation or for focusing on pediatric patients. As a result, 162 full-text articles were retrieved for more detailed information and screening of references resulted in another 94 potentially relevant articles. On the basis of a full-text review, 220 articles (86 percent) were excluded as not meeting the inclusion criteria. Finally, thirty-six studies were included in the review, ten of which focused on asthma (11;18;26;29;33;36;40;44;52;62), twenty-five on diabetes mellitus (1;4;6;7;10;12;17;19;22–24;28;34;35;39;41;43;45–47;51;53;56;57;59), and one on both diseases (37). No study on disease management for patients with COPD met the inclusion criteria.
The overall methodological quality of studies was moderate. Five studies scored 70 points or more on a scale from 0 to 100 and were considered as having good methodological quality (4;6;18;56;57); nineteen studies scored between 50 and 69 points (1;7;11;12;17;22–24;26;28; 34;36;37;39;40;43;46;47;59) and, therefore, were considered of moderate quality, whereas twelve studies scored between 30 and 49 points and were considered of inferior methodological quality (10;19;29;33;35;41;44;45;51–53;62).
Fit Between Aim of the Disease Management Program and Applied Indicators
In three (28;37;44) of the thirty-five (1;4;6;10–12;17–19;22–24;26;28;29;33–37;39–41;43–47;51–53;56;57;59;62) studies that described improvement of effectiveness being (one of) the aim(s) of the disease management program, no indicators for effectiveness were actually used in the evaluation; in ten (1;7;12;19;34;41;43;45;47;51) of twenty-five (1;4;7;10;12;17–19;22;26;28;34;37;40;41;43–47;51;52;56;57;62) studies that aimed to improve efficiency, no indicators for efficiency were presented. However, in four (29;33;35;39) of eleven studies in which no aim to improve efficiency was formulated (6;11;17;23;24;29; 33;35;36;39;59), indicators of efficiency were applied, as was the case for one study (7) in which no intention to improve effectiveness was announced but indicators for effectiveness were presented.
In Table 1, a more detailed overview of the specific structure, process, and outcome indicators that are applied in evaluating disease management programs is presented and related to the aim of the program as described in the studies. First, it appears that the use of structure indicators is uncommon. Second, efficiency of disease management is mainly evaluated by means of process indicators; the use of patient outcome indicators to evaluate efficiency is still relatively limited.
Relation Between Type of Disease Management Program and Indicators Used
All (combinations of) types of disease management were present in the included studies. In Table 2, an overview is presented of the indicators that were applied as related to these different (combinations of) types. In general, the efficiency of all types of disease management programs is evaluated by means of process indicators. Concerning effectiveness, it appears that no process indicators were applied in evaluating disease management programs that mainly consist of organizational and educational interventions.
Regarding the specific structure, process, and outcome indicators that are reported, it is questionable if actually those indicators are chosen that are expected to be most sensitive to changes caused by the type of interventions applied in the program. For example, educational interventions are explicitly reported in eighteen studies, whereas in twelve of these studies (11;19;24;26;28;37;40;43;46;51;52;62), disease-specific knowledge has not been used as an indicator for effectiveness.
Establishment of a Framework
An overview is provided of indicators used for the evaluation of each type of disease management program as related to the general aim of the program. Specific structure, process, and outcome indicators for effectiveness and efficiency are recommended for each type of disease management program in Figure 1. When following the arrows, indicating expected direct links between structure, process, and outcome indicators, it can be seen that the type of disease management is reflected in the process indicators, whereas the outcome indicators are the same for all three types of disease management programs. The structure indicators also reflect the type of disease management to some extent, but because most disease management programs are so-called mixed programs, there is some overlap between them.
Framework for health technology assessment of disease management.
DISCUSSION
The art of linking outcomes to organization and delivery of disease management programs depends on matching the right outcomes with the goals of the organizational arrangements and their conceptualization and measurement in health services research (42). This study shows that a link between aims of disease management and evaluated structure, process, and outcome indicators does not exist in a substantial part of published studies on diabetes and asthma/COPD disease management, especially when efficiency of care is concerned. Concerning the effectiveness of disease management, in three of thirty-five published studies (9 percent), no fit between aim of the program and assessed indicators exists, whereas for efficiency, this is the case in ten of twenty-five studies (40 percent). Furthermore, efficiency of disease management is mainly evaluated by means of process indicators, whereas the use of outcome indicators is less common. This problem could be overcome by explicitly combining process parameters with outcome parameters as illustrated by Mason et al. (32).
Because it is hardly ever possible to have one empirical study that gathers all the data needed for an HTA study, the choice for specific structure, process, and outcome indicators was expected to be influenced by (or focused upon) the type of disease management program that was evaluated. The data from this review do not support this expectation. Actually, there is hardly any reflection to be found in the content of the specific indicators that are applied, as related to the different types of disease management programs. This finding questions the sensitivity of the chosen indicators to detect (small but) important differences in the effectiveness or efficiency of disease management programs.
Structure
Structure indicators are to a large extent missing from the research literature. Because most disease management programs involve a form of process redesign, which is accompanied by a change in the structure of health care, information about the structure of care is essential. There are three types of structure indicators that affect both cost and outcomes and can differ between disease management programs: (i) financial arrangements/incentives, (ii) site of care delivery, and (iii) composition/organization of the care delivery team (21). To assess the generalizability and robustness of health care outcomes, one should understand to what extent the difference in cost and outcomes is attributable to differences in financial arrangements, differences in the sites of care delivery, or differences in the composition and organization of the care delivery system.
Although the lack of structure indicators might be partly explained by the fact that sometimes process measures also reflect structural characteristics of health care, this does not invalidate the observation that, in most studies, important data on the structure of health care (either measured by structure or process indicators) are missing.
Process
In assessing the efficiency of disease management programs, mostly process indicators are used. Two possible reasons for this strategy are (i) efficiency gain within disease management is commonly attempted by process redesign, therefore, assessing process indicators as outcome units seems a logical choice; (ii) continuous measurement of indicators is part of the feedback systems that are a component of disease management programs itself. For this purpose, indicators are assessed that are relatively easy to measure, which are mostly process indicators. It is supposed that these indicators are then continuously available for assessing the overall efficiency of the program.
An intrinsic advantage of process indicators is that they are more sensitive than outcome indicators to differences in quality of care, that they are easy to interpret, and that they are a direct measure of quality of care (31). On the other hand, process indicators are only of value if a robust link to (patient) outcome has been demonstrated.
Outcome
Outcome indicators reflect all aspects of the process of care, including those aspects that are not measurable or have not been measured. Moreover, the ultimate goal of HTA of disease management is to express efficiency in terms of patient outcome. An attraction of outcome indicators is that these are indicators of something important in their own right (31). In other words, not only the number of resources used in executing disease management programs but, in addition, a valuation of these resources in terms of costs and utilities are necessary for local or national decision making (16). For purposes of generalizability, the reporting of units of component resources, disaggregated from their unit costs, will help decision makers in other regions or countries to derive valid local costs of implementation by applying local patterns of resource use and prices (32).
Sensitivity, Comparability, and Generalizability
Currently, results from different disease management programs are hard to compare even when the type of disease that is focused upon has been taken into account. Apart from heterogeneity of programs, this finding is also due to the wide variety within indicators used to evaluate the programs. Therefore, it is suggested that process indicators are chosen that are sensitive to the specific interventions of a particular type of disease management program and are associated with the expected changes in outcomes of care (27). For example, when a disease management program primarily focuses on patient education, process indicators should be aimed at, for instance, compliance of patients with scheduled follow-up visits (61). An increase in compliance indicates a better control of the disease (9), resulting in improved overall health status as well as fewer complications and lower costs in the long run (30;50). Outcome indicators, on the other hand, should then be more general for purposes of generalizability and comparability (for example health-related quality of life or morbidity). Furthermore, developmental and implementation costs need to be included for purposes of decision making, because these costs should not out rule the cost-effectiveness of the intervention (32).
Framework
Theoretically driven inquiries are necessary to link systems and outcomes (42). Within the presented framework, all the before-mentioned aspects of evaluating disease management for purposes of decision making are integrated to guide the choice of structure, process, and outcome indicators. As a result, three sets of structure, process, and outcome indicators for each type of disease management are proposed that should guide the selection of indicators to be included in a minimum data set. This approach is in itself not new. For diabetes self-management education, for example, an analytical framework also has been proposed (38). However, that the framework presented in this study is applicable to all disease management programs regardless of the disease they focus upon, while taking the type of disease management into account, makes it innovative and more useful for purposes of decision making. To illustrate this concept, when evaluating a disease management program that can be characterized as professional, one should at least pay attention to structure indicators related to the health care system and characteristics of care providers, process indicators related to performance of care providers, and outcome indicators for effectiveness as well as efficiency. In addition, process indicators for continuity of care and self-efficacy of patients could be taken into account. Further empirical research is necessary to detect which measures of this minimum data set are the most valid ones or for which strong links to health outcomes exist.
CONCLUSION
To measure what matters in HTA of disease management, one has to realize that many different disease management programs exist and have been implemented under varying circumstances. The aim and type of the program should guide its assessment to produce useful information for policy and practice. When paying closer attention to the type of indicators that are used for evaluating disease management programs, a very diffuse picture appears. In HTA of disease management, the link between the aim of the program and the indicators that are used in evaluating the program does not exist in a substantial part of published studies, especially when efficiency of disease management is concerned. Furthermore, structure indicators are hardly presented, although these are of major importance for the interpretation of outcomes for purposes of decision-making. Process indicators, on the other hand, are very frequently used for assessing efficiency of disease management. A valuation of these process indicators in terms of costs or utilities, which is often lacking now, would provide more useful information for policy-makers and care providers. Also, no reflection of the type of disease management program (e.g., professional, organizational, or educational) was found in the structure, process, or outcome indicators mentioned, whereas the choice for specific indicators was expected to be influenced by this. Based on the flaws in the current evaluation of disease management programs and supported by existing theories on HTA, quality of care measurement as well as decision making, a framework is presented in which structure, process, and outcome indicators are linked. Use of this framework should guide researchers to define a minimum data set that provides relevant and useful information for decision makers in health care.
POLICY IMPLICATIONS
Decision making in health and medicine preferably should be based on the best scientific evidence informing us of which interventions are worth spending limited resources on. To date, the lack of a consistent framework for HTA of disease management has made comparisons of reported results hardly possible, meaning that much time and money have been spent inefficiently. Research budgets should be reserved for those studies addressing outcomes that have shown to be of added value to decision making in policy and practice.
CONTACT INFORMATION
Lotte Steuten, MSc (L.Steuten@zw.unimaas.nl), Research Associate, Department of Health Care Studies, Maastricht University, P.O. Box 616, 6200 MD Maastricht, The Netherlands
Bert Vrijhoef, PhD (B.Vrijhoef@zw.unimaas.nl), Assistant Professor, Department of Health Care Studies, Maastricht University, P.O. Box 616, 6200 MD Maastricht, The Netherlands; Senior Researcher, Department of Integrated Care, University Hospital Maastricht, P.O. Box 5800, 6202 AZ Maastricht, The Netherlands
Hans Severens, PhD (H.Severens@beoz.unimaas.nl), Professor, Department of Health Organisation, Policy and Economics, Maastricht University, P.O. Box 616, 6200 MD Maastricht, The Netherlands; Professor, Department of Clinical Epidemiology and Medical Technology Assessment, University Hospital Maastricht, P.O. Box 5800, 6202 AZ Maastricht, The Netherlands
Frits van Merode, PhD (F.vanMerode@beoz.unimaas.nl), Professor, Department of Health Organisation, Policy and Economics, Maastricht University, P.O. Box 616, 6200 MD Maastricht, The Netherlands
CorSpreeuwenberg, MD, PhD (C.Spreeuwenberg@zw.unimaas.nl), Professor, Department of Health Care Studies, Maastricht University, P.O. Box 616, 6200 MD Maastricht, The Netherlands
