No CrossRef data available.
Article contents
Clinicians' assessments of practice guidelines in oncology: The CAPGO survey
Published online by Cambridge University Press: 01 November 2004
Abstract
Objectives: Promoting the adoption of clinical practice guidelines is a challenging task. Research shows that adopters' beliefs about guidelines and the process of their development are important predictors of guideline use. Our objective is to report on the development and refinement of an instrument designed to measure oncologists' assessments of guidelines in oncology.
Method: A theoretically derived instrument was drafted. Oncologists were asked to complete a questionnaire for each draft guideline they were asked to review. Factor analyses and multilevel modeling techniques were undertaken to explore the properties of the instrument and refine it.
Results: A total of 488 Ontario clinicians were sent 1,494 new questionnaires regarding 34 clinical practice guidelines produced between 1999 and 2002. A refined eighteen-item questionnaire with four stable factors that predicted 60 percent of the variance emerged. The factors are interpreted as guideline quality, applicability, acceptability, and comparative value. The four factors predicted oncologists' endorsements of draft guidelines and, with the exception of quality, predicted their intentions to use the guidelines. As expected, variation in the factor scores could be attributed more to the differences among the oncologist who completed the survey than to the differences among the guidelines themselves.
Conclusions:An instrument composed of four stable and theoretically relevant factors emerged. The findings support the hypothesis that beliefs about guideline attributes and development attributes relate to oncologists' endorsement of and intentions to use guidelines. Our next step is to link the responses of the survey with actual clinical behavior.
Keywords
Information
- Type
- GENERAL ESSAYS
- Information
- International Journal of Technology Assessment in Health Care , Volume 20 , Issue 4 , November 2004 , pp. 421 - 426
- Copyright
- © 2004 Cambridge University Press
Evidence-based clinical practice guidelines are intended to assist in health-care decisions (6;13). However, the careful selection, synthesis, and packaging of evidence-based health messages, such as guidelines, have not resulted in their consistent application in clinical settings. This finding raises questions about the usefulness of this time-consuming and expensive enterprise (8;14;16). The findings from decades of social science research and, more recently, health services research attest to the complexity of predicting and modifying behavior (9;20). Nonetheless, there have been some very important advances in understanding the gap between evidence production and evidence uptake and identifying interventions to close the gap (8;20;21). Of particular note is the work of Rogers and others who have shown that different diffusion and uptake patterns for an innovation emerge as a function of adopters' beliefs about the attributes of an innovation (e.g., ease of implementation, the outcomes of implementation that are apparent) and the development process by which evidence is translated and incorporated into the innovation (e.g., credibility of developers, explicit use of evidence) (5;7;15;17;19;20;22;23;25;26).
This previous research focuses on the influential attributes of objective guideline qualities that are clear features of the guidelines themselves. For instance, Grilli and Lomas coded guideline recommendations based on an imposed definition of ease of use or trialability, but they did not measure users' perceptions of these guideline features. By contrast, it is arguable that it is the clinicians' perceptions of the guidelines that are important antecedents of behavior, regardless of how closely these perceptions correspond to an objective aspect of the guidelines. In deconstructing this further, one can consider the relative contributions of beliefs, attitudes, intentions, and behaviors. For example, the theory of reasoned action postulates that one's attitudes toward a behavior influences one's intentions to engage in the behavior, which, in turn, influences actual behavior (2;11;20). Attitudes themselves are a function of one's beliefs about the attributes ascribed to the object of evaluation (or the subjective belief that the attribute is expected or characterized by the object) and the value ascribed to each attribute (10;20). This expectancy-value principle has been well established and is used as the foundation of predictive behavior models (21).
Within the context of guidelines, we believe there is considerable value in being able to systematically measure adopters' beliefs. These data could be linked with attitude, intention, and actual behavioral indices to enable specific models of guideline uptake to be developed. These data would also enable developers to match and emphasize guideline attributes that appeal to stakeholders and that enhance uptake and to modify factors perceived negatively by stakeholders that impede message uptake.
Although there is a growing recognition of the importance of adopters' innovation and development beliefs on influencing practice, little effort has been directed at developing strategies to adequately measure these beliefs. The purpose of this report is to describe the development of a new instrument and to report on a multiyear study of its development.
METHODS
Context of Study
The development and validation of the instrument were conducted in a field setting, the Practice Guidelines Initiative (PGI) of Cancer Care Ontario's Program in Evidence-based Care. The Practice Guidelines Development Cycle (3) is the model used by the PGI to direct the development of clinical practice guidelines for oncologists in Ontario, Canada, and to facilitate best cancer care practice. The external review stage of the cycle, called practitioner feedback, uses a standardized questionnaire (the Practitioner Feedback Questionnaire [PFQ]) to elicit assessments of draft guidelines by practicing clinicians (4). The practitioner feedback process was chosen as the venue to develop a new PFQ.
Questionnaire Development
The diffusion of innovations research articulating the predictable role of an innovation's characteristics and development processes served as the conceptual framework from which to derive items for the new PFQ (5;7;15;17;19;20;22). This process yielded an instrument with eighteen core items related to scientific quality, methodological rigor, implementability and applicability, and acceptability of recommendations. A five-point Likert scale was used to assess the items (strongly agree–strongly disagree). For the two items for which it was relevant, a not applicable choice was added to the five-point scale to allow for judgment that the proposed recommendations reflected current practice.
Three additional items were added to the core of eighteen. In one item, clinicians were asked to indicate whether the guideline was relevant to their clinical practices (yes, no, unsure). Two items were included that tapped the extent to which respondents endorsed the draft document as a practice guidelines (five-point Likert scale, strongly disagree–strongly agree) and their intentions to use the recommendations in their own clinical practices, should the document be approved as a practice guideline (five-point Likert scale, not at all likely–very likely). The latter two items were used as outcome measures for the validity testing.
The items and scales were initially reviewed by five methodologists and five oncologists involved with the PGI, and modifications were made. A feasibility study using 354 clinicians to assess four draft guidelines was successfully undertaken and final modifications to the tool were made. All the modifications related to wording changes to increase clarity and understandability of the items. There were no suggested additions or deletions to the items.
Testing and Refinement Protocol
A study was undertaken to establish the psychometric properties of the instrument. To this end, 488 Ontario clinicians (46.6 percent medical oncologists/hematologists, 14.8 percent radiation oncologist, 25.8 percent surgeons, and 12.7 percent other specialists) were sent 1,494 new PFQs regarding thirty-four clinical practice guidelines between 1999 and 2002. Clinicians received an average of 3.0 guidelines each. The distribution of the surveys and reminder system followed current practice (see reference 4).
Treatment of the Data
Principle Components Analysis
To determine the instrument's dimensions, a principle components analysis was conducted using sixteen of the eighteen core items. Across the thirty-four guidelines, not applicable was chosen 39.7 percent and 36.9 percent of the time for the two items in which this response option was included. As such, these items were not included in the principle components analysis solution and were treated as their own factor on the basis of their tendency to illicit not applicable responses.
Reliability
Measures of internal consistency, Cronbach alpha coefficients, were calculated for each of the factors that emerged from the principle components analysis and the entire instrument. Respondents who chose the not applicable response option were excluded from the analysis. Inter-rater reliability is often a measure calculated to evaluate the reliability of a new instrument (24). However, because the practitioners are of primary interest here and not the guidelines, we want our instrument to be able to detect a range of beliefs among practitioners for each guideline rather than a consistency in beliefs. Therefore, inter-rater reliability is of limited relevance, as we would not expect consistency in attitudes among Ontario clinicians within this context. By contrast, variance components analysis was used to evaluate the percentage of variance in ratings associated with practitioners and that associated with specific guidelines (see below).
Distribution of Variance Within Factors
The study was designed so that each of the 34 guidelines was rated by multiple practitioners, many of whom rated more than one guideline. Multilevel modeling was used to estimate the proportion of variance within each factor that was associated with differences among the guidelines, differences among the practitioners, and the residual variance (12;18).
Validity Testing
As an initial assessment of the instrument's validity, the factors that emerged from the principle components analysis were used to predict clinicians' endorsement of and intentions to use practice guidelines, using multilevel modeling techniques (12;18). This method first provides estimates of the variance for each of the outcome measures, endorsement and intentions, which can be attributed to each of three variance components of each measure: differences in practice guidelines being evaluated, differences in practitioners conducting the evaluation, and unexplained differences or residuals. Second, this method then explains the proportion of the variance that can be accounted for, or explained, by the factors simultaneously; this method effectively adjusts each factor effect for the others, thereby assessing the unique contribution of each. Finally, this method can then explain whether each of the factors individually predicts each of the outcomes. For these analyses, two models were derived; the first (the three-factor model) incorporated the three factors of quality, acceptability, and applicability as predictors using data from all of the participants. A second model (the four-factor model) included all four factors but with only those respondents who chose among the five-point response options for the items comprising the comparative value factor.
RESULTS
Principle Components Analysis
The principle components analysis for the sixteen core items yielded a three-factor solution accounting for 58.8 percent of the variance (see Table 1). Factor one focuses on clinicians' beliefs of quality features (quality factor) and is composed of six items accounting for 24.8 percent of variance. Factor two focuses on clinicians' beliefs of the acceptability of the recommendations (acceptability factor) and is composed of six items accounting for 18.2 percent of the variance. The third factor, accounting for 15.8 percent of variance and composed of four items, focuses on the application, implementation, and barriers to application of recommendations (application factor). The fourth factor, comparative value, composed of items seventeen and eighteen, was not included in this analysis because of the frequency of not applicable responses.
Reliability
The alpha coefficient for the factors ranged from 0.75 (comparative value factor; restricted sample) to 0.85 (quality factor) (see Table 1).
Sources of Variance within Each Factor
Proportion of variance in factor scores attributable to differences among guidelines, practitioners, and residual are reported in Table 2. For each of the four factors, the vast majority of variance in attitude score is associated with differences among practitioners beliefs (>80 percent) and very little is associated with differences among the guidelines (<8 percent). This finding is true despite the fact that practitioners were rating different CPGs. There is little evidence of consensus among practitioners in beliefs about the attributes of the guidelines and the process of their development, whereas there is very strong evidence that the same practitioner responds in characteristic or similar ways to different guidelines.
Predicting Clinicians' Endorsements of and Intentions to Use Practice Guidelines
Multilevel modeling techniques revealed that the vast majority of variance in practitioners' endorsements of and intentions to use the practice guidelines are associated with differences among the practitioners rather than differences among the guidelines. Furthermore, the techniques revealed that the belief factors were successful at accounting for much of this variance and that the factors were significant predictors of the overall scores.
Endorsement of Practice Guidelines
For the first outcome measure, practitioners' endorsements of draft guidelines, the variance in scores was 0.70. The vast majority of this variance in the endorsement scores was attributable to differences among practitioners (0.60 or 85.7 percent), with very little variance attributable to differences among practice guidelines (0.02 or 2.9 percent) or the residual (0.08 or 11.4 percent). The factors accounted for and explained much of the variance in each component. Indeed, the three-factor model (with the complete sample) accounted for 50.4 percent, 100 percent, and 96.6 percent of the practitioner, practice guideline, and residual variance, respectively. With this model, each of the three factors was a statistically significant predictor of the endorsement ratings (p<.0001). Similar results emerged with the four-factor model.
Intentions to Use Guidelines
For the second outcome measure, practitioners' intentions to use guidelines, the variance in scores was 0.76. As with the endorsement measure, this variance was attributable primarily to differences among practitioners (0.65 or 85.5 percent) relative to differences among practice guidelines (0.02 or 2.6 percent) or the residuals (0.09 or 11.8 percent). When the three-factor model was considered, 29.2 percent, 91.9 percent, and 88.7 percent of the practitioner, practice guideline, and residual variances were explained by the factors. Similar patterns were seen in the four-factor model. In predicting the intention to use the practice guidelines, acceptability, applicability, and comparative value factors (four-factor model only) emerged as statistically significant (p<.0001), but the quality factor was not significant in either the three- or four-factor solution.
DISCUSSION
This study summarized the development and refinement of an instrument to assess clinicians' beliefs about guideline attributes and the process by which a guideline is developed. Our results show that the new PFQ has four stable factors accounting for close to 60 percent of total variance in scores. The four factors are (i) quality-quality of the practice guideline and rigor of its development, (ii) acceptability– acceptability of the recommendations, (iii) applicability–implementability and applicability of the recommendations, and (iv) comparative value–comparative value of the recommendations relative to current practice. The internal consistencies of the factors are excellent, ranging from 0.75 to 0.85. The factors correspond to beliefs regarding attributes that Rogers and others have found are linked to the uptake of innovations, such as guidelines (5,7–9,14–17,19–23,25,26).
In contrast to other instruments geared to give objective appraisal of the quality of practice guidelines (e.g., AGREE instrument [1]), the focus of this tool is on guideline adopters and their belief structures. Our data demonstrate that we were successful at reaching this goal. In the first analysis, the variation in factor scores could be attributed primarily to differences among the clinicians rating the guidelines rather than to differences in the practice guidelines being rated (Table 2). In other words, ratings were associated more with the characteristic response tendencies of practitioners than with the guidelines themselves. These data were further supported by findings from the multilevel modeling technique showing that the greatest variance in practitioners' endorsement scores and intentions to use guidelines was attributed to differences in beliefs among these individuals in contrast to perceived differences among the practice guidelines or the residual.
Roger's diffusion of innovations framework suggests that attributes of the innovation and the process by which evidence is incorporated into the innovation are important predictors of guideline uptake (17). As an initial test of the validity of our instrument, the factors that incorporate these attributes were used to predict clinicians' endorsement of guidelines and their intentions to use guidelines in practice, using multilevel modeling techniques. The four factors were significant predictors of endorsement and accounted for a large proportion of the variance attributable to differences among clinicians, practice guidelines, and the residual. In contrast, clinicians' intentions to use practice guidelines were predicted by the acceptability, applicability, and the comparative value domains; when adjusted for other beliefs, the quality domain was not predictive of this outcome. It is not exactly clear why this finding emerged. Grol and his colleagues, for example, found quality issues (e.g., that the guidelines were evidence-based) to be related to uptake (17). However, it may be that clinicians' views of a guideline's applicability, acceptability, and value are intricately linked with beliefs about the document's scientific rigor and quality. For example, acceptability of recommendations may be strongly related to the quality assessments so that quality beliefs add little above acceptability beliefs. In other words, quality may not be viewed as unique or distinct from other attributes; when beliefs about these features are sought simultaneously, rather than in isolation, there may be little additional unique variation left over.
In addition, it is possible that the overall quality of the different guidelines is perceived to be homogenous, producing little variation in quality scores, thus making other factors account for more of the variance. Indeed, the PGI strives to produce high quality documents. A set of guidelines that differs more extensively in the quality domain as assessed by a validated instrument might produce different results.
This instrument was developed in the context of the PGI of Cancer Care Ontario's Program in Evidence-based Care, an established guidelines program in oncology for the province of Ontario, Canada (3). We present a theoretically based framework for reliably measuring individual differences in practitioners' beliefs, an important component in understanding guideline adoption. It is unclear the extent to which this instrument would be generalizable to other clinical settings or guidelines programs that use alternative external review methodologies. However, we believe the strong theoretical basis of the instrument improves its generalizability, because we expect the general attributes that underlie beliefs about guidelines to be relevant to innovations across clinical conditions and the continuum of care.
POLICY IMPLICATIONS
We intend to use the results from the data we gather through the survey as building blocks to testing an overall model of guideline adoption. In this model, several characteristics, including practitioners' beliefs (e.g., guideline characteristics, norms), will be considered in trying to understand clinical behavior. This strategy will not only provide us with a framework through which to explain clinical decisions and the characteristics that predict these decisions, but it will also provide data on the predictive validity of our new instrument in relation to actual clinical behavior. These results will facilitate the development of policy to support the use of practice guidelines.
We thank Cancer Care Ontario and the Ontario Ministry of Health and Long-term Care for providing financial support for this project. IDG is a CIHR New Investigator.
References
Cluzeau F,
Burgers J,
Brouwers M, et al.
2003
Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: The AGREE project.
Qual Saf Health Care.
12:
18–
21.Google Scholar
Ajzen I,
Fishein M.
1980.
Understanding attitudes and predicting social behavior.
Englewood Cliffs, NJ:
Prentice Hall;
Browman GP,
Levine MN,
Mohide EA, et al.
1995
The practice guideline development cycle: A conceptual tool for practice guidelines development and implementation.
J Clin Oncol.
13:
502–
512.Google Scholar
Browman GP,
Newman TE,
Mohide EA, et al.
1998
Progress of clinical oncology guidelines development using the practice guidelines development cycle: The role of practitioner feedback.
J Clin Oncol.
16:
1226–
1231.Google Scholar
Cabana M,
Rand CS,
Powe NR, et al.
1999
Why don't physicians follow clinical practice guidelines?
JAMA.
282:
1458–
1465.Google Scholar
Committee to Advise the Public Health Service on Clinical Practice Guidelines, Institute of Medicine.
In: Field MJ,
Lohr KN, eds.
1990.
Clinical practice guidelines: Directions for a new program.
Washington, DC:
National Academy Press;
Conroy M,
Shannon W.
1995
Clinical guidelines: Their implementation in general practice.
Br J Gen Pract.
45:
371–
375.Google Scholar
Davis DA,
Taylor-Vaisey A.
1997
Translating guidelines into practice. A systematic review of theoretic concepts, practice experience, and research evidence in the adoption of clinical practice guidelines.
CMAJ.
157:
408–
416.Google Scholar
Eagly AH,
Chaiken S.
1993.
The psychology of attitudes.
Fort Worth, TX:
San Diego: Harcourt Brace Javanovich College Publishers;
Feather NT, ed.
1982.
Expectations and actions: Expectancy-value models in psychology.
Hillsdale, NJ:
Erlbaum;
Fishbein M,
Ajzen I.
1975.
Belief, attitude, intentions and behavior: An introduction to theory and research.
Reading, MA:
Addison-Wesley;
Goldstein H.
1987.
Multilevel models in educational and social research.
London:
Oxford University Press;
Grimshaw JM,
Russell IT.
1993
Achieving health gain through clinical guidelines. I. Developing scientifically valid guidelines.
Qual Health Care.
2:
243–
248.Google Scholar
Grol R.
2001
Success and failures in the implementation of evidence-based guidelines for clinical practice.
Med Care.
39
(Suppl 2):
1146–
1154.Google Scholar
Grol R.
1993
Development of guidelines for general practice care.
Br J Gen Pract.
43:
146–
151.Google Scholar
Grol R.
2000
Implementation of evidence and guidelines in clinical practice: A new field of research.
Int J Qual Health Care.
12:
455–
456.Google Scholar
Grol R,
Dalhuijsen J,
Thomas S, et al.
1999
Attributes of clinical practice guidelines that influence use of guidelines in general practice: Observation study.
BMJ.
317:
858–
861.Google Scholar
Klein R.
1998.
Principles and practice of structural equation modelling.
New York:
Guilford Press;
Mittman BS,
Tonesk X,
Jacobson PD.
1992
Implementing clinical practice guidelines: Social influence strategies and practitioner behavior change.
Qual Rev Bull.
8:
413–
422.Google Scholar
Rogers EM.
1983.
Diffusion of innovations.
3rd ed.
New York:
Free Press;
Rosser WW,
Davis D,
Gilbart E,
Guideline Advisory Committee.
2001
Promoting effective guideline use in Ontario.
CMAJ.
165:
181–
182.Google Scholar
Rosser WW,
Davis D,
Gilbart E.
2001
Assessing guidelines for use in family practice.
J Fam Pract.
50:
969–
973.Google Scholar
Shaneyfelt TM,
Mayo-Smith MF,
Rothwangl J.
1999
Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature.
JAMA.
281:
1900–
1905.Google Scholar
Streiner DL,
Norman GR.
1989.
Health measurement scales. A practical guideline to their development and use.
Oxford:
Oxford Medical Publishing;
Thomson R,
Lavender M,
Madhok R.
1995
How to ensure that guidelines are effective.
BMJ.
311:
237–
242.Google Scholar
Tornatzky LG,
Klein KJ.
1982
Innovation characteristics and innovation adoption-implementation: A meta-analysis of findings.
J Evol Econo Trans Engineer Manag.
EM-39:
28–
45.Google Scholar
Instrument Factors, Alpha Coefficients, and Individual Items
Proportion of Variance in Factors Scores Attributable to Differences among Guidelines, Practitioners, and the Residual (Unexplained) Variance (%)
You have
Access
- 20
- Cited by