Clinical practice guidelines (CPGs) are documents containing recommendations drafted according to a scientific criterion to facilitate decision making in health care (Reference Hart and Bailey22;Reference Lohr, Field, Field and Lohr28); the fundamental role they play in decision-making improvement strategies in healthcare systems is undeniable. This key role in healthcare management has generated growing enthusiasm among professionals to come join in guideline production. There are numerous organizations that draft CPGs which, in addition to the unnecessary cost of duplicating efforts, means that there are more and more guidelines available that address the same subject, with the risk of contradictory recommendations (Reference Jovell, Navarro-Rubio and Aymerich25;Reference Miller and Petrie30). Different studies have revealed a tremendous deficit with respect to both the quantity and quality of the information regarding the methodology used, an essential issue if users are to be able to properly evaluate the quality of the guideline in question (Reference Hart and Bailey22). Those guidelines for which information was available were designed in such a way that they failed to comply with appropriate methodological standards, with greater need for improvement in domains such as identification, evaluation, and synthesis of the scientific evidence used (Reference Shaneyfelt, Mayo-Smith and Rothwangel38). These deficits in the guidelines are particularly conspicuous when the CPGs have not been elaborated by multidisciplinary teams (Reference Miller and Petrie30). A recent study has revealed that the quality of CPGs developed by scientific societies is insufficient and recommends that specific methodological criteria be published for the production of CPGs (Reference Grilli, Magrini, Penna, Mura and Liberati17). A study conducted in Spain on the quality of national CPGs (Reference Navarro Puerto, Ruiz Romero and Reyes Domínguez31) revealed that most were of poor quality, probably due to the fact that they were created out of spontaneous initiatives that were not part of integral programs that included both elaboration and detection of healthcare priorities, development and implementation of CPGs, as well as outcomes surveillance (Reference Marzó, Alonso and Bonfill29).
Nowadays, scientific literature can overwhelm physicians who at times find it difficult to search for and review the relevant literature within the context of their routine work. Hence, CPGs play a critical role in assisting the clinician in making decisions based on the best evidence available. The problem is that there are too many, poor quality CPGs that are not based on the best evidence; at times, the evidence is clear for only one part of the guideline recommendations, and, furthermore, it is difficult to transfer the evidence to recommendations and then translate the CPG recommendations into decisions for any given patient (Reference Grol18).
Stroke is one of the leading causes of death and disability in developed societies. In the United States alone, there have been more than seven hundred thousand episodes, with a prevalence rate of four million patients. This has generated direct and indirect costs in excess of 40 billion dollars at the end of the millennium (Reference Gorelick, Sacco and Smith16) that have grown as a result of the more widespread use of fibrinolytic treatment. There is room for improvement in this regard; bearing in mind that stroke is preventable. In fact, it is one area in which CPGs can have their greatest impact on incidence rates and sequelae, making stroke an especially relevant pathology for individuals and for healthcare systems alike.
Therefore, we would like to examine whether clinical practitioners can consider the guidelines available on established ischemic stroke to be reliable, given that there is a certain degree of distrust due to the incongruities, interests, and poor methodological quality apparent among them. Within this context, we propose the following study that seeks to evaluate the quality of the clinical practice guidelines on established ischemic stroke in terms of their format and development process.
MATERIALS AND METHODS
Design
Descriptive study and qualitative research techniques suited to the objectives to be attained.
Objects of Study
CPGs drafted regarding the management of established ischemic stroke. CPGs on transient ischemic attack (TIA) and hemorrhagic cerebral vascular accident (CVA) have been excluded.
Search and Selection of Guidelines
A search was made for guidelines in English, French, Italian, Portuguese, and Spanish, published between 1999 and June 2004, in five bibliographic databases that index guidelines: Medline, National Guidelines Clearinghouse, Lilacs, EMBASE, and NeLH. The search was completed with the following databases, search engines, and clinical practice guideline producers: Fisterra, NZGG, GIN, CPG Rafa Bravo, CMA Guidelines, SIGN, Stroke-info guidelines, ICSI, European Stroke Initiative (EUSI), and Indice Médico Español (IME).
The results were screened and duplicates were eliminated. The inclusion and exclusion criteria previously subject to consensus in the research group were subsequently applied.
Inclusion and Exclusion Criteria
Inclusion Criteria. All the full-text clinical practice guidelines regarding the management of established ischemic stroke that were identified and retrieved were included in the study. CPG is understood as a recommendation document drafted according to a scientific criterion for the purpose of facilitating healthcare decision making. It should be multidisciplinary, scientific evidence should prevail, a posteriori consensus should be sought, and it should refer to a broad field of application and contain strategies for dissemination, updating, and evaluation.
Exclusion Criteria. Protocols or other mandatory documents for care organization are not contemplated in this study. A protocol, as opposed to a guideline, is a regulatory document, generally devised by means of a unidisciplinary, local process in which clinical judgment and expert opinion prevail; it seeks consensus a priori, does not follow a documented process, applies to a limited area and does not contain elements for evaluation, implementation, or updating.
Evaluation of CPG Development Methodology
The AGREE instrument (Appraisal of Guidelines Research & Evaluation) was selected as the appraisal tool, given that it is a validated instrument endorsed by the leading producers, raters, and compilers of international CPGs. The purpose of the AGREE instrument is to provide a framework for assessing the quality of clinical practice guidelines. It assesses both the quality of the reporting, and the quality of some aspects of recommendations. It provides an assessment of the predicted validity of a guideline (see AGREE trust for more details: http://www.agreetrust.org/).
After proper training in an accredited workshop, four experts conducted an independent evaluation of the guidelines according to the instructions of the AGREE instrument. Twenty-three items were independently assessed and grouped into the six domains defined by the instrument. The mean score of the four raters was calculated for item analysis and categorized into three values on the basis of the information provided by the guideline on each item and completeness of the requirements demanded by the item: “Scant information/noncompleteness” (<2 points), “some information/partial completeness” (2–3 points), and “good information and completeness” (>3 points). The scale ranges from 1 to 4. For the analysis of the six domains, results were stratified into quartiles of the percentage of the highest score possible.
The research group felt that it was convenient to complement the AGREE instrument with a selection of criteria from the Changing Professional Practice (CPP) (Thorsen and Mäkeelä, 1999; the Spanish version by Briones et al. 2002) taxonomy (Reference Thorsen and Mäkelä45). The CPP is an international collaboration between several countries. This Collaboration addressed the issue of transferring scientific knowledge into clinical routines and decision making. The project was designed to promote the transfer of research findings into clinical practice by learning how best to change professional behavior based upon well-grounded clinical guidelines, to integrate analysis of national implementation experiments, and to establish a European expert community of implementation researchers. The criteria were selected by consensus by the members of the research team, including those items or criteria not contemplated by the AGREE instrument with respect to availability of data for the definition of objectives, target groups, methods, types of intervention, and outcomes of projects implementing the guidelines, thereby contemplating the maximum criteria for guideline appraisal. A list of the CPP criteria selected for data collection was created that, in addition to the criteria, contained the code assigned to each guideline, title, and year of publication corresponding to the guideline (see for more details, the CPP document on: http://www.dsi.dk/projects/cpp/Monograph/DSI9905.pdf).
All the CPGs retrieved were independently catalogued by means of the adapted CPP. Whenever a disagreement arose, a third observer designated by consensus by the working group issued the final verdict.
Data Analysis
The data obtained by assessing the guidelines after applying the AGREE instrument and the items obtained from the CPP were tabulated in an Access 2000 database (Evaluation of Guidelines Obsolescence and Kindness-EGOKi®). A descriptive study of item frequency was carried out and the quality of the CPG development process underwent analysis, followed by stratification into taxonomies using SPSS 15.0.
RESULTS
The initial search identified 117 documents. After applying the inclusion and exclusion criteria, twenty-seven guidelines (Reference Adams, Adams and Brott1–Reference Alonso de Leciñana-Cases, Pérez-R and Díez-Tejedor3;Reference Arboix, Chamorro and Escudero5;6;Reference Coull, Williams and Goldstein12–14;19–Reference Hacke, Kaste, Skyhoj Olsen, Orgogozo and Bogousslavsky21;Reference Higashida, Furlan and Roberts23;24;Reference Kothari, Hacke and Brott26;Reference Larrue, Amarenco and Caussanel27;Reference Ng, Huang, Cheung, Wong and Lam32–37;40–44) were selected with a clear predominance of North American and European guidelines, although one African and another Asian (Hong Kong) document were included.
The main finding is that most of the guidelines (17 guidelines) were deemed not recommended. Only six guidelines were recommended with conditions or modifications and four guidelines were highly recommended (see Table 1).
List of Guidelines of Excellent Quality
The main object of the guidelines was the management of established ischemic stroke, including prevention, diagnosis, and treatment. Some guidelines focused on specific issues, such as thrombolytic treatment, angioplasty, and anticoagulant and antiplatelet treatment.
Evaluation of the CPG Development Methodologies
Approximately half of the guidelines were published in 2002 and 2003. Sixty-three percent of the guidelines were disseminated at a national level and 14.8 percent with a local or regional level; whereas only six guidelines (22.2 percent) had an international scope. Most of the guidelines were developed by a professional organization (74.1 percent) or by clinical services (14.8 percent).
In general, the guidelines received a low score on all of the AGREE domains (Figure 1). Table 2 presents the results according to quartiles of the highest score possible.
Results of the analysis of the guidelines on the six Appraisal of Guidelines Research & Evaluation (AGREE) domains. Numbers represent outlyers guidelines references.
Guideline Distribution (%) According to Score on Each of the Domains Assessed by the AGREE Instrument
The lowest rated domains were Applicability, Stakeholder involvement, Rigor of development, and Editorial independence; domains on which more than 80 percent of the guidelines scored as low or very low. The highest rated domain was Clarity and presentation, followed by Scope and purpose, with 77.8 percent of all the guidelines scoring high or very high on the first domain.
Insofar as the analysis of the different criteria that comprise the AGREE domains is concerned, 25.9 percent of the guidelines failed to provide any information about their general aims for the domain Scope and purpose. Appropriate clinical care was claimed to be the main purpose in the 96.3 percent that did state their objectives, with a decrease in management variability another, albeit less often, pursued aim. The clinical issues addressed by the guidelines, the key recommendation in particular, were very well defined in a mere 22.2 percent of the guidelines. Up to 40.7 percent of all the guidelines did not describe the patients the guideline was targeting.
As far as Stakeholder involvement is concerned, only 18.5 percent of the guidelines had been developed by a multidisciplinary team. A total of 74.1 percent of the guidelines were produced by professional organizations. The patient's opinion was taken into account during the development process in just one of the CPGs. Just over half of the guidelines provided some or excellent information about the users for which they were meant; they were intended for specialized care in all cases and also for primary care in 11.1 percent. None of the guidelines were piloted among the target users before validation.
In the domain of Rigor of development and with respect to the use of systematic search methods for evidence in the development process, only 18.5 percent of the guidelines reported the strategy used and a mere 14.8 percent reported the inclusion/exclusion criteria for selecting the evidence revealed by the search. Again, just over half (51.8 percent) of the guidelines furnished information describing the methods used for formulating the recommendations, with an explicit link to the supporting evidence the method in most cases, followed by an evidence-based method. The recommendations were explicit in 70.4 percent of the guidelines, as they contemplated health benefits, side effects, and risks more broadly. In 59.3 percent of the guidelines, there was an excellent, explicit link between recommendations and corroborating evidence. Only four guidelines were reviewed by external experts before publication. Most of the guidelines (77.8 percent) did not anticipate future updates; the ones that are updated tend to be updated regularly, and most of them are drafted by professional organizations. The most rigorously developed guidelines were drafted on a national level by professional organizations.
The criteria of the Clarity and presentation domain are generally stated more explicitly in the guidelines. In 74.1 percent of the guidelines, the recommendations were highly specific and easily identifiable in 92.6 percent. More than 85 percent (85.1 percent) of all the guidelines described different disease management options. More than 35 percent (37 percent) included implementation strategies; the most common was distribution of educational material (37 percent). Other implementation strategies used to a lesser extent were conferences (18.5 percent), processes of local consensus (14.8 percent), local opinion leaders (7.4 percent), audit and feedback (11.1 percent), and patient-mediated interventions in only one guideline. The guidelines rarely discussed Applicability. Thus, four guidelines addressed the possible obstacles for implementation; two factored in the costs involved in applying the guidelines, and five included some kind of indicator that would make future guideline adherence audits possible.
Insofar as Editorial independence is concerned, four guidelines were editorially independent of the funding agency. The conflicts of interest of the members of the guideline development panel were stated in five.
Finally, the degree of relationship between the CPP criteria and the recommendations of the evaluation was assessed by means of the AGREE instrument (Table 3).
Percentage of Guidelines According to Degree of AGREE Recommendation Based on CPP Criteria
DISCUSSION
In recent years, we have been witness to an increased number of clinical practice guidelines put out by different bodies. One of the aims of the guidelines is to reduce clinical practice variability, thereby aiding in decision making. They can hardly achieve that goal if the guidelines themselves differ in their recommendations regarding the decisions they seek to guide. Therefore, the emphasis of the debate among guideline producers in the past several years has shifted from “what” to “how,” because they must be developed from a multidisciplinary perspective, based on a systematic review of the existing literature and have a clear link between the recommendations and the evidence that underpins them.
This proliferation of CPGs has given rise to a growing need to establish internationally developed criteria for the elaboration of CPGs. The AGREE instrument was developed in 1998 in response to this need, as the main deliverable of the international collaboration for the promotion of research into the quality and methodological development of CPGs. It is a generic instrument validated for the evaluation of CPGs applicable to any field of medicine. Likewise, the AGREE instrument can lay the foundation for defining the steps involved in producing quality CPGs (Reference Burgers, Cluzeau, Hanna, Hunt and Grol8).
Most of the guidelines, 63 percent, are of poor quality and a minority, 37 percent, are good or excellent (14.8 percent). The AGREE domains that refer to the requirements that are indispensable if a guideline is to be deemed to have even a minimum level of quality in its elaboration, such as scientific rigor and the involvement of both a multidisciplinary task force, as well as patients, are the lowest rated domains in the assessment, with a low or very low score on each domain in 81.5 percent and the 89 percent of the guidelines, respectively. This results in CPGs of little value, given that the guidelines founded on expert consensus or on a nonsystematic review of the bibliography are vulnerable to significant bias, due to improper selection of the evidence and the lack of a link between recommendations and evidence (Reference Antman, Lau, Kupelnick, Mosteller and Chalmers4).
According to these findings, we might doubt the reliability of the CPG as documents that can aid healthcare professionals in care provision, resulting in them being used less with the consequent waste of the resources used in producing them. Moreover, this possible situation might be reinforced by the scant view toward the future with respect to the lack of applicability analyses in most cases.
It is possible that at times the issue of the poor quality of a guideline after its analysis with the AGREE instrument was not due to the development of the CPG per se, but to insufficient description of the methodology followed. Nevertheless, it is reasonable to assume that an “evidence-based guideline” should establish an explicit link between the recommendations it puts forth and the evidence used; hence, it can be presumed that the poor methodological quality of the guidelines detected in the study is real.
The guidelines with the best methodological quality are drawn up by multidisciplinary groups pertaining to professional organizations within the framework of a program dedicated to producing guidelines. These groups produce fewer guidelines than the panels within unidisciplinary and independent scientific associations or clinical services. In one study performed to identify predictors of high quality oncology CPGs (Reference Hart and Bailey22), the ones having the best quality were produced by government-backed organizations, with or without a structured program. Other characteristics analyzed, such as date of publication, type of CPG, format, level of care, and so on, were not independent predictors of quality. This finding is consistent with the results of another study (Reference Burgers, Fervers and Haugh9), in which the guidelines having the best methodological quality were developed by well-established CPG production programs and by governmental agencies. This may be due to the greater availability of resources these agencies have in comparison with professional associations and scientific organizations.
The low score the guidelines received on the Applicability domain is particularly striking; it was one of the AGREE domains that scored lowest. This appears to be fairly widespread among the CPGs, because in this regard, our results coincide with those of other studies, for instance, the previously cited study of oncology guidelines (Reference Fervers, Burgers and Haugh15), in which the Applicability domain also received a low score. These findings underscore the fact that CPG producers need to pay attention to CPG applicability and implementation during the development process if they are to have an impact on clinical practice.
Among the difficulties entailed in using the AGREE instrument, we would stress the need for training before its use, as well as the need to unify criteria among the raters, bearing in mind that it is not always possible to find four raters or to perform it with a learning period. The subjectivity of each evaluator must also be taken into consideration in the reproducibility of the evaluation; consequently four raters are recommended as a strategy by which individual subjectivity can be reduced, thereby providing greater objectivity in the use of the AGREE instrument.
Likewise, it must be pointed out that few of the guidelines included a conflicts of interest statement (found in only five guidelines) and the all but nonexistent editorial independence of the body funding the guideline (encountered in only four guidelines). This is currently an important and widespread issue. Despite that the conflicts of interest statement is a widely accepted policy by medical journals, it is still not widespread in clinical practice guideline production. These findings are in line with those of an analysis of editorial independence of twenty-eight guidelines published and available in the Guiasalud catalogue (Reference Briones, Vidal, Navarro and Marín7), where the information provided regarding the funding and editorial independence of said guidelines was scarce; in fact, possible conflicts of interest were mentioned in a mere 57.1 percent of the documents.
A recent article examined more than 200 guidelines (from different countries) and found that more than one third of the authors had economic ties with important pharmaceutical companies and yet more than half of the CPGs failed to provide any information about conflicts of interest. This can be as far-reaching as the existence of guidelines with contradictory recommendations, with some recommending the use of a drug and others recommending it not be used (10;11).
Insofar as updating the CPGs is concerned, it has been demonstrated that a guideline becomes obsolete in a mean period of 3.6 years (Reference Shekelle, Ortiz and Rhodes39) and that the validity of all guidelines should therefore be reevaluated every 3 years. However, of the guidelines regarding stroke analyzed, only 29.6 percent are regularly updated and 22.2 percent were updated irregularly.
Hence, an effort would have to be put into methodological training for guideline producers to enhance the average quality of the CPGs addressing stroke, seeking the cooperation of professional associations and groups with institutions dedicated to quality of care, such as technology evaluation agencies or the Cochrane Collaboration. Some kind of programming should be introduced that would avoid redundancies; contemplate updated versions of the same document and facilitate support for development.
Future studies should analyze the criteria of guideline content that diminish their quality, such as the lack of correlation between evidence and recommendations, the accuracy of their wording, or elements that are more prone to outdating the guidelines, among others.
In short, according to the outcomes of our study, the quality of CPG development can be objectively and reproducibly appraised using the Spanish version of the AGREE instrument. It constitutes a rating tool clinical guideline users can trust to ascertain ahead of time whether the guideline presented to him/her is worth considering. Likewise, this evaluation reveals that the guidelines have a low degree of reliability, in light of the fact that a mere 37 percent of the guidelines on stroke can be considered good or very good.
CONCLUSIONS
The first conclusion we can draw from our study is that the average quality of the guidelines addressing established ischemic stroke is very low. The second conclusion is using the AGREE list of criteria in its Spanish version is feasible for appraising available guidelines. Other important conclusions are that guidelines with the best methodological quality are drawn up by multidisciplinary groups pertaining to professional organizations within the framework of a program dedicated to producing guidelines and the low score the guidelines received on the Applicability domain. Our final remark is that the guidelines on stroke are of poor methodological quality; as a result, an effort is needed to generate and update high quality guidelines that make it possible to enhance the complex management of such a highly prevalent pathology, such as stroke.
CONTACT INFORMATION
Ma Asunción Navarro Puerto, MD (masunnp@telefonica.net), Assistant Physician, Department of Internal Medicine, Valme University Hospital, Ctra. Madrid-Cádiz, KM 548, Seville, Seville, 41014, Spain
Iñaki Gutiérrez Ibarluzea, MS, PhD (osteba7-san@ej-gv.es), Associate Profesor, Department of Biochemistry, Nursing University School Vitoria-Gasteiz, José Atxotegi z.g, 01009 Vitoria-Gasteiz (Basque Country); Health Technology Assessment Technician, Osteba-Basque Office for HTA, Department of Health–Basque Country, Donostia-San Sebastian, 1, 01010 Vitoria-Gasteiz (Basque Country), Spain
Oscar Guzmán Ruiz, MD, PhD (oguzman@supercable.es), Assistant Physician, Department of Internal Medicine, Francisco Moniche Alvarez, MD (pmoniche@terra.es), Assistant Neurologist, Department of Neurology, Rocío Gómez Herreros, MD (rotxogh@yahoo.es), Assistant Physician, Department of Internal Medicine, Ruth Engelhardt Pintiado, MSc (ruthengel33@hotmail.com), Research Fellow, Research Unit, Antonio Reyes Dominguez, MD, PhD (areyesdom@hotmail.com), Assistant Physician, Department of Internal Medicine, Valme University Hospital, Ctra. Madrid-Cádiz, KM 548, Seville, 41014 Spain
Ignacio Marín León, MD, PhD (ignacio.marin.sspa@juntadeandalucia.es), Associate Profesor, Department of Medicine, Faculty of Medicine, Seville University, Avda. Sánchez Pizjuan S/N, Seville, 41009, Spain; Chief, Department of Internal Medicine, Valme University Hospital, Ctra. Madrid-Cádiz, KM 548, Seville, 41014, Spain
