Reading medical literature

The temptation when reading any paper is to concentrate on the abstract, introduction and discussion, perhaps scan the results, and leave out the methods section. Some journals encourage this by putting the methods in a smaller font or a box. This is probably the wrong way to read a paper. The introduction is usually a brief review intended to set the study in context and outline the aims. It will be too short to be systematic or balanced. It is likely to take several minutes to read and the pay-back for this time commitment will be minimal. Many journals are now severely restricting the length of introductions to avoid over-long, redundant literature reviews.

The discussion will review the results, with the authors' interpretation. This may be interesting but it may not be balanced and it should be interpreted with caution. What is really important about any paper that describes a study is how much store the reader is willing to set by the results. This requires three separate judgements (Reference Guyatt, Sackett and CookGuyatt et al, 1993, Reference Guyatt, Sackett and Cook1994): How valid is the study? What are the results? Can I use the results with my patient or in my practice?

Referring to Boxes 1 and 2, which summarise the main points of critical appraisal for a single trial and a systematic review, may be helpful, both when reading this paper and when assessing published evidence.

Box 1.

Critical appraisal of a paper on treatment

Are the results valid?

Was assignment of patients randomised and was randomisation concealed?

Is there an explicit statement about the method of randomisation? Was the method open to bias? Was it possible that investigators had control over allocation of of subjects? Computer-generated independent randomisation methods are best.

Were all patients who entered the trial accounted for at the conclusion?

All trials have subjects who drop out. If drop-outs are ignored, the distillate of completers may be very atypical. Results should usually be reported on anintention-to-treat (ITT) basis, where everyone who was randomised is included in the final analysis. Common methods of ITT include last observation carried forward (LOCF).

Were patients, clinicians and study personnel blind totreatment allocation, and was blinding assessed?

If subjects are not blind to their treatment, they may (unintentionally) give misleading answers to outcome questionnaires. Likewise, study personnel and clinicians may influence outcome if they are aware of treatment allocation. It isnot enough to assume that initial blinding will be preserved. Subjects and clinicians alike can be de-blinded by adverse effects. One way to assess this is to ask all involved in the study to guess their allocation.

Were the groups similar at the start of the trial?

Often now there are no statistical comparisons on tables of baseline data. However, differences between groups at baseline can occur by chance or may indicate non-random allocation to groups. If differences are present, think how they may influence the results.

Apart from the intervention, were groups treated equally?

Apart from the intervention being assessed, subjects should have similar treatment. Look especially for different adjunctive treatments between groups, and for different management or investigations by the study team.

Did the study have adequate power?Was there a pre-trial power calculation? This is especially important if the study did not show significant differences between groups. It is important to be able to judge whether no difference was found because there truly was none, or whether the study was too small (type II error).

What are the results?

How large is the treatment effect?

What is the number needed to treat (NNT)? This is a four-stage process:

1. (1) calculate the proportion of subjects improving in the experimental group (the experimental event rate, EER) – this may be given or be derivable;

2. (2) calculate the proportion of subjects improving in the control group (control event rate, CER);

3. (3) subtract (1) from (2) to give the net effect that treatment confers over the control (absolute risk reduction, ARR);

4. (4) divide 1 by the ARR (the reciprocal) to give the NNT.

How precise is the treatment effect?

What are the confidence intervals? The following equation will give the 95%confidence limits of the ARR.

ARR ñ 1.96 x

\batchmode \documentclass[fleqn,10pt, legalpaper]{article} \usepackage{amssymb} \usepackage{amsfonts} \usepackage{amsmath} \pagestyle{empty} \begin{document} \[\sqrt{\frac{CER\ x\ (1\ -\ CER)}{no.\ of\ control\ points}\ +\ \frac{EER\ x\ (1\ -\ EER)}{no.\ of\ experimental\ points}}\] \end{document}

The reciprocal of these numbers gives the confidence limits for the NNT.

Will the results help me in caring for my patient?

Are my patients similar to those in the trial?

Consider exclusion factors of the study being appraised (medication, generalhealth, etc.) and the population from which the sample was drawn (e.g. are they the same age and ethnicity as your patient?).

Were all clinically relevant outcomes considered?

Do the authors report adverse events as well as benefits.

Are the benefits worth the harms and costs?This is a matter of judgement. Think not only of monetary cost, but of the risk– benefit ratio of the intervention.

What does my patient think? A most important question. The clinician is now in a position to take the results to a patient and discuss the evidence.

Conclusion

Where on the continuum between very good and very bad does the study lie? Decide for yourself.

Very good

Very bad

Critical appraisal

Two things are important when assessing a piece of evidence. First, by reading the methods section, it is possible to appraise the way the study was conducted and judge whether the study is applicable to a particular clinical setting. The methods should provide sufficient information to make judgements about the process of randomisation, concealment of treatment, completeness of follow-up, how missing data were accounted for and the representativeness of the sample in relation to patients in a particular clinical setting (Table 1). Second, it is important to assess how significant the results of the study are in clinical as well as statistical terms. In other words, are the results sufficiently impressive to alter clinical practice? Different types of study best address different clinical questions and these are outlined in Table 1. Here I concentrate on evidence relating to treatment (which is the subject of the bulk of ‘evidence’ in the literature) and focus on single, randomised controlled trials and systematic overviews.

Developing critical appraisal skills

Reading this paper is insufficient preparation for how to read the literature in an efficient and effective way. However, there are strategies and resources that will help develop and maintain these skills, and some of these are shown in Box 3.

Even if none of these methods is available, simply concentrating on the validity of a paper's study methods and assessing the results for clinical relevance will enhance skills over time. Books by Sackett et al (2000) and Lawrie et al (2000) give sound guidance on critical appraisal, and a list of useful websites is given in Box 4.

EBM tyranny

Judgement is an essential part of the EBM process. Some advocates of EBM, who seem to want to tyrannise rather than inform, suggest that all medical studies should be dichotomised into ‘good’ or ‘bad’, with the implication that bad studies should be ignored. This would result in ignoring many medical studies. This is not what EBM is about. All medical studies lie on a continuum between the very good and very bad. Where a study is on this continuum is a matter of judgement. Almost all studies will be useful to some degree and the judgement comes in deciding just how ‘good’ or ‘bad’ a particular study is, and consequently how much that study will inform practice. It is worth noting that the good reputation of the journal or of the authors is no guarantee of quality and that studies funded by the pharmaceutical industry do tend to report better results than independent researchers.

Another example of EBM tyranny is the devaluing of evidence of therapeutic efficacy that is not from an RCT. It is a fallacy that only RCTs can yield useful information on therapy. For example, faced with a particular clinical question, the only evidence available may be from a series of case reports. This is better than no evidence and should be given some credence. In open-label studies without blinding or randomisation bias may be a significant problem, but the evidence presented should not be disregarded: it can be interpreted in the light of potential flaws. Even RCTs may not be perfect, and you should consider the validity, results and applicability of each piece of evidence found.

EBM nihilism

There are two main sources of nihilism that most EBM practitioners (except the evangelists) will have to negotiate. First, there will be times when it is not possible to find a piece of evidence relating to a question, and second, there will be many times when the evidence found is towards the ‘very poor’ end of the spectrum owing to problems with validity or applicability. Recognising that the evidence base for much of clinical practice is inadequate can be very therapeutic. It is also helpful to bear in mind that evidence does not necessarily ‘trump’ clinical experience and the absence of evidence is not the same as evidence of absence.

The future

There are already several journal sources of critically appraised evidence, including Clinical Evidence, Evidence-Based Mental Health and Bandolier. With the increase of medical knowledge, improvements in study design and reporting of results, and better information technology, there is likely to be a move towards the use of instant, high-quality information to guide medical practice. Evidence-based technologists may become integral to ward rounds and community meetings. Armed with a laptop computer with internet access, it is now possible to search, identify and download full-text articles from thousands of medical journals. These can then be critically appraised and an answer to a clinical question provided within an hour or so. EBM is not a fad, and the need to embrace it, master the basic skills and use it sensibly is becoming essential.

Conclusions

This paper is a guide to effective reading of the medical literature. It will take practice and dedication for the skills to become second nature. Remember, research is very hard to do and very easy to criticise, so be gentle with your criticisms.

Multiple choice questions

1. 1. An RCT reported remission rates for depression of 65% for fluoxetine and 60% for imipramine; the NNT is therefore:

   1. a 60

   2. b 20

   3. c 15

   4. d 10

2. 2. Important factors when assessing the validity of an RCT include:

   1. a the patient's opinion of the treatment

   2. b the method of randomisation

   3. c the adverse-effect profile of the drug

   4. d the power of the study

   5. e how missing subjects were accounted for.

3. 3. The following statements are true:

   1. a a meta-analysis is always better evidence than a single randomised trial

   2. b studies with large numbers tend to have wider confidence intervals

   3. c Forest plots are a graphical representation of the studies in a systematic review

   4. d in a meta-analysis, if the 95% CIs of a pooled weighted mean difference are –2.3 to –0.8 the results are statistically significant

   5. e P values are a good indicator of clinical significance.

4. 4. When making decisions about a patient's drug treatment, the following should be considered whenever possible:

   1. a the patient's opinion

   2. b the doctor's experience of using the drug

   3. c the risk–benefit ratio of the drug

   4. d the validity of the evidence for the treatment

   5. e the effect size of the treatment.

5. 5. The following evidence types are matched with a suitable study method:

   1. a diagnostic test: case–control study

   2. b prognosis: prospective cohort study

   3. c therapy: cross-sectional survey

   4. d aetiology: case–control study

   5. e epidemiology: cross-sectional survey.