This is the second request I have had to review this book. The first was an invitation from the publishers to write a piece of ‘advance praise’ that might appear on the back cover or in accompanying publicity material. My little blurb went like this:
If I were seeking advice on a policy decision the last persons I might chose to consult would be a philosopher (. . . well it all depends on what you mean by policy), and a chartered accountant who happened to chair a major restaurant chain (. . . we should be able to balance the books, let's drink to that). And how wrong would I be! Cartwright and Hardie have produced an admirably clear and immensely practical guide on the use of evidence in policy making.
As it turned out Oxford decided not to use it. I can imagine the cry – ‘too damn frivolous!’ It is a little pleasure, therefore, to put these words back into print because they capture the very essence of the book. That is to say, the reader will discover an exacting and mathematically precise critique of the explanatory credentials of randomised controlled trials combined with a mass of worldly wise illustrations of the evidence actually needed to support practical policy decisions.
The major thesis is as follows:
Like us, you want evidence that a policy will work here, where you are. Randomised controlled trials do not tell you that. They do not even tell you that a policy works. What they tell you is that a policy worked there, where the trial was carried out, in that population. Our argument is that the changes in tense – from ‘worked’ to ‘work’ to ‘will work’ – are not just a matter of grammatical detail. To move from one to the other requires hard intellectual and practical effort. The fact that it worked there is indeed fact. But for that fact to be evidence that it will work here, it needs to be relevant to that conclusion. To make RCTs relevant you need a lot more information and of a relevant kind. What kind? That is what the book is about. (p. ix)
The rest of the book goes on to show that evidence-based policy is a matter of securing the causal inference (it will work here) by producing a rigorous theory to explain that linkage between interventions and outcomes and by supporting that theory with a pyramid of evidence drawn using a medley of methods. This idea of the ‘argument pyramid’ is one of the great practical contributions of the book. It is, incidentally, a notion that lends itself to some captivating graphical representations – pyramids not being that stable in the early stages of construction.
For critical balance, I should mention a couple of the book's limitations. As can be seen in the quotation above, it is deliberately written with a fresh and combative style. Evaluation is a well-established and yet broad church, orchestrated from many different standpoints and conducted through a multitude of methods. Old hands might perceive Cartwright and Hardie's style as somewhat ‘arriviste’ in that very, very little reference is made to that history. A similar point can be made in terms of the practical illustrations, which roam from integrated nutrition programmes to class-size reduction interventions to cycle helmet provision to child protection policy to (you've guessed it) introducing French food in pubs. These examples will be perceived as mere sketches from the point of view of the hard-bitten evaluator who thinks of research design in terms of the precise technical details of sample selection, data collection, data analysis, dissemination strategies and so forth.
For what it is worth, I think these considerations miss the book's point, which is to deliver a sharp dig in the ribs of the evidence clearinghouses and to shake evaluation research out of doldrums. Quite an achievement – at just over a tenner.
