Clinical Effectiveness

The evidence for assessing the clinical effectiveness of new or existing interventions may be derived from either experimental or observational studies (Reference Rawlins1;Reference Rawlins2).

Experimental Studies

These are represented by randomized controlled trials in which two (or more) interventions are allocated randomly to groups of patients. The outcomes between the groups are then compared; and if the difference between them reaches “statistical significance” (usually p less than .05) one treatment is considered to be superior.

Randomized controlled trials have three great advantages over observational designs (Reference Rawlins1). First, the technique minimizes bias—especially selection bias—because every patient has an equal chance of receiving each treatment. Second, it minimizes the malign influence of confounding factors because these are likely to be evenly distributed between the groups. And, third, provided the groups are of an appropriate size, it minimizes the play of chance. Randomized controlled trials therefore have an established place in the assessment of the effectiveness of new and established interventions; and they will continue to have an important place in the future especially if they take advantage of Bayesian methods in their design and analysis (Reference Rawlins2).

But randomized controlled trials also have limitations (Reference Rawlins1;Reference Rawlins2). First, the design and analysis is traditionally based on the “null hypothesis” which assumes, at the outset, that there is no difference between the effectiveness of the two treatments. Only if the probability of the observed difference is less than 1 in 20 (i.e., p < .05) is the null hypothesis rejected. The difficulty with this approach is that the null hypothesis is only of interest if the intention is to investigate whether one treatment is superior to another. Although statisticians have developed equivalence, inferiority and futility designs to overcome the restrictions of the null hypothesis, these are clumsy in both their design and are unsuited to the tasks demanded of them (Reference Rawlins2). Added to which the p-value is too often erroneously believed by many nonstatisticians to distinguish between truth and falsehood.

Second, randomized controlled are undertaken in homogenous patient populations, with few or no co-morbidities, and for relatively short periods of time. Moreover the young, the frail elderly and ethnic minorities are often under-represented. The generalizability (external validity) of the results, to the real world of routine clinical care and particularly patients with co-morbidities, will inevitably be uncertain (Reference Rawlins2).

Third, randomized controlled trials have become outrageously expensive (Reference Rawlins2) due, in no small part, to the requirements of drug regulatory authorities. This is part of the reason why new pharmaceutical products have become so expensive. And finally, as discussed later, randomized controlled trials are sometimes unnecessary.

Observational Studies

There are a variety of observational designs that have substantial utility in the assessment of the effectiveness of therapeutic interventions (Reference Rawlins1). Of these, two—historical controlled trials and case-controlled designs—have been particularly important in evaluating the effectiveness of interventions.

Historical controlled trials compare the effects of an intervention with what is reliably known about the natural history of the condition or from a cohort of patients that have previously been assembled and appropriately observed (Reference Rawlins2). The use of many therapeutic interventions—of unquestioned effectiveness—is based on historical controlled trials. Some are shown in Table 1. Colchicine, the active ingredient of the plant colchicum, has been used since ancient Egyptian times for the treatment of acute gout; but was only licensed by the U.S. Food and Drug Administration, in 2009, after the results of a placebo controlled trial “confirmed” its efficacy (Reference Kesselheim and Solomon3). The effectiveness of ganciclovir, in preventing blindness due to cytomegalovirus retinitis in patients with HIV/AIDs, was accepted in the United Kingdom on the basis of evidence from historical controls; but the U.S. Food and Drug Administration required a placebo controlled trial (Reference Rawlins2). In both instances, patients in the placebo arms of these trials were required to suffer solely in the cause of regulatory science.

Table 1.

Some Interventions Whose Effectiveness Is Based on the Results of Historical Controlled Trials^1,2

In a case-control study exposure to the drug of “interest” is compared among people with and without a suspected iatrogenic disorder (Reference Rawlins1). An increased exposure in those taking the product suggests a causal association. The technique has been used to great effect in the identification of the harms associated with many pharmaceutical products and a few examples are shown in Table 2. The technique was also used to demonstrate the absence of harms associated with measles immunization (Reference Smeeth, Cook and Fombonne4).

Table 2.

Evidence of Harms Based on Case-Control Studies^1,2

Observational designs too have potential defects (Reference Rawlins1). The opportunities for bias and confounding are substantially greater than with randomized control trials but these are reduced if the effect size is substantial (Reference Glasziou, Chalmers, Rawlins and McCullough5). Observational techniques, however, offer a degree of generalizability that is unattainable with experimental designs (Reference Rawlins1); and, generally, they are much less expensive to undertake.

Overall, therefore, it should be obvious that both experimental and observational techniques have an important role to play in the assessment of the effectiveness of therapeutic interventions. Yet many—indeed too many—agencies have constructed and used so-called “hierarchies” of evidence in their assessment of therapeutic interventions (Reference Rawlins1;Reference Rawlins2).

Hierarchies of Evidence

The hierarchy of evidence shown in Supplementary Table 1, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000154, is just one of over sixty that have been published over the past 30 years (Reference Rawlins2). They all place randomized controlled trials at their summits with observational techniques at the foothills. They have been widely used not only by HTA agencies, but also by public health bodies and guideline developers, to assess the quality of the available evidence and inform the strength of any recommendations that they may make.

The use of such hierarchies is not only intellectually (and scientifically) unsound but is also unsafe. Confidence in the effectiveness of penicillin in the treatment of lobar pneumonia (level 2 according to the schema in Supplementary Table 1) is no less than confidence of the effectiveness of statins for the prevention of cardiovascular events (level 1 in Supplementary Table 1). Moreover, these hierarchies are also incapable of handling mixed study designs that are being increasingly used in decision-analytic modeling (Reference Rawlins2).

The fundamental flaw with the development and use of hierarchies of evidence is that they fail to recognize that it is not the method that matters, but whether the particular method is appropriate to answer the particular question. In many instances questions about the effectiveness of a pharmaceutical product can only be answered by randomized controlled trials (Reference Rawlins2). But this is not invariably the case; and practitioners of HTA harm themselves, their discipline—not to mention the patients they seek to serve—by slavish adherence to hierarchies of evidence.

Cost-Effectiveness

In the economic evaluation of healthcare interventions, there are four ingredients that have to be taken into account (Reference Rawlins1): The economic perspective that is to used, The type of economic analysis to be adopted, The threshold distinguishing cost-effective from cost-ineffective interventions, and The approach that is to be taken in resolving the problem of “distributive justice.”

Economic Perspective

An economic analysis of the cost-effectiveness of a therapeutic intervention will incorporate the health gains it brings about together with the costs and savings to the healthcare system itself. There are, though, secondary potential costs and savings to families and carers, to the public services (such as sickness benefits or disability payments). Taking a wider perspective, encompassing societal costs and benefits (such as effects on employment or consequences for future healthcare consumption), are described by economists as “productivity” gains and losses (Reference Rawlins1).

There are, at least superficially, attractions in taking a broader economic perspective because it places the healthcare costs and benefits in a broader economic context. There are, though, disadvantages. For example, taking account of the effects of employment has—in the current economic climate—a net zero sum effect from the perspective of society as a whole. With unemployment rates of around 10 percent to 12 percent across the European Union someone currently unemployed will replace someone who loses their job due to ill health.

The economic perspective to be used in any economic evaluation of healthcare therefore has to be carefully considered. In particular, the inclusion of productivity gains may disenfranchise those members of society who are what the economists describe as “economically inactive” and include children, adolescents and the elderly. Ultimately it is a political, rather than a strictly scientific, decision that has to be made.

Economic Analysis

The favored approach to estimating cost-effectiveness, by many HTA agencies, is the use of cost utility analysis. In this the direct and indirect costs (and savings) are estimated but, in doing so, the scope will inevitably be predicated by the economic perspective that is to be used.

In cost utility analysis, the health-related benefits are expressed from the degree of improvement in the health-related quality of life. The degree of improvement is multiplied by the number of years for which the improvement is maintained to yield the quality-adjusted life-year gained (the “QALY”). The increased net costs are then divided by the QALY gained to yield the “incremental cost-effectiveness ratio (the “ICER”).

Cost utility analysis has a major role to play in priority setting. It allows an explicit examination of the consequences of adopting one intervention, for one condition, compared with another intervention for a completely different condition.

Cost-Effectiveness Thresholds

Estimating the incremental cost-effectiveness ratio—the ICER—is one thing, but identifying a threshold distinguishing a cost-effective, from a cost-ineffective, intervention is another. Indeed, there is no consensus among health economists how this might most appropriately be accomplished. Several approaches have been suggested (Reference Rawlins1).

1. 1. If the ICERs of all the interventions used by a healthcare system are ranked in ascending order the point at which healthcare resources are exhausted might be a basis for defining the threshold. Apart from being a daunting task, such a rigid threshold would reflect a utilitarian approach (see later) that many may find offensive.

2. 2. A threshold could be devised on the basis of the public's “willingness to pay” for incremental healthcare gains. The technique has, for example, been used in transport economics but attempts to undertake this in health economics has yielded very variable results. Furthermore, its applicability for a publicly funded healthcare system is questionable.

3. 3. Another approach, and one favored by some health economists, especially in the United Kingdom, is to examine current patterns of expenditure (so-called “program budgets”) and impute the point at which benefits are foregone if a new intervention is adopted. This must necessarily assume that all current healthcare expenditure is cost-effective.

4. 4. The World Health Organization (WHO) has suggested that the threshold should be based on a country's GDP (6). Indeed, WHO advocates that an ICER below a country's per capita GDP should invariably be considered to be cost-effective; and one that is three times greater that per capita GDP should always be considered to be cost ineffective. Although there is logic to relating the threshold to a country's GDP, I am unaware of an empirical basis for the quantitative suggestions of WHO.

5. 5. Finally, there is the approach adopted by the UK's National Institute for Health and Care Excellence (NICE) which is based, essentially, on the collective judgment of the UK's health economic community.

Perhaps fortuitously, the methods described in 3) and 4) above give results that are not too dissimilar to the ICER threshold range used by NICE.

Distributive Justice

Distributive justice is the term used by political and moral philosophers in discussing what is right or just in the allocation of goods within society (Reference Rawlins7). Three theories of distributive justice—libertarianism, utilitarianism, and egalitarianism—each have a particular resonance to the allocation of resources in healthcare (Reference Rawlins7).

Libertarianism is based on the premise that individuals should be able, and expected, to finance the healthcare needs of themselves and their families; and that market forces should enable them to do so at a reasonable price. Even in countries where this approach has been most widely adopted, such as the United States, arrangements are in place to protect the less affluent members of society.

Utilitarianism requires that healthcare expenditure is used to maximize the health of the population as a whole. While emphasizing the importance of efficiency in healthcare it does little for either minorities with rare diseases or in reducing health inequalities associated with race, gender, or socioeconomic status.

Egalitarianism tries to distribute healthcare resources so as to allow each individual a fair share of the available opportunities. It seeks to ensure an adequate, but not necessarily a maximum, level of healthcare but raises difficult questions about what is fair—and what is bad luck (Reference Beauchamp and Childress8).

The tensions between utilitarianism and egalitarianism can be overstated (Reference Beauchamp and Childress8). Many utilitarians accept that social values should be incorporated into their approach to distributive justice; and qualified egalitarians accept the concept of opportunity costs. There is, though, no formal synthesis of these two approaches to distributive justice. At NICE, an attempt has been made to capture the social values of the general public through its Citizens Council.

Capturing Social Values

Social values take account of the ethical principles, preferences, culture, and aspirations that should underpin the nature and extent of the care provided by a healthcare system (Reference Rawlins7). These include the answer to questions such as whether special priority should be given to children and young people? Whether a healthcare system should be prepared to pay premium prices for drugs to treat very rare serious diseases. Who, though, should decide what social values should HTA agencies and decision makers adopt in prioritizing healthcare?

It might be argued that it should be the role of either Parliament or the government of the day to determine the social values adopted in the provision of healthcare (Reference Rawlins7). Parliament after all makes laws, raises taxes, and decides on how tax revenues should be spent. Whether it has any special legitimacy to make social value judgments for a healthcare system, even if (as in the United Kingdom) it is funded from general taxation, is unclear. Experience in the United Kingdom, at least, suggests that politicians find it extraordinarily difficult to make such decisions in the face of electoral pressures.

Public meetings are a time-honored way in the British National Health Service to sound out public opinion (Reference Rawlins7). They provide little opportunity, however, for reflection or deliberation and attendees are usually dominated by those with a vested interest in the particular issue under discussion (Reference Rawlins7).

Opinion polls and surveys, when conducted competently, can elicit the public's immediate preferences on particular issues. Responses, however, may be colored by inaccurate media activity, and there is no opportunity for discussion or considered thought (Reference Rawlins7). Replies, moreover, are exquisitely sensitive to the precise manner in which the question is phrased or framed. Polls and surveys do not provide the public's considered conclusions, based on deliberation, about the complexities surrounding priority setting in healthcare (Reference Rawlins7).

A better understanding of the reasons underlying the public's immediate preferences might be elicited using focus groups which can provide some insight into why the public believes as it does (Reference Rawlins7). Focus groups, however, are an extension of polling, and the time available (usually not more than 2 to 4 hours) does not allow much opportunity for discussion and deliberation.

NICE's Citizens Council

In an attempt to capture the social values of the British public NICE, in 2002, established a Citizens Council (Reference Littlejohns and Rawlins9). The concept of the Council is based on citizens juries (Reference Rawlins7) and comprises thirty members drawn from the British public. Members were (and still are) recruited by advertisements in the media. Applications are encouraged from anyone, provided that they are not involved, professionally, with healthcare or the healthcare industries. Those appointed reflect the demographic structure of England and Wales, with respect to gender, age, ethnic background, socio-economic status, and disability. Their ages have ranged from 18 to 76 years, and members have included a London cab driver, a scaffolder, a single parent, and a retired airline pilot.

The Council meets for two and a half days, once or twice a year, and considers a particular question that is of concern to the Institute. Some examples of the topics considered are shown on Table 3. Meetings start with an explanation of the question and why the answer is important. Members then have an opportunity to listen, and cross-examine, external experts who provide evidence and views supporting both sides of the question that has posed. For example, in 2011, the Council was asked “Under what circumstances are incentives to promote individual behavior change an acceptable way of promoting the health of the public?” The reason for asking the question was that there is evidence, including from randomized controlled trials, that offering material or financial rewards to individuals to lead healthier life styles (such as abstaining from smoking or elicit drug use) resulted in significant positive results. NICE, though, was unsure whether the use of public funds would meet with general approval. The Council, in its report (10), accepted that that there was strong scientific evidence to support the use of such incentives, but that members were nevertheless uncomfortable with using public funds to “bribe” people in this manner.

Table 3.

Some Topics Considered by NICE's Citizens Council

Decision Making

When making decisions about the findings arising from HTA, judgments have to be made (Reference Rawlins and Culyer11). These fall into two groups: scientific judgments; and social value judgments.

The evidence underpinning any HTA is always incomplete (Reference Rawlins and Culyer11) and scientific judgments are needed if patients are to be provided with appropriate care. These judgments include whether the totality of the evidence for an intervention's effectiveness is sufficiently compelling. They often also encompass issues such as whether, despite the homogeneity of the patients included in the clinical studies, the evidence is generalizable to routine clinical practice or whether changes in the quality of life be adequately captured especially in children where quality of life is notoriously difficult to assess. Or whether, on grounds of clinical or cost-effectiveness, it is appropriate that use be restricted to one or more subgroups of patients. These are scientific judgments that HTA agencies, or their advisory committees, should be able to make.

Social judgments relate to the social, rather than the clinical, sciences (Reference Rawlins and Culyer11). Should an additional year of a child's life be valued differently from that of its parents or grandparents? Are the last few months of life, for people with fatal conditions, so precious that additional healthcare resources should be available to them by increasing the ICER threshold? These judgments are not ones that HTA agencies, or their advisory committees, have any legitimacy to make.

NICE, therefore, has prepared a guideline (12) on social value judgments for use by its staff and its guidance-producing advisory bodies. This guideline brings together the reports of its NICE's Citizens Council together with relevant legislation especially the Equality Act 2010. NICE's program directors are required to confirm that all NICE guidance conforms to the tenets of the guideline or, if there are any departures from it, that the reasons are compelling and explicit. It is important, though, to appreciate that NICE's social values are specific to the United Kingdom and should not be extrapolated more widely.

In the light of the social values developed by the Citizens Council, and enshrined in NICE's social values guideline, there are occasions when the Institute's advisory bodies have exceeded the ICER threshold of £20,000 to £30,000 per QALY gained. Examples (Reference Rawlins, Barnett and Stevens13) include the use of riluzole (with an ICER of £38,000 to £42,000 per QALY gained) to delay the need for tracheostomy in patients with motor neuron disease (amyotrphic lateral sclerosis); or the use of insulin pumps, in children, where the increment in the quality of life was deemed to be unreliable, but nevertheless substantial, and the overall budgetary impact likely to be modest.