The article “Key principles for the improved conduct of health technology assessments for resource allocation decisions” presents fifteen principles for health technology assessment. Many of these cannot be disputed, and application of the principles as stated would undoubtedly improve HTA as it is developing in the world at large. My question is, are these the most important principles? The document does not really try to answer this question.
My main problem with the paper is with the statement “for Resource Allocation Decisions” in its title and how this is presented in the introduction and interpreted in the entire article. This statement is presented as if the principles are addressed to a subset (how large?) of all HTAs. Resource allocation is not defined explicitly, nor is there a statement concerning what is a resource allocation decision. It is said that the article is intended to apply primarily to HTAs that “are linked to, or include a particular resource allocation decision.” The key question in such decisions is characterized as “is the technology worth it?” Examples given are development of drug formularies or coverage decisions concerning pharmaceuticals. Implicitly, by definition, the group seems to consider that all resource allocation decisions require economic analysis, or conversely, that HTA decisions that do not include economic analysis do not deal with resource allocation decisions. Is this true? Is it a fair representation of the authors' perspectives? The article does not give a clear answer to these questions.
In the conclusion of the paper, the statement is made that the paper shows that it is possible to develop a set of “key principles” for HTA. Now the emphasis seems to be on all HTA, not just those associated with resource allocation decisions. And there are several reassuring statements: HTA differs greatly between places, HTA is a very complicated process, and so on. But the paper has developed the case mainly related to explicit resource allocation decisions, such as drug formularies and insurance coverage decisions. It has not explored the many other uses of HTA. It has not made the case that HTA can generally be improved through these principles.
ARE THE PRINCIPLES REALISTIC TO ALL THE WORLD OF HTA?
In 1980, when I was Manager of its Health Program, the Office of Technology Assessment published a report on cost-effectiveness analysis (7). An important goal of this report was to develop a set of ten key principles for all cost-effectiveness studies, quite similar to those presented by Drummond et al. in 1987 (Reference Drummond, Stoddart and Torrance3). Bryan Luce, from the present group, was instrumental in developing the OTA principles.
The OTA report was mainly concerned with stating a set of principles that were general enough so that even the brief analyses often used to help policy makers and managers make decisions could be based on them. I believe that that list of principles was quite useful, at least potentially. I remember that an administrator/consultant on the advisory panel for the study stated that economic analysis was used all the time, but that most was of the “back of the envelope” type of analysis. He was interested in seeing if improved standards could help these types of analyses, and at the end of the assessment, he said that they would be helpful. But an important reason that such principles can be useful is because health economics is a discipline.
HTA is not a discipline. As the definition in the authors' paper says, HTA is a multidisciplinary activity. To be fully useful to all HTAs, the principles must be drawn from all disciplines that may be working on a particular HTA, or must be general enough to deal with problems addressed by different disciplines. That is certainly not true in the case of this paper, where the emphasis is on economic analysis.
Furthermore, HTA is a very pragmatic activity. Pragmatism is inherent in a field that is a form of policy analysis, as defined by INAHTA and acknowledged in the paper. The particular policy issue may be concerned with cost-effectiveness of an intervention. But it may be concerned only with the ethical implications of the technology. Or, perhaps, only or mainly with the technical characteristics of the technology. It is obviously a quite challenging task to develop a set of principles for a field of such scope and complexity.
The EUR-ASSESS project tried to do that. The EUR-ASSESS project consisted of almost 100 individuals engaged in HTA in Europe, mostly working in public sector or publicly-funded HTA agencies and programs. The eleven-member methodological subgroup (Reference Liberati, Sheldon and Banta8) of the project considered the question, could a common framework for all HTAs be developed? Thirteen general principles were stated. They have some overlap with those stated here, but included other principles addressed to issues not even considered in this paper. These included the necessity for presenting information on the stage of development and use of the technology, technical considerations (in the case of medical equipment, e.g.), qualitative synthesis of available information, social and psychological effects, effects on the organization of health services, and ethics (Reference Liberati, Sheldon and Banta8). For the larger group of HTA studies, I think these principles are considerably more interesting and useful than those presented here. (The EUR-ASSESS report is not mentioned or referenced in this paper.).
One problem is the make-up of the present group. As far as I am aware, Bryan Luce is the only person with substantial experience working in a public HTA program. A fuller representation of those working in “public” HTA might have helped in developing and illustrating the principles.
Most HTAs could not meet these standards at this time. If the standards were seriously used, many fewer assessments would be the result. These assessments might be higher in quality in an academic sense, but they would be less useful because of not being timely. When possible, HTA staff uses substantial time and resources on assessments. It is often just not possible.
THE FOCUS OF MOST HTA
In my experience, the focus of most HTAs is efficacy/effectiveness and safety, or as called in this report, “clinical issues.” As Archie Cochrane said, in his 1972 book, “The most important type of inefficiency is really a combination of two separate groups, the use of ineffective therapies and the use of effective therapies at the wrong time (Reference Cochrane1, p. 31).” I believe that this is still true.
Thus, as noted in this article, such important HTA activities as the Preventive Services Task Force have not considered costs or cost-effectiveness, but have tried to do a thorough analysis of efficacy (12).
The paper does not pay a great deal of attention to efficacy. It seems to take for granted that problems from efficacy have already been solved, or perhaps that they will be solved in a cost-effectiveness analysis. Neither of these points is likely to be true.
ECONOMIC ANALYSIS IN HTA
Probably, only in the area of pharmaceuticals has there been a concerted effort to develop cost-effectiveness studies for reasons of resource allocation or “rationing.” The authors note several reasons that this is true. One reason not mentioned is that in most areas of health care, cost-effectiveness information is just not available.
Sonnad et al. (Reference Sonnad, Greenberg, Rosen and Neumann11) carried out a systematic review of the literature from 1976 to 2001 using as inclusion criteria the guidelines of the Panel on Cost-Effectiveness Analysis in Health and Medicine (Reference Gold, Siegel, Russel and Weinstein4) for cost-utility studies; they found only 539 studies that adhered to these guidelines. The question is, is this a lot or a few? If we consider the number of 10,000 new randomized controlled trials (RCTs) that are added annually to the database of the Cochrane Collaboration, which contains as many as 1 million RCTs (2), we could easily conclude that economic evaluation is of minor importance .
As the U.S. Prevention Task Force observed (12, p. 91): “Cost-effectiveness studies are currently available on many health care services. . .. A much larger group of services remains for which cost-effectiveness is not yet established. Information on costs and outcomes is inadequate for many interventions. For others, the cost-effectiveness analyses have not been done, or their quality is insufficient to provide conclusive evidence. Finally, the variation in cost-effectiveness methodology often makes it difficult to take cost-effectiveness results at face value.” Similar comments are to be found in reports from the United Kingdom (9;10) and Canada (Reference Goldsmith, Henderson and Hurley5), noting the disappointing lack of comparable studies of cost-effectiveness of preventive interventions.
DRUGS VERSUS OTHER TECHNOLOGIES
The authors make an important point that drugs should often be compared with others types of technology in an HTA. This seems obviously true. However, the key fact that they do not acknowledge is that what they call “public HTA,” mostly represented by HTA agencies and drug coverage programs in countries outside the United States, have been brought into existence by policy makers. The considerations concerning whether drugs should be evaluated along with other interventions is not one that those working in HTA can usually make.
Furthermore, as stated just above, such comparisons could usually not be done. In the area of “pharmacoeconomics,” the pharmaceutical industry often funds studies of cost-effectiveness of drugs hoping to gain a positive coverage decision. It is doubtful if the pharmaceutical industry would give substantial funding to other HTA activities.
PRIORITY SETTING
The paper gives useful comments on the need for priority setting. The EUR-ASSESS report included attention to this issue as a key one for HTA (Reference Henshall, Oortwijn and Stevens6).
Principle 4 on priority setting gives some useful examples of priority setting activities. It then makes the statement that many HTA program use a priority setting process that is ill-defined and unclear. They give no examples to illustrate such an ill-defined process and present no references to support the point. I think it is an overstatement to say “in many HTA programs.”
THE QUESTION OF SCOPE AND SCALE
The authors do not effectively address the reality of HTA as practiced in the public HTA programs. These programs are generally under a great deal of pressure to give answers under tight deadlines. As the authors themselves acknowledge, timeliness is key.
These pressures have led many HTA agencies (and others) to do brief assessments or “quick and dirty” assessments when the policy maker's need for information is pressing. HTA professionals know that these HTAs are not optimal. But I believe, and I think that most public HTA professionals believe, that a “quick and dirty” assessment is better than no assessment at all. The alternative would often be a purely political decision. Or a clinical decision without a clear reference to the evidence.
THE ISSUE OF FEEDBACK OR EVALUATION
Feedback is obviously key for the success of HTA. The paper rightly emphasizes this issue. Relatively few evaluations of the impact of reports have been done. One important reason for paucity of evaluations is that HTA programs are usually under a great deal of pressure to produce results, and the leisure for doing evaluation of impact does not exist.
Another problem with evaluation is that it is difficult to organize a useful evaluation that meets any sort of quality standard. A well-known expert in evaluation was asked to give comments on evaluations of HTA reports that had been carried out. He derided the effort, stating that they were too low in quality to be useful. But his standard seemed to be research at the level of an RCT or its equivalent. The expert could not answer the question concerning how an RCT could be done when one is trying to find out if the Ministry of Health has found an assessment useful. The kind of rigor sought by experts in evaluation does not seem feasible in the real world of HTA.
I agree that this is an important and pressing issue. But the solution is difficult to see.
THE ISSUE OF STAKEHOLDER INVOLVEMENT
One of the most valuable parts of the paper is its repeated emphasis on the importance of stakeholder involvement in HTA from the inception and design of an assessment, to the actual carrying out of the assessment, and to the implementation of assessment results. As noted, this is seldom done. One problem here is that it seems to me that the term “stakeholder” is too large a grab-bag of reviewers. I would have preferred more emphasis on policymakers.
CONCLUSION
In summary, the document has many good suggestions, and interesting examples, but the authors need to clarity what they are addressing. If they are addressing the relatively small part of HTA that is evaluating pharmaceuticals for such decisions as coverage and drug formularies, that should be clearly and explicitly stated.
If the intention is to try to apply these principles to all or most HTAs, I think that this can only be usefully done with substantial additional work, including a detailed review and comment by people thoroughly familiar with the broad sweep of HTA. In the meantime, I think that the EUR-ASSESS principles offer the best source of guidelines for that purpose available today.
You have
Access
The article “Key principles for the improved conduct of health technology assessments for resource allocation decisions” presents fifteen principles for health technology assessment. Many of these cannot be disputed, and application of the principles as stated would undoubtedly improve HTA as it is developing in the world at large. My question is, are these the most important principles? The document does not really try to answer this question.
My main problem with the paper is with the statement “for Resource Allocation Decisions” in its title and how this is presented in the introduction and interpreted in the entire article. This statement is presented as if the principles are addressed to a subset (how large?) of all HTAs. Resource allocation is not defined explicitly, nor is there a statement concerning what is a resource allocation decision. It is said that the article is intended to apply primarily to HTAs that “are linked to, or include a particular resource allocation decision.” The key question in such decisions is characterized as “is the technology worth it?” Examples given are development of drug formularies or coverage decisions concerning pharmaceuticals. Implicitly, by definition, the group seems to consider that all resource allocation decisions require economic analysis, or conversely, that HTA decisions that do not include economic analysis do not deal with resource allocation decisions. Is this true? Is it a fair representation of the authors' perspectives? The article does not give a clear answer to these questions.
In the conclusion of the paper, the statement is made that the paper shows that it is possible to develop a set of “key principles” for HTA. Now the emphasis seems to be on all HTA, not just those associated with resource allocation decisions. And there are several reassuring statements: HTA differs greatly between places, HTA is a very complicated process, and so on. But the paper has developed the case mainly related to explicit resource allocation decisions, such as drug formularies and insurance coverage decisions. It has not explored the many other uses of HTA. It has not made the case that HTA can generally be improved through these principles.
ARE THE PRINCIPLES REALISTIC TO ALL THE WORLD OF HTA?
In 1980, when I was Manager of its Health Program, the Office of Technology Assessment published a report on cost-effectiveness analysis (7). An important goal of this report was to develop a set of ten key principles for all cost-effectiveness studies, quite similar to those presented by Drummond et al. in 1987 (Reference Drummond, Stoddart and Torrance3). Bryan Luce, from the present group, was instrumental in developing the OTA principles.
The OTA report was mainly concerned with stating a set of principles that were general enough so that even the brief analyses often used to help policy makers and managers make decisions could be based on them. I believe that that list of principles was quite useful, at least potentially. I remember that an administrator/consultant on the advisory panel for the study stated that economic analysis was used all the time, but that most was of the “back of the envelope” type of analysis. He was interested in seeing if improved standards could help these types of analyses, and at the end of the assessment, he said that they would be helpful. But an important reason that such principles can be useful is because health economics is a discipline.
HTA is not a discipline. As the definition in the authors' paper says, HTA is a multidisciplinary activity. To be fully useful to all HTAs, the principles must be drawn from all disciplines that may be working on a particular HTA, or must be general enough to deal with problems addressed by different disciplines. That is certainly not true in the case of this paper, where the emphasis is on economic analysis.
Furthermore, HTA is a very pragmatic activity. Pragmatism is inherent in a field that is a form of policy analysis, as defined by INAHTA and acknowledged in the paper. The particular policy issue may be concerned with cost-effectiveness of an intervention. But it may be concerned only with the ethical implications of the technology. Or, perhaps, only or mainly with the technical characteristics of the technology. It is obviously a quite challenging task to develop a set of principles for a field of such scope and complexity.
The EUR-ASSESS project tried to do that. The EUR-ASSESS project consisted of almost 100 individuals engaged in HTA in Europe, mostly working in public sector or publicly-funded HTA agencies and programs. The eleven-member methodological subgroup (Reference Liberati, Sheldon and Banta8) of the project considered the question, could a common framework for all HTAs be developed? Thirteen general principles were stated. They have some overlap with those stated here, but included other principles addressed to issues not even considered in this paper. These included the necessity for presenting information on the stage of development and use of the technology, technical considerations (in the case of medical equipment, e.g.), qualitative synthesis of available information, social and psychological effects, effects on the organization of health services, and ethics (Reference Liberati, Sheldon and Banta8). For the larger group of HTA studies, I think these principles are considerably more interesting and useful than those presented here. (The EUR-ASSESS report is not mentioned or referenced in this paper.).
One problem is the make-up of the present group. As far as I am aware, Bryan Luce is the only person with substantial experience working in a public HTA program. A fuller representation of those working in “public” HTA might have helped in developing and illustrating the principles.
Most HTAs could not meet these standards at this time. If the standards were seriously used, many fewer assessments would be the result. These assessments might be higher in quality in an academic sense, but they would be less useful because of not being timely. When possible, HTA staff uses substantial time and resources on assessments. It is often just not possible.
THE FOCUS OF MOST HTA
In my experience, the focus of most HTAs is efficacy/effectiveness and safety, or as called in this report, “clinical issues.” As Archie Cochrane said, in his 1972 book, “The most important type of inefficiency is really a combination of two separate groups, the use of ineffective therapies and the use of effective therapies at the wrong time (Reference Cochrane1, p. 31).” I believe that this is still true.
Thus, as noted in this article, such important HTA activities as the Preventive Services Task Force have not considered costs or cost-effectiveness, but have tried to do a thorough analysis of efficacy (12).
The paper does not pay a great deal of attention to efficacy. It seems to take for granted that problems from efficacy have already been solved, or perhaps that they will be solved in a cost-effectiveness analysis. Neither of these points is likely to be true.
ECONOMIC ANALYSIS IN HTA
Probably, only in the area of pharmaceuticals has there been a concerted effort to develop cost-effectiveness studies for reasons of resource allocation or “rationing.” The authors note several reasons that this is true. One reason not mentioned is that in most areas of health care, cost-effectiveness information is just not available.
Sonnad et al. (Reference Sonnad, Greenberg, Rosen and Neumann11) carried out a systematic review of the literature from 1976 to 2001 using as inclusion criteria the guidelines of the Panel on Cost-Effectiveness Analysis in Health and Medicine (Reference Gold, Siegel, Russel and Weinstein4) for cost-utility studies; they found only 539 studies that adhered to these guidelines. The question is, is this a lot or a few? If we consider the number of 10,000 new randomized controlled trials (RCTs) that are added annually to the database of the Cochrane Collaboration, which contains as many as 1 million RCTs (2), we could easily conclude that economic evaluation is of minor importance .
As the U.S. Prevention Task Force observed (12, p. 91): “Cost-effectiveness studies are currently available on many health care services. . .. A much larger group of services remains for which cost-effectiveness is not yet established. Information on costs and outcomes is inadequate for many interventions. For others, the cost-effectiveness analyses have not been done, or their quality is insufficient to provide conclusive evidence. Finally, the variation in cost-effectiveness methodology often makes it difficult to take cost-effectiveness results at face value.” Similar comments are to be found in reports from the United Kingdom (9;10) and Canada (Reference Goldsmith, Henderson and Hurley5), noting the disappointing lack of comparable studies of cost-effectiveness of preventive interventions.
DRUGS VERSUS OTHER TECHNOLOGIES
The authors make an important point that drugs should often be compared with others types of technology in an HTA. This seems obviously true. However, the key fact that they do not acknowledge is that what they call “public HTA,” mostly represented by HTA agencies and drug coverage programs in countries outside the United States, have been brought into existence by policy makers. The considerations concerning whether drugs should be evaluated along with other interventions is not one that those working in HTA can usually make.
Furthermore, as stated just above, such comparisons could usually not be done. In the area of “pharmacoeconomics,” the pharmaceutical industry often funds studies of cost-effectiveness of drugs hoping to gain a positive coverage decision. It is doubtful if the pharmaceutical industry would give substantial funding to other HTA activities.
PRIORITY SETTING
The paper gives useful comments on the need for priority setting. The EUR-ASSESS report included attention to this issue as a key one for HTA (Reference Henshall, Oortwijn and Stevens6).
Principle 4 on priority setting gives some useful examples of priority setting activities. It then makes the statement that many HTA program use a priority setting process that is ill-defined and unclear. They give no examples to illustrate such an ill-defined process and present no references to support the point. I think it is an overstatement to say “in many HTA programs.”
THE QUESTION OF SCOPE AND SCALE
The authors do not effectively address the reality of HTA as practiced in the public HTA programs. These programs are generally under a great deal of pressure to give answers under tight deadlines. As the authors themselves acknowledge, timeliness is key.
These pressures have led many HTA agencies (and others) to do brief assessments or “quick and dirty” assessments when the policy maker's need for information is pressing. HTA professionals know that these HTAs are not optimal. But I believe, and I think that most public HTA professionals believe, that a “quick and dirty” assessment is better than no assessment at all. The alternative would often be a purely political decision. Or a clinical decision without a clear reference to the evidence.
THE ISSUE OF FEEDBACK OR EVALUATION
Feedback is obviously key for the success of HTA. The paper rightly emphasizes this issue. Relatively few evaluations of the impact of reports have been done. One important reason for paucity of evaluations is that HTA programs are usually under a great deal of pressure to produce results, and the leisure for doing evaluation of impact does not exist.
Another problem with evaluation is that it is difficult to organize a useful evaluation that meets any sort of quality standard. A well-known expert in evaluation was asked to give comments on evaluations of HTA reports that had been carried out. He derided the effort, stating that they were too low in quality to be useful. But his standard seemed to be research at the level of an RCT or its equivalent. The expert could not answer the question concerning how an RCT could be done when one is trying to find out if the Ministry of Health has found an assessment useful. The kind of rigor sought by experts in evaluation does not seem feasible in the real world of HTA.
I agree that this is an important and pressing issue. But the solution is difficult to see.
THE ISSUE OF STAKEHOLDER INVOLVEMENT
One of the most valuable parts of the paper is its repeated emphasis on the importance of stakeholder involvement in HTA from the inception and design of an assessment, to the actual carrying out of the assessment, and to the implementation of assessment results. As noted, this is seldom done. One problem here is that it seems to me that the term “stakeholder” is too large a grab-bag of reviewers. I would have preferred more emphasis on policymakers.
CONCLUSION
In summary, the document has many good suggestions, and interesting examples, but the authors need to clarity what they are addressing. If they are addressing the relatively small part of HTA that is evaluating pharmaceuticals for such decisions as coverage and drug formularies, that should be clearly and explicitly stated.
If the intention is to try to apply these principles to all or most HTAs, I think that this can only be usefully done with substantial additional work, including a detailed review and comment by people thoroughly familiar with the broad sweep of HTA. In the meantime, I think that the EUR-ASSESS principles offer the best source of guidelines for that purpose available today.