Hostname: page-component-74656f9884-j7cmm Total loading time: 0 Render date: 2026-11-07T07:36:39.119Z Has data issue: false hasContentIssue false
  • English
  • Français

Validity of original studies in health technology assessment reports: Significance of standardized assessment and reporting

Published online by Cambridge University Press:  04 August 2005

Franz Porzsolt ,
Hana Kajnar ,
Aziz Awa ,
Margrit Fässler  and
Barbara Herzberger
Franz Porzsolt
Affiliation:
University Hospital Ulm
Hana Kajnar
Affiliation:
Ludwig-Maximilian University Staedtisches Krankenhaus Sindelfingen
Aziz Awa
Affiliation:
Ludwig-Maximilian University
Margrit Fässler
Affiliation:
Ludwig-Maximilian University Georg-August University
Barbara Herzberger
Affiliation:
Ludwig-Maximilian University
Rights & Permissions [Opens in a new window]

Abstract

Objectives: As health technology assessments (HTA) may have considerable impact on health-care decisions, it is essential to guarantee the expected quality of these assessments. Variations in the methods used or lack of transparency can affect the important role of HTA reports.

Methods: In our study, we analyzed the methods used in two corresponding HTA reports to assess the validity of two key papers, which were included in both reports. Also the discussions and the final conclusions of both reports were compared.

Results: The authors of the two HTA reports used different instruments to assess the validity of the original studies. A minor problem is differences that were found in the assessments of identical validity aspects of the included studies and in the information provided in the HTA reports. A more serious problem was found in both HTA reports which identified the weakness of the key papers and expressed these limitations in the discussions (read mainly by scientists) but not in the conclusions (read mainly by policy-makers).

Conclusions: The results of this study may be important for any institution that prepares recommendations for policy-makers. In the case of HTA reports, no new checklists are necessary as the checklist offered by the INAHTA contains the needed information. It may be necessary, however, to consider an “intramural quality board,” which helps the member organizations guarantee both the transparency of the original studies included in HTA reports and the transparency of the HTA report itself.

Keywords

Standards for assessmentStandards for reportingTransparencyValidity of original studies

Information

Type
RESEARCH REPORTS
Information
International Journal of Technology Assessment in Health Care , Volume 21 , Issue 3 , July 2005 , pp. 410 - 413
Copyright
© 2005 Cambridge University Press

Health technology assessments (HTA) are policy-oriented and may have a considerable impact on medical, economic, societal, and ethical consequences of health practices and technologies. This power of HTA reports requires scientists to identify and exclude potential confounders that may affect the validity of HTA reports. Investigating the validity of HTA reports, we realized that different HTA groups use different criteria (3;9) for assessment of the validity of included original papers. This variable may influence the result of the reports and eventually influence the final conclusions. In this study, we compare the methods used for assessment of the validity of the same two original studies included as key studies in two HTA reports and investigate the influence of the assessed validity on the final conclusions of the reports.

METHODS

Two corresponding HTA reports prepared on behalf of the National Institute of Clinical Excellence (NICE) and The Deutsches Institut fuer Medizinische Dokumentation und Information (DIMDI; 6;7) were identified. Both reports were based on the same original papers (1;4). The criteria used by the two groups to assess the validity of the two papers as well as the result of this process were extracted from the HTA reports. In addition, we assessed the validity of the two original studies using the criteria of the Evidence-Based Medicine (EBM) Working Group according to Guyatt et al. and Sackett et al. (2;5). Finally, we investigated the possible effect of the differences in the validity assessments on the final conclusions of the reports.

RESULTS

The list of questions suggested by the EBM Working Group (2;5) to assess the validity of a treatment study (Table 1) was identical to the list of questions used in the NICE report, except two additional questions (presentation of the eligibility criteria and presentation of variance in addition to point estimates for the primary outcome parameters) that were included in the NICE report.

Compared with the EBM list, the DIMDI report (7) asked 18 additional questions shown in Table 2 but did not include questions concerning blinding of patients and physicians, intention-to-treat analysis, and sufficient follow-up observation of the patients.

The catalogue used in the NICE report did not include questions concerning blinding of physicians (only blinding of outcome assessors and patients), sufficient follow-up observation of patients, and the similar treatment of patients, aside from the experimental therapy (Tables 3 and 4).

In addition to the structural variance of the instruments used, an operational variance is recognized, as some of the questions that are identical in all three checklists are answered differently (Tables 3 and 4).

The validity of both key papers demonstrates considerable limitations. In the study by Manns et al. (4), the requirements of correct randomization, concealment of the randomization plan, blinding of physicians, patients and assessors, inclusion of all patients in the analysis, intention-to-treat analysis, the comparability of the groups, and the same treatment of all patients were not or only partially fulfilled. In the paper by Fried et al. (1), the requirements on concealment of the randomization plan; blinding of the physicians, patients, and assessors; inclusion of all patients in the analysis; intention-to-treat analysis; comparability of the groups; and the same treatment of all patients were also not or only partially fulfilled.

In summary, the available data (1;4) cannot support the superiority of therapy with peginterferon and ribavirin over conventional therapy with interferon and ribavirin. There is considerable lack of transparency in both papers (1;4), which could possibly be clarified by examining the original data. Without this clarification, we rank neither of these studies as sufficiently valid to confirm the postulated difference.

In the discussion part of both HTA reports, the authors clearly admit to “limitations” of the validity of the studies included. This important statements, however, are not repeated in the conclusions parts of the reports.

DISCUSSION

In this study, we provide evidence that the authors of HTA reports are using different instruments to assess the validity of the original studies that are included in their HTA reports. We found also evidence that the answers given by different scientists to identical questions—concerning the validity of the studies included—are not necessarily identical, and last but not least found—in both investigated HTA reports—differences in the information provided in the discussion part and in the conclusions.

The last point is most critical, as it may affect the credibility and societal role of HT assessments. It is essential that the scientific community itself sets the quality standards for their products and guarantees the observation of these standards. This statement is true for all types of services of the medical community. More than 10 years ago, an economist from the U.S. general accounting office who attended our conference on “Goals of Palliative Cancer Therapy” predicted:

If physicians will not decide on how to decide, others will…. If the profession does not develop a strategy for resolving the crisis, the courts and legislatures will become increasingly involved in making clinical decisions (8).

At first glance, it seems obvious that an internationally recognized checklist would be helpful in the evaluation of the validity of original papers to achieve uniformity of methodology. The more details of this topic are discussed, the more convinced we are that the available INAHTA checklist contains all important aspects to avoid the three problems described in this communication. The checklist recommends an external review (§4 of the checklist) and emphasizes the important difference that policy-makers, in contrast to scientists, will read only the summary/conclusion of the report (§5). Information that is not expressed in the summary/conclusions will not be recognized by policy-makers. A description of the quality of studies included (§11), the clear presentation of the results (§12), and the interpretation of data (§13), as well as clear conclusions (§13) are also requested in the INAHTA checklist.

Additional recommendations that deal with more details will make the process more complicated but not necessarily better. It is probably sufficient if we, the members of the medical community, guarantee that the available recommendations will be observed and applied. Misunderstood politeness that tolerates incorrectness will support individual but not societal interests. If the self-control of our profession fails to meet the expectations of the society, our profession will loose its autonomy.

CONTACT INFORMATION

Franz Porzsolt, MD, PhD, Head (), Clinical Economics, University Hospital Ulm, Frauensteige 6, 89075 Ulm, Germany

Hana Kajnar, MD, Scientific Assistant, Human Science Center, Ludwig-Maximilian University, Goethestr. 31, 80366 Munich, Germany; Medical Doctor, Department of Internal Medicine, Staedtisches Krankenhaus Sindelfingen, Arthur-Gruber-Str. 70, 71065 Sindelfingen, Germany

Aziz Awa, MD, Scientific Assistant (), Human Science Center, Ludwig-Maximilian University, Goethestr. 31, 80366 Munich, Germany

Margrit Fässler, MD, Scientific Assistant (), Human Science Center, Ludwig-Maximilian University, Goethestr. 31, 80366 Munich, Germany; Scientific Assistant, Department of Medical Ethics and History of Medicine, Georg-August University, Humboldtallee 36, 37073 Goettingen, Germany

Barbara Herzberger, MD, Scientific Assistant () Human Science Center, Ludwig-Maximilian University, Goethestr. 31, 80366 Munich, Germany

References

Fried M, Shiffman M, Reddy R, et al. 2002 Peginterferon alfa-2a plus ribavirin for chronic hepatitis C virus infection. N Engl J Med. 347: 975– 982.Google Scholar
Guyatt GH, Sackett DL, Cook DJ. 1994 Users' guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. JAMA. 271: 59– 63.Google Scholar
Jadad A, Moore A, Carroll D, et al. 1996 Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Control Clin Trials. 17: 1– 12.Google Scholar
Manns MP, McHutchison J, Gordon S, et al. 2001 Peginterferon alfa-2b plus ribavirin compared for initial treatment of chronic hepatitis C: A randomised trial. Lancet. 358: 958– 965.Google Scholar
Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. 2000. Evidence-based medicine, how to practice and teach EBM, 2nd ed. Toronto: Churchill Livingstone;
The British HTA Report commissioned by NICE contains three separate components (separate papers): Shepherd J, Waugh N, Hewitson P. 2000 Combination therapy (interferon alfa and ribavirin) in the treatment of chronic hepatitis C: a rap id and systematic review. Health Technol Assess. 4: 1– 67.Google Scholar
Technology Appraisal Guidance 75. 2004 Interferon alfa (pegylated and non-pegylated) and ribavirin for the treatment for chronic hepatitis C (review and extension of Technology Appraisal Guidance No. 14 issued 2000). London: the National Institute for Clinical Excellence; January Shepherd J, Brodin H, Cave C, et al. (Southampton Health Technology Assessment Centre, Wessex Institute for Health Research and Development). Pegylated interferon alpha 2a and 2b in combination with ribavirin in the treatment of chronic hepatitis C: A systematic review. Technology assessment report commissioned by the HTA Programme on behalf of The National Institute for Clinical Excellence. London: NICE; 2003.
Siebert U, Sroczynski G. 2003 Antivirale Therapie bei Patienten mit chronischer Hepatitis C in Deutschland. Medizinische und ökonomische Evaluation der initialen Kombinationstherapie mit Interferon/Peginterferon und Ribavirin; Vol 8. DAHTA @ DIMDI;
Silberman G. 1993 Cancer Palliation: Economic and societal implications. Cancer Treat Rev. 19 (Suppl A): 97– 102.Google Scholar
Verhagen A, de Vet H, de Bie R, et al. 1998 The Delphi List: A criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol. 51: 1235– 1241.Google Scholar
Figure 0

List of Questions for the Validation of Therapy Studies in Accordance with Sackett/Guyatt (2, 5)

Figure 1

Additional Points in The Deutsches Institut fuer Medizinische Dokumentation und Information (DIMDI) Compared to Sackett/Guyatt (2,5)

Figure 2

Items Used to Assess the Quality of the Paper by Manns et al. (4)a

Figure 3

Items Used to Assess the Quality of the Paper by Fried et al. (1)a