Environmental Context

The HTA Program was established in January 1993 by the AHM to provide RAs, within 1 to 3 months, to inform funding and coverage decisions. In November 1995, the HTA Program moved to an independent, arms-length provincial research funding organization, where the range of assessments was expanded to include full HTAs and other products. In July 2006, the Program moved to an independent, arms-length research organization, the IHE. Despite these major relocations, the AHM has remained the main client of the HTA Program (Figure 1). The political environment in Alberta was relatively stable during this 25-year period, with the government being led by the same political party from 1971 to 2015 and by the same Premier from 1992 to 2006.

Fig. 1.

History of the health technology assessment (HTA) Program in Alberta.

Introduction Phase (1993–1995)

The HTA Program initially produced only one species of rapid response, the TechNote, which was a narrative synthesis of systematically selected evidence from full-text articles conducted by one researcher within 3 months. These were developed as a way of rapidly responding to AHM decision makers who needed quick answers to their policy questions. Given the lack of a definition or universal methodology for RAs among HTA organizations at the time, the methods and format of the TechNotes were informed by the general intent to search for, review, and summarize the best available evidence on the topic of interest as quickly as practicable. The literature searches were performed by the researcher without support from an information specialist or reference management or study selection software, and they were not systematic. Although expert opinion was usually included in the TechNotes, the focus was typically narrow, and data extraction from the included studies was limited. The methodology was neither reported nor standardized, and potential methodological limitations in the TechNotes and their included studies were not reported.

Because the TechNote format was largely created without requester input, difficulties arose in matching requester needs with the final product. This led to the creation of an HTA Request Form, a pro forma used to capture initial discussions, encourage critical evaluation by the requester about what was required, and provide a reference point for later negotiations should there be a discrepancy between AHM expectations and the final product.

Establishment (1996 to June 2006)

When the HTA Program relocated to a research funding organization in 1996, the Program increased its staff and gained access to information specialists and support staff, as well as improved connections with other HTA organizations. Following the development of a collaborative relationship between the HTA Program and the AHM, as well as more positive responses to the TechNotes, considerable effort was invested in advancing RA methods. The time saved by more efficient literature searching and screening processes was re-invested into improving evidence synthesis in the TechNotes, which began to include more comprehensive evidence tables and more detailed evidence syntheses in an effort to make them more useful to the requester.

In tandem with these improvements, there was a push from the requester for more rapidly produced reports. This led to the creation of the QwikNote, which was essentially a list of abstracts of available evidence derived from searches of two major medical literature databases, screened for relevance by a researcher. These were produced within 1 to 2 weeks and became a useful resource for the AHM as pre-assessments for prioritizing policy questions (see Figure 2).

Over time, the RAs began to display phenotypic heterogeneity. The advent of new staff with different perspectives and expertise, combined with researchers’ efforts to simultaneously fulfill requester needs and meet new standards in HTA best practice, led to (confusing) variation among the RAs delivered to the AHM. Consequently, TechNotes became more detailed and started to incorporate elements of quality appraisal not seen hitherto, and elements of information synthesis began to appear in QwikNotes. As a result, decision makers came to expect more detail and in-depth synthesis within unrealistic timeframes from all the Program's products.

To combat these entropic trends, and inspired by a recent publication on best practice in HTA (Reference Busse, Orvain and Velasco9), the HTA Program revamped its product line and formalized a new set of RA “species” based on requester needs. A third level of RA was created later, the CompNote, to formally recognize, define, and categorize the overly detailed TechNotes, which were verging on full HTAs. These new standards helped curb the problem of the AHM expecting a full HTA within a RA timeframe by providing a systematic way of delineating the differences in methods, and consequent level of uncertainty, among the RA product types. The original HTA Request Form was also revised to reflect the increased sophistication of the requester-producer partnership. In contrast to the original form, which was constructed from an academic researcher perspective and focused heavily on methodology, the new form emphasized the decision-maker's perspective, with less upfront focus on the methodological machinery used to construct the reports.

Diversification and Consolidation (July 2006 to 2018)

In July 2006, the Program moved to an independent, arms-length research organization, the IHE, which provided access to health economic expertise and expanded information services support. This amalgamation was likely influenced by the government's need to demonstrate appropriateness in health spending during a time of budgetary constraint by incorporating health economic information in their decision-making processes. The increased access to information services led, over time, to the creation of a new report type, the Scoping Review. These reports were short scoping searches produced by the information specialists without researcher oversight, which freed researchers to concentrate on the more complex reports (see Figure 3).

The HTA Program was also tasked to play a key role in the Alberta Health Technologies Decision Process (AHTDP), a mechanism established by the AHM in 2004 for introducing and diffusing publicly funded, non-pharmaceutical health technologies (Reference Borowski, Brehaut and Hailey10). Through the AHTDP, the AHM exerted a strong “pull” on the information flow between the research community and policy makers. Instead of researchers reactively designing RA products to suit decision-maker needs, the decision makers proactively specified the types of RAs they required. The new STE report debuted in 2006, and comprised three main components: social and system demographics (S), technology effects and effectiveness (T), and economic evaluation (E).

While the STE report matched the comprehensiveness of a full HTA report, it was to be completed as a RA within a 3-month timeframe (Figure 2). Its preparation required multiple researchers in addition to an information specialist, and included external review by a committee of relevant stakeholders. A quick-turnaround STE scoping report was also added to the Program's product list, as the AHM often needed preliminary scoping searches to inform decisions about which topics to assess using the AHTDP. This report type mirrored the Scoping Review, whose genesis had resulted from the increased information services support garnered from the Program's transfer to the IHE (Figure 3).

Fig. 2.

Types of rapid assessments produced over the 25-year time period.

Fig. 3.

Rapid assessment phylogeny over the 25-year time period.

Over the next 5 years, escalating demands and expectations from decision makers, coupled with large variation in the types of questions received through the AHTDP, led the HTA Program to reactively establish three categories of STE reports (Levels 1, 2, and 3) (Figure 3) to better facilitate the structure outlined by the AHM.

Since 2016, with the advent of a new government, the decision processes within the AHM have been re-evaluated and re-designed, the AHTDP was renamed Alberta Health Evidence Reviews in 2017, and the STE report lineage became extinct. This was due in part to the realization by the AHM that the AHTDP process, while helpful for addressing technology adoption questions, was not providing adequate outputs to inform decision making on healthcare service implementation or optimization, and that improvements in topic selection and the scope and scale of HTA products were needed. Issues arising from the health system had become more complex, with an increasing focus on system-wide problems, such as how to optimize healthcare delivery, implement program-level changes, and strike a balance between costs and improvements in health. The fixed S, T, and E structure imposed by the STE reports was not well suited to answering these types of questions. In addition, the STE reports continued to suffer from vestiges of scope creep and “methods bloat”.

Reasons for Adaptive Changes

The HTA environment within and outside of Alberta has changed over the past 25 years. The developmental changes in the RAs completed for the AHM during this time have been driven by the requesters, the researchers, and the external policy environment. This is reflected in the fact that the components of earlier RAs were driven more by what the researchers thought the requester wanted, whereas those of later RAs were informed by requester feedback and were better tailored to their audience. Requesters have become savvier in knowing what they need, and the HTA Program has become more flexible and better able to meet those needs while maintaining best practice standards on the minimum essential components required for a reliable RA.

Changes in RA products have also been fueled by the substantial transformations that occurred in the information technology landscape over the past quarter century. Improvements in literature database interfaces and access, as well as the availability of dedicated information specialist services and data management software, have allowed the HTA Program to expand the type and depth of RA products that can be produced within short timeframes. At the same time, the issues confronting Alberta's health system have also shifted from “simple” funding decisions about single technologies to more complex considerations such as program-level implementation, policy review, and disinvestment. Thus, the demand for RAs increased while their scope became broader, forcing a drive to greater efficiency and more fit-for-purpose products within the HTA Program.

The HTA Program needed to become faster and more flexible in preparing and using evidence, without becoming arbitrary in how this was done. It adapted by altering the product line while maintaining a focus on HTA best practice, and leveraging a critical mass of expertise from its placement within arm's length organizations to form a multidisciplinary staff of experts in general medicine, epidemiology, biomedical engineering, biostatistics, ethics, information science, and health economics. The latter aspect required a concerted effort to overcome the silo effect, particularly between the HTA and health economics disciplines. The HTA Program's ability to call on and quickly mobilize this latent capacity has enabled it to serve as an effective knowledge translation hub, linking AHM decision makers with various health system stakeholders and researchers.

What Works in Alberta: The Dominant RA Phenotypes

The longstanding, symbiotic relationship that developed between the HTA Program and the AHM initially led to increased diversity in the RA lineage, followed by controlled extinction of the less fit (useful) transition species (the STE reports) (Figure 3). It is no coincidence that the STE lineage exhibited convergent characteristics with the TechNote and QwikNote lines despite the fact that the STE lineage was initiated by the requester. This suggests that, within the milieu of Alberta's health system, the RA products that best meet decision maker needs within the constraints of HTA best practice are the Scoping Review, QwikNote, and TechNote, which have been honed through experimentation over 25 years with various methods, formats, reporting styles, and document lengths, and continuous feedback from the AHM. Although the CompNotes have been requested by other stakeholders, none have yet been produced for the AHM; they are an “endangered” species.

The HTA Program's experience highlights the need to have a thorough understanding of the intended audience's requirements and a commitment to scientifically rigorous and transparent methods before undertaking RAs. The multiple product types demonstrate that many different methods may be needed to fulfill requester needs when negotiating the complex, time-critical business of information brokerage. Producers must be flexible in what they can provide, and also be able to help requesters articulate their needs within the bounds of HTA best practice to ensure that their decision-making processes are credible and defensible. In addition, the Alberta experience suggests that successful RA phenotypes arise when both requester and producer perspectives are incorporated; using only one perspective may doom the lineage to extinction.

New environmental niches are emerging in the Alberta health system, particularly with the increased focus on personalized medicine, real-world evidence, and implementation and disinvestment issues. The current RA products of the HTA Program have reached their range limit, and new RA “species” are required that can efficiently answer more complex questions with highly contextualized evidence and information synthesis. Experience over the past 25 years has shown that the most successful speciation in RA products occurs when there is relative balance between the push-pull tension of producers and requesters. The plethora of RA products available across the various health systems demonstrates that each producer-requester partnership results in unique RA products that suit the idiosyncratic needs of each “ecosystem.” Instead of attempting to standardize this diversity, the HTA community is well placed to catalogue and learn from these diverse RA phylogenies, and to provide a quality control function to ensure that RA speciation events produce functional and methodologically robust lineages.