A project has been defined as, “directed work that is aimed at achieving specific goals within a defined budget and schedule” (1). An international project “involves multiple locations, entities, organizations and business units” (Reference Lientz and Rea1) (we take the view that the requirement for multiple locations must also involve multiple countries). Such projects are complicated, due to the large number of organizations, wide purpose and scope, and high cost (Reference Lientz and Rea1).
There have been several European networks and projects for Health Technology Assessment (HTA); EUR-ASSESS 1994–97 (Reference Banta2), HTA Europe 1997–98 (Reference Banta and Oortwijn3), ECHTA/ECHAHI 2000–02 (Reference Jonsson and Banta4), and EUnetHTA 2006–08 (Reference Borlum Kristensen, Makela and Allgurin Neikter5). The International Network of Agencies for Health Technology Assessment (INAHTA) is a nonprofit organization and in 2013 had fifty-seven member organizations from thirty-two countries across the globe (6). HTAsiaLink was established in 2011 and connects nine (in 2013) relevant organizations in Asia (7).
Evaluation is an important facet of such international projects and networks for HTA. Evaluation has been defined as, the “systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy” (Reference Weiss8). Project evaluation allows monitoring of the processes of the project and achievements against specified criteria for success. This enables assessment of the effectiveness and achievements of the project and the formation of “lessons learned” recommendations to inform future projects. Evaluation can reveal what factors are important for the success of international HTA projects.
The European network for HTA Joint Action (EUnetHTA JA) project was a 3-year project spanning 2010 to 2012 (Reference EUnetHTA9). At its start it included thirty-eight government appointed organizations from twenty-six EU Members states, Norway and Croatia. Seventeen umbrella organizations were designated as stakeholders for the EUnetHTA JA, representing industry, patients/consumers, providers and payers. A Stakeholder Forum was set-up to organize the contribution of stakeholders to the project. Of the seventeen stakeholder organizations, twelve were appointed as members of this EUnetHTA JA Stakeholder Forum and five were not successful in gaining a place, although their views were representable by a representative member of their interest group.
The overarching objective of the EUnetHTA JA project was to “ establish an effective and sustainable HTA collaboration in Europe that brings added value at the regional, national and European level.” The project had three specific objectives: development of a general strategy and a business model for sustainable European collaboration on HTA; development of HTA tools and methods; and application and field testing of developed tools and methods. The project had a central Secretariat and, in common with other European projects, the work was subdivided into distinct workstreams (in this case, eight), referred to as workpackages.
The aim of this article was to identify the factors that are important for evaluating the success of international HTA projects, using the EUnetHTA JA project as an exemplar.
METHODS
A comprehensive literature review was undertaken to identify appropriate methods for evaluating large scale collaborative projects. The first stage in evaluating an international network or project for HTA is to create key performance indicators for the project and then a Project Evaluation Plan should be devised which specifies the main purpose of the evaluation. Such critical reflection can be performed retrospectively (after the project has ended) or ideally prospectively (designed at the start of the project). Evaluation can be internal (performed by staff directly involved in the project work) or external (performed by an external expert who is not involved in the project) (Reference Weiss8).
The outcome and processes of the project can both be evaluated (Reference Weiss8):
-
i. The impact of a project can be assessed by an outcome evaluation, which identifies the success of delivering the stated project deliverables,
-
ii. The effectiveness of a project can be evaluated by its processes (identifying the effectiveness of the processes used during the project).
Evaluation should include consideration of the views of both project participants and external stakeholders. Several research methods can been used to perform an evaluation of an international HTA project, and these are described in Table 1.
Possible research methods for evaluating international projects
Key performance indicators were developed for the EUnetHTA JA project by literature review:
-
• Project impact; production of deliverables according to the three year workplan and Grant Agreement, objectives (as defined in the Grant Agreement) met and additional “added value” generated,
-
• Project effectiveness; effective communication within the project, effective project administration by the Secretariat, optimal involvement of external stakeholder and good management of the constituent workpackages,
-
• Lessons learned; progress from the predecessor EUnetHTA 2006–08 project.
A prospective methodology is the gold-standard for evaluation research and was used to evaluate the Joint Action. As recommended in conducting evaluations of European projects, the evaluation plan was a key component and integrated within the JA project from the beginning. The evaluation participants were in two survey populations of project participants who were members of EUnetHTA JA partner organizations and external stakeholders with an interest in the project (from industry, patients and consumers, providers, and payer groups).
It was necessary to select which approach would be the most appropriate and feasible to evaluate the EUnetHTA JA within its economic, geographic and time restraints, and the resources assigned to the evaluators. The methods chosen were self-completion questionnaires and documentary review. Annual questionnaires were sent to project participants and external stakeholders and key project documents reviewed.
The following strategy was used for the questionnaires:
-
• The questionnaire was designed according to good practice (e.g., it was made visually attractive, there was a logical sequence of questions, all appropriate answer options were provided and double-negative wording avoided etc.) Some “core questions” remained the same each year, with other questions added as relevant to the current stage of the project.
-
• The questionnaire was electronically distributed, to allow respondents to complete it conveniently.
-
• The questionnaires were presented as obligatory to EUnetHTA JA participants.
-
• An accurate sampling frame for the recipients was obtained to ensure that nonresponse could not be attributed to using an incorrect email address. It was feasible to survey the entire population of project participants and stakeholders and, therefore, no sample selection was required.
-
• The questionnaires were piloted for quality assurance to ensure the questions contained the correct spelling, were grammatically correct, followed a logical sequence and were understood as intended by the evaluators.
-
• A prenotification email was sent to recipients 1 week before the questionnaire send-out to notify them to expect the questionnaire and informed them about the importance of completing it.
-
• Two follow-up questionnaires were sent to nonrespondents at 3-weekly intervals.
-
• Although a coding number was needed to allow follow-up of nonresponders, respondents were assured of confidentiality within the evaluation team. Respondents were not identifiable to their responses in the reports produced.
It is worth discussing the piloting in more detail here. With a multinational project with many participants, few of which have English as their first language, it is imperative that the language used in assessment tools is absolutely clear. We assured this in the EUnetHTA JA evaluation by (i) piloting all the assessment tools with staff internal to our organization with a non-English first language (over time the availability of assessors varied, but included French, German and Spanish), (ii) piloting with individuals from several countries engaged in the project, and (iii) undertaking a final check with the project's Executive Committee which included native speakers from most European language families.
The analysis depended on the type of questions used;
-
• “Closed” questions had answer-possibilities predefined and this quantitative data was analyzed with the aid of the computer software package Statistical Package for the Social Sciences (SPSS)®. Descriptive statistics were included for categorical data, showing frequency and percentages.
-
• “Open” or “free response” questions allowed respondents to provide their own answers to express their thoughts in their own language. This qualitative data was analyzed with the aid of the computer software package NVivo®.
RESULTS
Response Rate
It was interesting to note that there was a high turnover of staff during the project – one-third of the project participants in 2011 (the second year of the project) were new. The changing number of project participants is illustrated in Figure 1.Response rates of between 86 percent and 88 percent were achieved for the three annual questionnaires sent to project participants (although not all respondents answered every question).
Project participant members in different years of the EUnetHTA JA project.
An individual was contacted to provide a combined response from each umbrella stakeholder organization. Only members of the Stakeholder Forum were surveyed in the interim year of the project (2011). Response rates for questionnaires sent to external stakeholders ranged from 65 percent to 88 percent over the three years of the project (although not all respondents answered every question). The overall response in each year is shown. The overall response rates to participants’ and stakeholders’ questionnaires in each year are shown below in Table 2.
Response Rates to Participants’ and Stakeholders’ Questionnaires in Each Project Year
Evaluation / Documentary Review Results
Project Impact
The deliverables of tools to help the production of HTA reports, were produced by the end of the project; an online Tool & Service for producing, publishing, storing and retrieving HTA information - The HTA Core Model, an HTA Core Model on screening, a set of two Core HTAs, an operational web-based toolkit including database containing information on evidence generation on new technologies and a quarterly communication protocol for information flow on on-going/planned national assessments of same technologies. The methodological guidance appropriate for the assessment of relative effectiveness of pharmaceuticals was delivered in March 2013 – 3 months after the project's end. The proportion of participants who predicted they would find HTA tools useful in practice varied between two-fifths and one-half (depending on the tool). The other project-management deliverables were all delivered; the Information Management System and related documentation, processes and policies, a Communication and Dissemination Plan and the Stakeholder Policy. Another of the project's objectives was to produce a business model for sustainability, which was also delivered according to plan.
Delivery of the business plan and of the HTA tools and methods fulfilled two of the objectives. However, although a description of a comparison of a national report with the Core HTA Model is included in this special edition, there was limited opportunity to apply the HTA tools in practice and to test them.
The three most useful positive project attributes according to project participants (in 2012) were; “networking with colleagues”, “information sharing”, and “increased awareness of HTA developments”. When asked six months before the end of the project what they would find most useful, the facet rated the highest by both participants and stakeholders was “Networking with contacts made from participating in the EUnetHTA JA”. Three-fifths of participants and over two-thirds of stakeholders, rated this very highly, with the vast majority of respondents finding it of at least some use.
Project Effectiveness
Overall, communication within the project seemed effective and approximately three-quarters of respondents had not experienced any significant problems when communicating in English during the project. However, there was recognition of the inherent difficulty in communicating in English, which was not most participants’ native language.
The coordinating Secretariat was reported to have provided effective administrative support by approximately three-fifths of project participants. Some suggestions for improvement included facilitating relationships, providing greater feedback and providing advice about monthly budgeting.
Three-quarters of stakeholder organizations thought the Stakeholder Forum had fulfilled its purpose and approximately two-thirds thought that the project had achieved what their organization had hoped. One-half of stakeholders disagreed that their organization's expertise had been appropriately used in the EUnetHTA JA project.
Overall the workpackages seemed to have been managed appropriately. Some concern was expressed about possible overlapping of work, and the importance of avoiding duplication emphasized.
Lessons Learned
The recommendations from the evaluation report of the previous project (Reference Koster16) had been followed, where applicable. These included maintaining a Secretariat, to continue developing tools, to involve people in the work, use face-to-face meetings and communicate in English. However, to “evaluate the tools in real work settings” did not seem to have been followed. Summary results for the project's key indicators are shown below in Table 3.
Summary Results for the Project's Key Indicators
DISCUSSION
To achieve success in an international project on HTA there must be a clear purpose, specific plans, commitment, open communication, respect & trust, collaboration, political support, clear roles & responsibility and an effective leadership style (Reference Turner and Zolin17). Other factors that lead to the success of a complex project include (Reference Koster16) clarity of the goals and commitment to them by the project team, establishing smooth communications with supporting infrastructure, recruiting project team members with sufficient technical capabilities, context of the project considered and a supportive project culture.
Key performance indicators were defined for the EUnetHTA JA project and these were assessed by evaluation questionnaires and documentary review.
The project aimed to establish an effective and sustainable HTA collaboration in Europe that brings added value at the regional, national and European level. Project participants and stakeholders understood that this would have been achieved when a formal EU HTA agency or network not dependent on project funding had been formed, collaboration was achieved, EUnetHTA tools were adopted at a regional or national level, a library of HTA reports and topics was available, HTA was included in decision-making and impact was evaluated. However, the fact that a follow-up EUnetHTA JA2 project began in October 2012 indicated that this had not yet been achieved. An “effective and sustainable HTA collaboration in Europe that brings added value at the regional, national and European level” was not yet formed. This follow-up project, reliant on project funding from the EU Commission, was set-up after EUnetHTA JA—the EUnetHTA JA2 project (2012–15) (18). In this respect Article 15 of the European Directive 2011/24/EU of the European Parliament and of the Council of 9 March 2011 on the application of patients’ rights in cross-border healthcare (19) is important—as it is this which makes funds available for an on-going network. In essence the EUnetHTA Joint Action could not meet its overall objective because this directive was not promulgated in time. This objective has now been carried over to the EUnetHTA JA2 project, “To develop a general strategy, principles and an implementation proposal for a sustainable European HTA collaboration according to the requirements of Article 15 of the Directive for cross-border healthcare.” (18).
One of the objectives of the project was to produce deliverables by its end, including tools to help in the production of HTAs. All of the tools proposed were delivered by target of the end of the project, apart from the methodological guidance which was finalized 3 months afterward. It would be useful to follow-up actual use of the tools in practice, and evaluation of their usefulness. It is important that tools are trialled in “real-world” situations and necessary improvements made. It is hoped that this will be achieved during EUnetHTA JA2.
The main added benefits of the project were in networking with colleagues in a European network and this could possibly lead to future collaborations, information sharing and increased awareness of HTA developments. It is recommended that evaluation of the EUnetHTA JA2 includes consideration about the tangible benefits of networking in international HTA projects. This could include a case-study approach to demonstrate the practical benefits of networking. Hopefully the benefits of EUnetHTA JA2 will ultimately lead to measurable improvements in population health.
In any international project, communication is likely to be difficult because all members do not share a native language. It was, therefore, encouraging that almost three-quarters of participants had not experienced a significant problem in communicating in English. Participants working on tasks in the individual workpackages are examples of virtual teams, “..teams of workers who are dispersed across geographical, temporal, and organizational boundaries, yet collaborate using information and telecommunications technology” (Reference Baskerville and Nanhakumar20). Due to the large and complex structure of the project, it was essential that communication was optimal and to help facilitate this a large number of different types of methods were used. Of these, the apparent most useful mechanism was face-to-face meetings and this was very much appreciated by two-thirds of participants. In the hierarchy of communication methods, “in person” communication has been ranked as the “gold standard” method because it is possible to see a person's body language and catch the tone of voice and any specific nuances (Reference Lientz and Rea1). A preference for face-to-face meetings goes hand-in-hand with the importance of networking and reinforces the importance of participants meeting in person as opposed to working solely in virtual teams. However, the benefits of this communication method need to be balanced with the inherent implications in terms of financial and logistical costs. Therefore, it is important that face-to-face meetings are conducted in an optimal manner.
A prerequisite of the EUnetHTA JA project was an acknowledgement of the importance of communicating with stakeholders (21). Knowledge about relevant external stakeholders is vital for international projects. It is important that their influence level is assessed and their objectives in relation to project strategies and deliverables identified. It was encouraging that overall the mechanism for organizing stakeholder input (the Stakeholder Forum) was viewed positively. However, there seemed to be a view that the stakeholders had more knowledge than the project had tapped into, and this could be further explored in EUnetHTA JA2.
The recommendations from the evaluation report of the previous project (Reference Edwards, Roberts and Clarke22), where applicable, had been followed. These included maintaining a Secretariat, to continue developing tools, to involve people in the work, use face-to-face meetings and communicate in English. However, to “evaluate the tools in real work settings” did not seem to have been followed.
Effective project administration is essential for any international project. Such a coordination role can organize the project files & project history, oversee the “lessons learned” and “issues log”, and support and mentor the project leaders. As such, the Secretariat functioned as the “project management office” and it had an important role in both the internal project processes (e.g. monitoring the performance of individual workpackage projects) and connection to the external world (e.g. by interacting with external stakeholders).
Project performance can be measured during the lifetime of a project. However, the success or failure of a project can usually only be evaluated in a period of months or years after its finish, when the resultant impact can be measured (Reference Koster16). A major limitation of the evaluation of the EUnetHTA JA was that the funding provided by the European Commission required the evaluation be submitted at the end of the project. It would have been preferable to have evaluated the impact after the end of the project and identified a more complete picture of effectiveness of the project, added value to participants and engagement of stakeholders.
Self-completion questionnaires were used because they were the only viable survey format when trying to obtain information from the large cohort of respondents that were within an internationally dispersed population (Reference Edwards, Roberts and Clarke22). It would have been useful to have also used key informant interviews to obtain richer, qualitative data but unfortunately this was not possible due to time and cost restraints (according to the scope of the evaluation agreed by the European Commission).
It was important to investigate viewpoints from both project participants and external stakeholders. Different individuals can have different measures of project success, depending on their relationship to the project. A project team member's perspective often includes whether they had a satisfactory experience with the project and it met their needs, whilst the sponsor considers if the project has provided the desired performance improvement (Reference Varvasovszky and Brugha23). It is important that high response rates are received to self-completion evaluation questionnaires because it is impossible to conjecture about the opinions of nonrespondents, and how they might differ in their responses from respondents. However, it can be very challenging to collect the opinions of nonresponders. Therefore, the fact that over four-fifths of project participants completed their questionnaires indicates that the questionnaire strategy had been successful, which makes the findings robust. The response rate of external stakeholders was lower (at over two-thirds). Although a possible reason for this could have been lack of commitment to the project this is purely speculation and it would have been useful to have telephoned nonrespondents (this was outside of the scope of the evaluation agreed with the European Union).
A large proportion of project participants joined the EUnetHTA JA in its second year. It is impossible to speculate the reasons for this, but it has important implications for succession planning in organizations that participate in international HTA projects. Loss of knowledgeable staff has implications on the effective management of an agency's contribution and it is important that this is effectively communicated to the relevant work package leaders. It would seem important that international projects make comprehensive induction materials available for new members, so that they can quickly participate in activities.
In conclusion we have discussed the characteristics of successful evaluation of an international collaborative HTA project. While not the first, EUnetHTA JA may be the most successful example so far. The strategies that we followed led to a high level of engagement with the evaluation process, and these strategies are recommended for other similar international HTA projects or networks.
There were some weaknesses in the evaluation—the most telling being the requirement for the final evaluation to be conducted 6 months before the end of the project. Future projects should arrange for evaluation funding to continue beyond the rest of the project. This would allow at the least a “post project assessment”, and may allow some assessment of impact.
The experience of this project could inform the evaluation of future HTA collaborations, such as the EUnetHTA 2nd Joint Action and HTAsiaLink, which could consider assessment of collaboration objectives, efficiency of meetings, stakeholder involvement and avoidance of duplication. Above all, future assessments should strive to measure outcomes and impact, not just outputs and process.
CONTACT INFORMATION
E. Woodford Guegan, PhD, Senior Research Fellow, National Institute for Health Research (NIHR) Evaluation Studies and Trials Co-ordinating Centre, University of Southampton, Southampton, UK.
A. Cook, MBBS (andrewc@soton.ac.uk), Consultant in Public Health Medicine and Fellow in Health Technology Assessment, National Institute for Health Research (NIHR) Evaluation Studies and Trials Co-ordinating Centre, University of Southampton and University Hospital Southampton NHS Foundation Trust, Southampton, UK.
CONFLICTS OF INTEREST
Both authors report grants from NIHR HTA Programme and the European Commission - DG Sanco. Both were also members of the EUnetHTA executive committee.
