Governance and development of the Program

The Master's Program is the product of a joint effort of an international consortium that combines the expertise of five universities and five HTA agencies in Europe and Canada:

• Department of Health Administration, Faculty of Medicine, University of Montreal, Quebec, Canada
• Departments of Epidemiology and Biostatistics, and Occupational Health, Faculty of Medicine, McGill University, Quebec, Canada
• Faculty of Medicine/Faculty of Administration, University of Ottawa, Ontario, Canada
• Department of Medicine, Faculty of Medicine, University of Barcelona, Spain
• Università Cattolica del Sacro Cuore (Catholic University of Rome; UCSC), Policlinico Universitario “Agostino Gemelli” (Agostino Gemelli Polyclinic), Italy
• Agence d'évaluation des technologies et des modes d'intervention en santé (Quebec Health Services and Technology Assessment Agency; AETMIS), Quebec, Canada
• Institute for Clinical Evaluative Sciences (ICES), Ontario, Canada
• Agència d'Avaluacio de Tecnologia i Recerca Mèdiques (Catalan Agency for Health Technology Assessment and Research; CAHTAR), Spain
• Agenzia per i Servizi Sanitari Regionali (Regional Health Care Agency; ASSR), Italy
• Agenzia di Sanità Pubblica della Regione Lazio (Agency for Public Health—Lazio Region; ASP), Italy

As required by the Canada–EC Programme guidelines, the list includes organizations from two different European member states and two Canadian provinces. The funds allocated could be used to cover coordination and curriculum development, the use of information technology, and faculty and student mobility.

Creation of Five Unique Administrative Solutions at the Participating Universities

It is not easy to harmonize a curriculum jointly offered by different universities (12). During the first 2 years of the Ulysses Project, Coordinating Committee members were responsible for locally negotiating official recognition of the Master's Program. In most institutions, this process proved to be a bureaucratically lengthy, time-consuming process. A new program has been established at the University of Montreal, the Catholic University of Rome, and the University of Barcelona, while both McGill University and the University of Ottawa have created a new option within an existing master's program. There is some variation between universities with respect to the total number of credits (45 versus 48) and the options offered to students. One issue on which our collaborators could not exert much influence was student tuition fees, which are usually set by the university's higher administration.

Target Clientele

Due to the multidisciplinary and action-oriented nature of HTA, physicians, nurses, health scientists, ethicists, lawyers, biomedical engineers, social scientists, administrators, and policy-makers are all potential candidates for a master's program in HTA. The needs of these candidates, who often work full-time and possess solid experience in their discipline, are not well met by traditional graduate programs (2). We targeted two groups of students: (i) evaluators who produce HTA reports, and (ii) decision-makers at all levels of the health-care system who use HTA reports. This specialized master's program enables all participants to grasp the fundamental principles behind HTA and to use scientific results more systematically. Furthermore, participation in the program should foster a long-overdue dialogue between health policy-makers and researchers (9;11).

Structure and Content of the Program

The Ulysses Program relies on four intensive modules that each comprises two courses. Each module lasts 13 days, including 1 day off, for a total of 40 hours of teaching per course. In addition to these course requirements, students complete an internship and/or a master's thesis, or a policy analysis project. The following topics are covered:

• Understanding what HTA is all about: Course 1 (Principles and Practice in HTA)
• Learning how to do HTA: Course 2 (Methods in HTA) & Course 3 (Economic Evaluation)
• Understanding the implications and role of HTA: Course 4 (Health Policy Analysis) & Course 6 (Ethical and Social Issues, Dissemination, and Impact of HTA)
• Learning how to use HTA: Course 5 (Management of Health Organizations) & Course 7 (Clinical Practice Guidelines and Decision Aids)
• Applying what has been learned: Internship & Course 8 (presentation of student project/thesis).

First Cohort of Students

A total of twenty-three students completed all course work. The academic backgrounds and employment status of the students were diverse: eight students were women (35 percent); four were 30 years of age or younger (17 percent); eight were trained in medicine (35 percent), four in engineering (17 percent), five in health sciences (22 percent), five in social sciences or administration (22 percent), and one in law (0.4 percent). Nineteen of the students (83 percent) were working full time in health-related institutions.

Pedagogical Activities between Modules: E-Debates

Given the spread of time zones between Canada and Europe, the Consortium decided to use asynchronous e-mail exchanges to initiate e-debates, a pedagogical activity whose purpose was to maintain links between students and faculty members and foster discussion of some of the policy and methodological issues taught in the modules (13). Two e-debates were organized (March 2002 and January 2003). The first debate examined the methodological challenges in assessing telemedicine, whereas the second debate focused on drug reimbursement policies. In both cases, reading material and a brief list of questions were sent to students. Faculty members moderated the discussions. A summary of the key points discussed and several take-home messages were provided at the end of each debate.

Internships

Some universities required students to complete an internship at either an HTA agency or relevant HTA-user unit. Participants doing an internship were expected to (i) learn how to plan, carry out, and present an HTA report or an HTA-based policy document; (ii) learn how to work in a team; (iii) learn how to apply and refine methodological tools; and (iv) have the opportunity to work in a different cultural and political environment. A total of eleven students completed an internship. These internships took different forms based on the level of intensity (part vs. full time): three Ontarians (4 months; 12 months; 6 months part-time), one Quebecer (8 months part-time), six Italians (2×4 months, and 4×24 months), and one Spaniard (4 months). The most common type of training site was an HTA agency.

Results from the Self-Administered Questionnaire

Seven of the eight courses were assessed using the standardized questionnaire. Table 1 gives the mean scores for close-ended items.

The highest scores were found for attainment of stated objectives in Courses 1, 3, 4, and 7 (mean score, respectively = 4.26, 4.1, 4, and 4); usefulness of readings in Courses 1, 3, and 6 (respectively, 4.26; 4.1, and 4); and usefulness of exercises in Courses 1, 2, and 6 (respectively, 4.45; 4.47, and 4.4). Six courses, thus, have been highly appreciated for at least one of three important pedagogical components: fulfillment of objectives, readings, and exercises. The lowest scores were found for level of feedback in Course 2 (3), attainment of stated objectives (2.8) and level of feedback (3) in Course 5, and level of feedback (2.76) and course workload (2.9) in Course 7. These weaker items were found in three courses and, in addition to “fulfillment of objectives,” they reflect the level of support (feedback) and learning efforts (workload) students expect. Finally, a very large majority of students (average, 87 percent) said they would recommend the courses to fellow students. Table 1, thus, indicates that students were satisfied with the course content and particularly appreciated the exercises and materials presented in an intensive format but needed more systematic feedback from faculty members. Students could also suggest ways to improve the course. A summary of the comments we received is shown in Table 2.

Results from the Interviews with Students

As described earlier, the first cohort was composed of students from a variety of disciplines—medicine, physics, economics, administration, epidemiology, health sciences, law, engineering, and political science. The cultural backgrounds of students varied as well. There were French- and English-speaking Canadians, Italians, Catalans and Spaniards, as well as a Columbian, Argentinean, and Latvian. On several occasions, we were struck by the cultural differences in both the pedagogical approaches of faculty members and the learning strategies of students. The interviews revealed differences in the preferences of full-time students vs. full-time working professionals, and in Canadian versus European students. In general, although the issue is complex and nuanced, Europeans tended to favor a lecture-based (expert–student) pedagogical approach, whereas Canadians felt more comfortable with an interactive (mentor–professional) approach.

The students believed that the pedagogical approach should be more homogenous across courses (see Table 3). As observed earlier, some students found that courses with several invited speakers (8–13) would benefit from a tighter common thread and were slightly less successful in meeting the pedagogical objectives. On the other hand, one student reported: “It's a really neat program. Some professors really brought good perspectives.” On some occasions, students found the readings to be somewhat redundant. Students requested that the final assignments be practice-oriented, so as to facilitate mastery of the concepts taught during the courses.

The international scope of the Program forced the students to adapt to different teaching styles and to the opinions of fellow students, which was perceived as both challenging and stimulating. One student appreciated the discussions with fellow students from other countries: “I found it very useful to have various viewpoints. It is interesting to see how other health-care systems are working and have various perspectives on that.” One student was confident that it would lead to future collaborations between alumni: “The international component of the Program is very important. It helps us create links with people from other countries and develop fruitful collaborations.” Hence, being exposed to various viewpoints and getting more familiar with other health-care systems were identified by several students as important advantages.

Canadian students requested that the courses be more interactive. The same comment applied to the e-debates, which were instrumental in maintaining contacts between modules. Responses to close-ended items in the interview questionnaire, indicate that 60 percent of students “agreed” or “strongly agreed” about their pedagogical value. All students agreed that two e-debates were enough. Full-time working professionals could not increase their involvement in pedagogical activities between modules. Some students believed that the format should be improved. They said they received too much information over too long a period of time, which resulted in their easily losing track of the debate. One student suggested using a Web site to post students' comments and organize them in a way that graphically shows the flow of responses and replies. Another student expected the debates to be more incisive and less of a collegial sharing of individual opinions. Up to 40 percent of interviewees “agreed” or “strongly agreed” that the format was too demanding. Full-time students mentioned that 2-week modules were too short and intense, saying 3-week modules would be preferable to maximize the learning process. However, professionals with full-time jobs liked the 2-week modules, which were more compatible with their professional obligations. Most of them received support from their employers, such as paid leaves. In some cases, the employer reimbursed the tuition fees or partially covered the travel expenses. Overall, 80 percent of the interviewees “agreed” or “strongly agreed” that the workload was reasonable.

In terms of career development, professionals who enrolled in the Program stressed that it would trigger a crucial shift in their career. Students mentioned that the training program would (i) help them to create an HTA Unit in their organization; (ii) acquire skills to more thoroughly assess technology and services; and (iii) provide them with tools to analyze and concretely manage clinical problems. For instance, one student stressed, “It's a total shift in my career—the opportunity to implement an HTA Unit in our hospital. It will enable us to gather and analyze evidence, which is lacking in our health-care system. My goal was to be able to coordinate an HTA Unit. The Ulysses Program made it possible by giving me the tools I needed.” For full-time students, career opportunities were not as clearly defined, although some were intending to enroll in a PhD program or medical school. The practical experience gained through internships was appreciated by the full-time students. According to one of them, “It gave me more of a perspective of what decision makers are looking for in terms of information on health technology.”

For the second edition of the Program, students recommended increasing the methodology content in the courses. They also suggested that the modules alternate between Canada and Europe. Finally, students recommended that an English proficiency test be a requirement for enrollment.

Results from the Interviews with Internship Supervisors

Although student satisfaction is a major dimension of the quality of a training program, other aspects should also be considered. Five interviews were conducted with HTA experts who had supervised trainees. Four of the training sites were HTA agencies (AETMIS, ICES, CCOHTA, and ASP) and one was a university-based research unit (Clinical Epidemiology Research Unit, University of Ottawa). All training sites provided trainees with a computer and workspace, and some provided a stipend. One training site signed a 6-month contract with its trainee. Opinions regarding the training experiences varied: Two agencies reported that it benefited their organization, whereas one answered that it did not. One supervisor said that “the trainee's job was useful for the Technology Assessment Unit because it was fully integrated into the Unit's work.”

All training sites mentioned that a 6-month period was needed to give the trainees time to adapt to the organization's culture. One supervisor suggested identifying goals and objectives before the start of the training period to optimize the experience. Finally, all supervisors agreed that an internship is a valuable pedagogical complement to a master's program because, as one of them put it, “the practical conducting of studies is the real way to demonstrate the usefulness of education.”