Theme/Mandate of the Assessment

The participants believed the choice of subject was justified (82.8 percent), and it was clear to them who the commissioner was (89.7 percent) (Table 2). However, twenty (34.5 percent) believed that the mandate was not clearly stated and had to be discussed. Of these twenty responses, nine indicated that this lack of clarity slowed down the process, whereas six responses did not. Thirty-six believed the mandate was clearly expressed, and twenty-five believed that it nevertheless was useful for the members of the team to discuss the issue under study and assist in formulating the mandate.

Composition of the Review Team

Forty-nine responses were to questions on the importance of the composition of the review team. The factors that increased credibility and legitimacy of the report were the interdisciplinary structure and large group size, which gave improved competence and added to good scientific discussions that promoted high quality. Another comment was that it was important that the participants had enough time. It was also noted that Norway has few experts, and it may be that the use of experts creates barriers. In some instances, it was considered of importance that the different regions of the country were represented.

Organization of the Review Team Work

Forty-four responses (75.9 percent) showed satisfaction with the scientific review team leader and forty-six responses (79.3 percent) with the SMM's project leader (Table 2). There were nineteen responses for how to improve the leadership of the group. The replies centered on reducing the time spent, keeping steady progress, achieving goals, setting tighter time limits, and improving guidance in the writing process. In all, forty-nine responses (84.5 percent) indicated satisfaction with the service provided by SMM. Ten persons had suggestions for improvements. These centered on delays due to members of the group not doing their assignment in time due to a heavy workload and lack of flexibility in the writing stage of the report. There were forty-nine responses (84.5 percent) that showed satisfaction with the balance between expected workload and the SMM assistance. There were four missing responses to this question. Fifteen responses indicated the wish for more contact with the SMM Board during the work.

The Working Method

The working method, based on the principle of systematic review, was commented upon in fifty-one responses (Table 2). Many expressed that they had achieved good knowledge of the method. They had learned a valuable method, the work gave scientific credit, and the experience was useful for later work. Skepticism to single studies as the basis for review conclusions was reinforced. It was expressed by some that it had been a lot of work “for little use” and that systematic reviews easily can be seen as negative if they do not give evidence for a change of current practice for the technology in question. One person expressed that the work had been a waste of time.

Update of Report

The great majority of participants, forty-eight in all (82.8 percent), were positive to taking part in updating their HTA review assessment after some time using the same working method (Table 2). Only one expert answered “no” to this question.

Other Comments

Additional comments were given by forty-two responses. In short, the responses were as follows: important with a concise mandate; too much time spent on discussing concepts; the time spent on the report was too long; the time intervals between meetings were too long; important to have enough time for the job; meetings over several days would possibly have saved us time; a problem that not all the participants met deadlines; important to have the assistance by SMM's staff. The participants found the HTA work informative, interesting, of professional value, and it increased their qualifications. It was commented that SMM had developed into an important institution in Norway. In general, it was important that HTA studies also include low-status medicine short of resources.

Differences Between Review Teams

With regard to differences between review teams, nine questions had responses with the necessary variation required for ANOVA analyses. Five questions showed significant differences (p<.05). These findings showed disagreement about the importance of the composition of the review teams regarding interdisciplinarity, clear formulation of the mandate, a too high a workload considering the results, and a wish for more formal training. There were no significant differences regarding the importance of all Norwegian regions having representatives in the groups, any lack of clinical or scientific competence in the groups, additional review team contact with the SMM Board or publication of the work.

Time-trend analyses were only applicable on three questions—about clinical or scientific competence missing, about the need for further training, and publication strategy—and none of these showed any change with time.

Six significant associations were found concerning the associations between the total time spent and the time spent at meetings versus the participant response. The importance of interdisciplinary teams was found to be directly associated with increasing number of meetings. This finding is possibly because many of the reports covered larger medical issues (1;20;25). These review teams have most likely had more meetings and a higher requirement for being interdisciplinary, causing the establishment of large review teams to achieve a satisfactory assessment. Increasing time spent on the work was associated with the desire for more communication with the Board. The satisfaction with the project management was inversely related to the time spent. A satisfactory balance between the group members and SMM service was positively related to increasing time spent. Even in long-term projects, there was an understanding for factors influencing the time spent. However, the discontent with the workload in relation to the result was significantly associated with increasing time spent.