Introduction
Identifying how well a behavioral intervention is received by the intended target population and the degree to which it adheres to a set protocol can have significant implications for translation of the program into clinical practice and to determine the validity of the findings. These two processes, commonly labeled “acceptability” and “treatment fidelity,” should routinely be adopted as due process in the early stages of planning in health behavior interventions and programs.
The acceptability of an intervention can be influenced by several factors, including the specific nature of the treatment (e.g. opportunity for personal gain), the complexity of the intervention, and the format of the delivery of the specific treatment (Cowan and Sheridan, Reference Cowan and Sheridan2003). Subjective evaluation measures, such as the use of self-report questionnaires, are frequently adopted to identify consumer ratings regarding the appropriateness, effectiveness, and fairness of the intervention, which are characteristics that exemplify acceptability (Finn and Sladeczek, Reference Finn and Sladeczek2001). Consumer satisfaction plays a particularly important role in determining whether the intervention or treatment procedures are likely to be adopted, implemented, and maintained, beyond the setting in which it was delivered.
Treatment fidelity can be defined as the methodological strategies and practices used to address the reliability and validity of research interventions (Bellg et al., Reference Bellg2004). It aims to determine if the treatment was delivered as intended and the extent to which newly learnt skills are applied to daily life (Lichstein et al., Reference Lichstein, Riedel and Grieve1994). Treatment fidelity becomes important when determining the degree to which the effectiveness (or lack thereof) of an intervention may have been influenced by other factors, and has implications for the degree of confidence placed upon proposed mechanisms for observed change (Lichstein et al., Reference Lichstein, Riedel and Grieve1994). This becomes a particularly salient issue for interventions delivered by different instructors across varied time points within a trial.
Despite the potential implications for the interpretation of study outcomes, the reporting of acceptability and treatment fidelity has not been consistent (Bellg et al., Reference Bellg2004). For example, Dane and Schneider (Reference Dane and Schneider1998) reviewed the degree to which consideration was given to fidelity assessments in a series of primary and early secondary prevention programs. They established a set of criteria against which studies were assessed, focusing specifically on aspects of program integrity, rating fidelity according to adherence, exposure, quality of delivery, participant responsiveness, and program differentiation (Dane and Schneider, Reference Dane and Schneider1998). They found that less than a third of the 162 outcome studies addressed the issue of fidelity.
The Promoting Healthy Ageing with Cognitive Exercise (PACE) study was designed to determine whether a cognitive activity intervention could decrease the rate of cognitive decline amongst older adults with mild cognitive impairment (MCI). The aim of identifying the acceptability of the program was to investigate consumer satisfaction with the interventions and to determine how the target population rated the program components. They were also asked to rate the relevance of the program to their perceived needs and whether the intervention was something in which they would be willing to participate in their community.
The aim of the fidelity assessment was to ensure consistency in delivery of the content of the intervention. In accordance with the criteria utilized by Dane and Schneider (Reference Dane and Schneider1998), attention was also paid to exposure, participant responsiveness, and program differentiation.
Methods
Participants, randomization, and blinding
Detailed information regarding the participants, the training program, the experimental design, and the outcome measures of the PACE study has been published elsewhere (Vidovich et al., Reference Vidovich, Lautenschlager, Flicker, Clare and Almeida2009). In brief, 160 adults were deemed eligible to participate in the trial, based on the following inclusion criteria – a diagnosis of MCI (according to Portet et al., Reference Portet2006, criteria), aged 65 years or older, able to travel to the research center, and proficient in spoken and written English. Individuals with a diagnosis of dementia according to International Classification of Diseases 10th Revision (ICD-10) criteria for research (WHO, 2007) or suffering notable cognitive impairment, as evidenced by a Mini Mental State Examination (MMSE; Folstein et al., Reference Folstein, Folstein and McHugh1975) score of 23 or less, were excluded from the study. Additional exclusion criteria included current psychiatric disorder, current hazardous or harmful alcohol consumption (based on Saunders et al., Reference Saunders, Aasland, Babor, de la Fuente and Grant1993), and current medical condition that prevented participation in the study tasks (such as sensory impairment) or was associated with reduced survival over a 12-month period (e.g. advanced cancer). Individuals who reported a clinical history of stroke associated with permanent disability were also excluded. A baseline neuropsychological assessment was conducted prior to randomization into either a five-week cognitive activity (CA) program or a control general education (GE) program. Randomization was undertaken in blocks of 12–18, according to a random list of numbers generated by computer, with six to nine individuals allocated to each group. Due to the nature of the intervention, participants were not blinded to group membership; however, research assistants undertaking the follow-up assessments were. All assessments and the program interventions were undertaken at the Western Australia Centre for Health and Ageing at Royal Perth Hospital.
This project was conducted in accordance with the Helsinki Declaration for Human Rights. The ethics committee of the Royal Perth Hospital approved the study protocol and all participants offered written informed consent.
Intervention
During the early stages of conceptualization of the trial, eight individuals with MCI, aged 60 years and older, were invited to participate in a one-hour focus group. Their opinions were sought regarding the development of a program of CA specifically designed for older adults concerned about cognitive decline. The consistent opinion of the group was that any CA program needed to be challenging and personally stimulating and to have a social component. Two program interventions were then manualized comprising the CA and GE arms. The content of the manuals incorporated known information regarding age-associated cognitive decline and strategies to manage difficulties, as well as addressing health issues relevant to older adult populations. These programs were piloted with a group of ten older adults with MCI and served as the basis for the design and implementation of the intervention, which is described below.
Study participants were asked to attend ten 90-minute sessions over a five-week period. The group randomly assigned to CA received information about age-associated cognitive changes and ways to manage common difficulties, as well as completing pencil and paper activities to illustrate strategies for enhancing attentional capacity, memory functions, and executive processes. Details about the intervention have been described elsewhere (Vidovich et al., Reference Vidovich, Lautenschlager, Flicker, Clare and Almeida2009). In contrast, participants randomly assigned to the GE program took part in a series of presentations about healthy aging, including the impact of lifestyle factors, with sessions comprising videos and group discussion (please see Vidovich et al., Reference Vidovich, Lautenschlager, Flicker, Clare and Almeida2009, for further details). A clinical neuropsychologist (MRV) ran all groups, exposing participants to the same instructor, time with instructor and social contact. All participants received a manual that contained each session content and material covered for each session. Six months after the beginning of the intervention, participants received a “booster” telephone call that consisted of a summation of their program material and addressed any doubts or concerns. A further face-to-face booster group session, one hour in duration, was conducted one month after the 52-week follow-up assessment. All booster sessions were conducted by the same clinical neuropsychologist who had run the group interventions (MRV). Finally, all participants underwent the same assessment protocol, regardless of group assignment (please see Vidovich et al., Reference Vidovich, Lautenschlager, Flicker, Clare and Almeida2009, for details).
Study measures
Study participants who attended the final session of their five-week group program were asked to complete a brief questionnaire (see Box 1), rating the acceptability of the program. These questionnaires were completed anonymously within the group environment whilst the instructor was out of the room and the data were collated and entered into an excel spreadsheet by an independent research assistant.
1. Please indicate how relevant you found this program to your needs
(1 = very irrelevant and 5 = very relevant)
1 2 3 4 5
2. If you had the opportunity to attend this program in your own area would you attend?
Yes or No
If “No” please explain why not.
If “Yes” would you be willing to pay a fee for his service: Yes or No
3. Do you expect to make any changes to your life as a result of this program?
Yes or No
If “Yes” please explain
4. Are there other topics you think should be covered in this program?
Yes or No
If “Yes” what are they
5. Please rate the overall program presentation (1 = strongly disagree and 5 = strongly agree)
(a) Clear 1 2 3 4 5
(b) Relevant 1 2 3 4 5
(c) Interesting 1 2 3 4 5
(d) Informative 1 2 3 4 5
6. Please provide any further comments/recommendations about the program.
Eleven GE and 11 CA groups were conducted over the course of the study resulting in a total of 110 presentations under each condition. The content of each session was digitally recorded and following each session, the voice recordings were downloaded onto a PC. An excel spreadsheet was used to identify a random list of session (e.g. 2–9) and group (e.g. 1–11) numbers, excluding sessions one and ten, generated by computer. Session one was excluded as it was considered an introductory session which reviewed the nature of the trial and program material to be covered, and allowed for participants to informally get to know each other. As session ten was used as a revision session and no new material was presented, it was also excluded. An excel spreadsheet was used to identify a random list of numbers generated by computer, with a total of 44 sessions (i.e. 22 from each arm) randomly selected and transcribed by research assistants.
A template was devised that identified the individual content components of each session, against which ratings were later made to determine whether material contained in the instructor program manual had been presented. The amount of content and topic structure to be reviewed varied across each session. Topics which were rated and the amount of item content within each topic are shown in Table 1. A rating key was devised to indicate the degree to which the topic was covered and numerical ratings were applied.
Content of session topics rated for fidelity assessment
A research assistant, who was not involved in the transcribing process and did not have any contact with participants, listened to the transcripts and completed the independent ratings. Once the initial ratings of session content had been completed, the rating scale was further collapsed and new values were assigned to assist in analysis. “Not covered” and “Uncertain/unclear” were assigned a value of “0,” “Partially covered” was scored as “1,” and “Covered” was scored as “2,” as outlined in Table 2.
Rating scale used in template
Exposure was maintained for each of the 11 groups in the respective CA and GE arms, by adhering to a strict 90-minute time limit for each session and ensuring that the same instructor consistently delivered all ten sessions, within each intervention, across the course of the trial. Attendance at each session was manually recorded in order to monitor levels of participation across both groups. A minimum of seven sessions was set a priori for participants to be considered as having completed the program.
Program differentiation was maintained by ensuring that there was no potential for overlap between groups, with participant sessions separated by a two-hour time frame. If a husband and wife wished to participate in the program and both met criteria, they were asked to self-select the individual who would take part in the trial. Participants were explicitly asked at each assessment not to discuss information regarding the intervention with research staff and, similarly, staff conducting assessments were instructed not to discuss with participants any aspects of the intervention.
Statistical analyses
Data were managed in Excel and analyzed with Stata version 11.0 (StataCorp, College Station, TX). We used descriptive statistics (mean, standard deviation of the mean (SD), proportions) to summarize our data, and Pearson's χ2 statistic to test the frequency distribution of ratings. We calculated a fidelity score for each session by adding the ascribed ratings and dividing them by the maximum possible rating. For example, if there were 40 content items, a maximum possible score would be 80/80, with the 40 items having attracted a rating of “two” if they had been entirely covered. These values were converted to percentages for each session (e.g. 80/80 = 100%). An overall fidelity rating was later calculated by summing the percentages and determining the mean score for the 22 sessions randomly chosen to be rated. We established, a priori, that an overall fidelity rating of 90% or more indicated acceptable fidelity. Alpha was set at 5% and all tests reported are two-tailed.
Results
Of 160 participants, 80 were randomly assigned to the CA group. Participants in the CA and GE groups did not differ in age, education, or cultural background (p > 0.05; data not shown), though there were significantly more women in the CA group (p < 0.05; data not shown).
Seventy-four feedback forms were collected from the CA group and 68 from the GE group (out of a possible 80 participants in each group, respectively). The proportion of responses to questions one to five is presented in Table 3. The only significant difference between the two groups was the observation that the CA group reported the intervention to be significantly more relevant to their needs (χ2 = 24.5, degrees of freedom (df) = 5, p < 0.01), as expected. Each program was rated equally highly with respect to clarity, interest, and how informative the content was and over 85% of participants from both groups surveyed indicated that they would attend this sort of program in their local area. More than two-thirds would be willing to pay a fee to receive the service, with no significant differences between the groups (Table 3).
Percentage ratings for feedback questionnaires post-program completion
An overall fidelity score was calculated for each group's presentations, with these being 96.1% (SD = 2.39; 95% CI = 95.10, 97.10) and 94.7% (SD = 4.40; 95% CI = 92.87, 96.55) for the CA and GE groups, respectively. There was no significant difference in the number of sessions attended between groups, with mean attendance for the CA group being 8.1 sessions (SD = 2.8, range = 0–10), with 82.5% of the participants attending seven or more. Mean attendance for those randomized to the GE group was 8.4 sessions (SD = 2.6, range = 0–10), with 87.5% attending seven or more.
Discussion
The PACE trial was designed to identify the effect of a five-week CA intervention on cognitive decline for older adults with MCI, although it was critical to the interpretation of the main results of the study that appropriate acceptability and fidelity are established. We found that there was no difference in attendance between the CA and GE groups, the ratings of content material, and willingness to participate in such a program within a community setting. The only difference was that older people in the CA group rated the program to be significantly more relevant to their needs, as expected.
Assessment of treatment fidelity and, more specifically, delivery of the intervention revealed a high level of consistency, confirming that the intervention programs were delivered as intended, with no apparent bias with respect to the quality of the presentations. In addition, participant attendance was generally high, which provides a further indication of satisfaction with the program content and delivery.
Manualizing the program interventions was certainly critical to ensure reproducibility of the intervention, which should now be tested in different settings and populations. This process, together with the recording of sessions, ensured consistency in delivery and assisted in minimizing issues of contamination. Furthermore, contamination was also avoided by adopting a lengthy time interval between groups so that participants from the differing conditions did not come into contact with each other. Resource materials were also securely stored so as to reduce opportunity for accidental exposure.
With respect to identifiable limitations, participants were motivated volunteers with a vested interest in improving their cognition. It remains unclear as to whether the acceptability of the intervention would be as high in a general community setting. Furthermore, the adopted measure of acceptability was possibly limited in scope, given the brevity and specific nature of the questionnaire. The degree to which these forms of programs are readily adopted by participants and practically applied will only be determined once we are able to establish efficacy and the interventions implemented in clinical practice. With respect to the fidelity assessment, the number of sessions randomly chosen to be reviewed was relatively small. This in part reflected the time and costs involved in transcribing the extensive number of sessions which were run with each of the groups. More precise estimates of fidelity would require extension of the study activities. Finally, measures of fidelity were limited to the number of content units delivered by the two interventions. It is unclear if good fidelity translates into greater benefits for patients and we are currently in the process of evaluating the results of the trial.
Assessing a research trial's acceptability and fidelity allows investigators and independent reviewers to draw more accurate conclusions regarding the merits of any observed outcomes. Ideally, these assessments should be reported in publications of intervention trials. The consequences associated with treatment fidelity may potentially be minimized if the intervention is designed by the therapist intended to deliver it. However, the delivery of the intervention by the same individual becomes crucial, particularly if procedures are not in place to manage situations whereby an alternate party is required to deliver the program. Additionally, and in large recruitment pools, by utilizing a manualized script and recording sessions, consistency in delivery can be ensured. The type of questionnaire/measure utilized in the PACE trial can readily be adapted for all intervention programs to ensure integrity, regardless of whether the format is group or individualized, computer versus pencil and paper etc. The criteria proposed by Dane and Schneider (Reference Dane and Schneider1998) have relevance for all forms of intervention delivery – computer-based training or otherwise and future standardization on the minimum details to be reported would help with pooling of data across trials. To our knowledge, there are no other recently published RCTs of individuals with MCI who have participated in a cognition-focused intervention, and whereby the acceptability and fidelity have been systematically reported.
In conclusion, this study showed that the PACE interventions were delivered in accordance to the program manual and that the project activities were well received by participants. Future studies will determine if the intervention is similarly well accepted in other clinical settings and if the efficacy of the program is directly related to acceptability and fidelity measures.
Conflict of interest
None.
Description of authors’ roles
All authors are members of the PACE project group and participated in the conceptualization and implementation of the study. MRV acts as guarantor of the data and has been responsible for the day-to-day supervision of research staff and delivery of the interventions. All investigators have contributed to design the study and to obtain funding. MRV has drafted this manuscript, which has been critically reviewed by OPA, NTL, LC, and LF.
Acknowledgments
We are most grateful to all volunteers taking part in the study and to our research staff. This project is supported by grants from the Centre of Excellence in Alzheimer's Disease Research and Care (L0602) and the National Health and Medical Research Council (513772).
