Study design

This is a multicountry, pragmatic, hybrid type II effectiveness–implementation, cluster-randomized, clinical trial (ISRCTN 11913964). Clusters were clinicians working in outpatient mental health services in five countries: Bosnia and Herzegovina, Kosovo*, Montenegro, North Macedonia, and Serbia. Clinicians were randomly assigned to either DIALOG+ (intervention) or standard care (control). The intervention arm received DIALOG+ during routine clinical appointments instead of usual care. The control arm received usual care during also routine clinical appointments. A cluster randomization design was used to avoid potential contamination of the practice by clinicians when treating patients in both arms. The trial explored effectiveness, cost-effectiveness and fidelity of the DIALOG+ intervention relative to standard care. Detailed description of methods can be found in the study protocol [Reference Jovanovic, Francis, Maric, Arenliu, Barjaktarov and Dzubur Kulenovic12]. The trial was conducted between March 2019 and July 2020.

A mixed-method process evaluation was nested within the IMPULSE trial, using the United Kingdom Medical Research Council process evaluation of complex intervention guidance [Reference Moore, Audrey, Barker, Bond, Bonell and Hardeman13]. The objectives of the process evaluation were to explore the context influencing the implementation of the DIALOG+ intervention, intervention fidelity, and acceptability of the intervention to patients and clinicians [Reference Pemovska, Jovanović, Radojičić, Arënliu, Džubur-Kulenović and Novotni14].

Settings and participants

In total, 11 outpatient mental health services across five countries (three in Montenegro and two in each of the other four countries) participated in the trial. Clinicians were eligible if they had a professional qualification and more than 6 months’ experience working in mental health care. The clinicians’ caseloads were screened to identify eligible patients aged 18 years or older, primary diagnosis of psychosis in remission (ICD-10 F20–29, F31) [15], attending the outpatient clinic or day hospital, lifetime history of at least one hospital admission, and capacity to provide informed consent. Exclusion criteria included diagnosis of organic brain disorders and severe cognitive deficits thus being unable to provide information to study instruments. Written informed consent was obtained from all clinicians and patients. The study received approvals from institutional research ethics committees.

Intervention arm

The intervention arm received DIALOG+ during routine clinical appointments instead of usual care. The DIALOG+ intervention was offered to patients six times over 12 months (once per month for the first 3 months and then once every 3 months). The main features of the DIALOG+ intervention have been outlined in the Introduction. Each DIALOG+ session begins with patients’ self-assessment on a tablet computer [Reference Priebe, Golden, Katevas, Healey and McCabe10]. The patients assess their satisfaction with eight life areas (mental health, physical health, job situation, accommodation, leisure activities, friendships, relationship with family/partner and personal safety) and three treatment areas (medication, practical help and meetings with professionals). Satisfaction with each area is rated on a 1—(totally dissatisfied) to 7—(totally satisfied) point scale. Next, the patient and clinician both reflect on the patient’s concerns, potentially comparing the current ratings with any previous ones to see which domains the patient has improved or deteriorated. Patients are then invited to choose an area they wish to discuss further during the meeting. A four-step, solution-focused approach (Understanding, Looking Forward, Exploring Options, and Agreeing on Actions) is used as a discussion guide to recognize and apply the patient’s existing resources. The session concludes with the patient and the clinician agreeing on actions that the patient, clinician and/or another person from the patient’s life should accomplish before the next meeting to improve the patient’s life and treatment satisfaction [Reference Priebe, Golden, Katevas, Healey and McCabe10].

The first session should last up to 60 min and follow-up sessions up to 30 min [Reference Priebe, Golden, Katevas, Healey and McCabe10]. The offered intervention always has the same format and clinicians use the same technique in each session. The content of sessions varies as per patients’ choice, for example, patients could choose to discuss different life domains in each session.

Control arm

The control arm received standard care which includes routine clinical appointments, following the same delivery schedule as the intervention arm. These appointments generally focused on reviewing prescribed medication (e.g., is the medication effective, should the dose be modified, are there any side effects), discussing other aspects of care (e.g., referral to psychological therapy or social care), and providing general support (e.g., reassuring the patient stressed due to experiencing derogatory auditory hallucinations) [Reference Jovanovic, Francis, Maric, Arenliu, Barjaktarov and Dzubur Kulenovic12]. At the beginning of the study, clinicians were asked not to change their usual clinical approach. The trial process evaluation included audiotaping and analyzing sessions from both arms [Reference Pemovska, Jovanović, Radojičić, Arënliu, Džubur-Kulenović and Novotni14]. The purpose of this work was first to assess contamination and differentiation between sessions in both arms. Secondly, we wanted to improve our understanding of the content and form of control sessions.

Implementation strategy

The implementation strategy is a data-driven and theory informed document that was created before the trial to guide implementation of DIALOG+ [Reference Hunter, McCabe, Francis, Pemovska, Ribić and Smajić Mesević16]. The strategy was based on data from interviews and focus groups with stakeholders conducted prior to implementation of the intervention. The strategy provides detailed descriptive information of each potential barrier and the actions and resources to overcome it. For example, during pretrial work the most common barrier was clinicians’ perceived lack of capability and skills to deliver DIALOG+ to patients with psychosis. As a potential solution, DIALOG+ training was modified: training materials were adapted and appropriate behavior change techniques were used to classify active ingredients of the intervention and to address the identified barriers to implementation [Reference Hunter, McCabe, Francis, Pemovska, Ribić and Smajić Mesević16, Reference Michie, Richardson, Johnston, Abraham, Francis and Hardeman17]. Each clinician in the intervention arm received face-to-face training by a local research team member before the first DIALOG+ session, followed by top-up training after delivering the third session. Each clinician was able to access individual supervision provided by the study researchers after each session.

Outcome measures

Impact of the COVID-19 pandemic

The COVID-19 pandemic was announced in March 2020 when the trial was in its final phase. The delivery of fifth and sixth DIALOG+ sessions and 12-month patient assessments had to be conducted remotely. This led to substantial changes in the intervention, patient assessments, and data collection. The patient–clinician pairs were not able to meet in person or use the tablet together. Researchers were not able to complete researcher-observed ratings on BPRS [Reference Ventura, Lukoff, Nuechterlein, Liberman, Green and Shaner19] and CAINS [Reference Kring, Blanchard, Horan and Reise20], so only patient-reported items were collected and analyzed for these scales. Furthermore, patients’ responses to the questionnaires could have been influenced by distress and lack of social support during pandemic. All of this made 12-month data difficult to interpret. The IMPULSE consortium agreed to focus on outcomes at 6-months post-randomization which was approved by the Scientific Advisory Board.

Randomization and blinding

Eligible clinicians and patients were identified and consented. Once a patient gave informed consent, researchers completed the baseline assessment which included sociodemographic, clinical characteristics, and outcomes measures as listed above. To minimize selection bias within clusters, clinicians were randomized once all patients from their caseload had been recruited and all baseline assessments completed. Clinicians were randomized 1:1 to intervention or control. Randomization was conducted by an independent statistician who used sequential computer-generated random numbers to determine allocation. To prevent an unequal allocation across groups, participants were stratified before randomization on two factors: gender (e.g., male vs. female) and professional status (e.g., psychiatrists vs. nonpsychiatrists). Owing to the nature of the intervention, clinicians and patients could not be blinded to their allocation. Researchers who conducted the assessments were blinded to the allocation of patients.

DIALOG+ adherence scale

The DIALOG+ adherence scale (DAS) was developed by DIALOG+ experts at Queen Mary University of London. The scale is used to assess clinician behaviors specific to the delivery of DIALOG+ (e.g., selection of areas for further discussion, the four-step approach) [25] as specified in the manual [Reference Priebe, Golden, Katevas, Healey and McCabe10]. The scale comprises 19 items, scored using a two-point scale (0, 1) to indicate either the absence or implementation of a specific behavior. There are three subscales:Adherence to the initial DIALOG scale and review of ratings (seven items), Adherence to the four-step approach (nine items), and Quality of interaction (three items). A mean score (min, max) for each item, three subscales, and the total score was calculated across the sample. Structured analysis of audiotapes from 17 clinicians from the intervention arm was used to assess clinicians’ Adherence to the DIALOG+ manual. Audiotaped control sessions were also assessed against the DAS to evaluate intervention differentiation and contamination. A higher score indicates a better clinician’s Adherence to the DIALOG+ manual [Reference Priebe, Golden, Katevas, Healey and McCabe10]. Similar DAS scores in both arms would indicate that clinical work was not substantially different and that clinical appointments in the control arm could have been contaminated.

Sample size

Preliminary sample size calculations were based on the data from the original DIALOG+ trial in which the effect size was 0.35 [Reference Priebe, Kelley, Omer, Golden, Walsh and Khanom8]. This effect size reflected improved quality of life ratings for five of 12 life domains in the DIALOG+ intervention, which is considered clinically and socially relevant. To detect an effect size of 0.35, this trial needed 260 patients. The number was inflated to account for cluster design and 10% dropout of clinicians. The intracluster correlation coefficient (ICC) was 0.05, and the design effect due to cluster randomization was 1.2 [Reference Kerry and Bland26]. Assuming a standard deviation of 1.0, with 80% power at 5% significance level, the study required a minimum of 36 clinicians in each group (72 in total). To allow for equal number of clinicians per country, the study aimed to recruit a minimum of 16 clinicians per country (eight in the intervention arm, eight in the control arm). In total, the study aimed to recruit 80 clinicians and 400 patients.

Data analysis

The randomization of patients was performed at the cluster level, so there were potential imbalances in patient characteristics between arms. Therefore, the first analyses compared characteristics of the arms at baseline. The outcome measures were compared between arms at both 6- and 12-months based on observed data only. To allow for the clustering of data into different sites and different clinicians, analysis was performed using multilevel regression methods. Three-level models were utilized. These considered measurement from individual patients contained within measurements from the same clinician. These in turn were nested within the individual sites (i.e., countries). The mean difference in outcome between arms, indicative of the magnitude of the effect, was calculated as outcome for the intervention arm minus outcome in the control arm and presented along with corresponding confidence intervals. The arm difference was adjusted for outcomes at baseline as well as for other covariates included in the models, including patient age, patient diagnosis, and clinician profession (psychiatrist vs. non-psychiatrist).

Analyses were performed on an intention-to-treat basis, including all randomized patients. To allow for missing data, multiple imputation methods were used in a sensitivity analysis (Supplementary Appendix I). Subgroup analyses examined data from Serbia only, as this was the only country to complete the trial before the COVID-19 restrictions. Two-level multilevel models were used as all data came from the same country (Supplementary Appendix II). A p-value less or equal to 0.05 was considered statistically significant.