Introduction
With the wide availability of computers and ready access to the internet, a number of psychological interventions have been developed into web-based instruments (Proudfoot et al., Reference Proudfoot, Goldberg, Mann, Everitt, Marks and Gray2003; Shaw et al., Reference Shaw, Marks and Toole1999). There is now good evidence that computerized versions of interventions are effective, especially for mild to moderate cases (Andrews and Titov, Reference Andrews and Titov2009; Proudfoot et al., Reference Proudfoot, Ryden, Everitt, Shapiro, Goldberg and Mann2004). One important limitation of computer-based tools is that they rely on clients accessing and using the web-based resources appropriately and without direct therapist supervision, making it important to develop software that is very accessible and straightforward to use for clients. Authors have therefore argued that it would be beneficial to develop computerized treatments aiming to be equally applicable across presenting problems (Mansell et al., Reference Mansell, Harvey, Watkins and Shafran2009; McManus et al., Reference McManus, Shafran and Cooper2010).
There is a growing evidence base for computerized cognitive behaviour therapy (cCBT), with current research suggesting that interventions where there is therapist support are associated with larger effects than fully automated programs (Andersson et al., Reference Andersson, Carlbring, Ljotsson and Hedman2013; Hedman et al., Reference Hedman, Ljotsson and Lindefors2012; Johansson and Andersson, Reference Johansson and Andersson2012; Palmqvist et al., Reference Palmqvist, Carlbring and Andersson2007; Spek et al., Reference Spek, Cuijpers, Nyklicek, Riper, Keyzer and Pop2007). Supported treatments have also been shown to have lower rates of attrition than unsupported ones. The amount of support can vary, with some programs requiring as little as one short email per week per client, while others delivered in ‘real-time’ require as much therapist time as face-to-face treatment (Kessler et al., Reference Kessler, Lewis, Kaur, Wiles, King and Weich2009). Although the evidence for unsupported treatments is weaker, they have been found to be associated with medium effect sizes (Christensen et al., Reference Christensen, Griffiths, MacKinnon and Brittliffe2006). Open-access unsupported treatments have advantages in terms of cost and have the potential to fill a large unmet need in current services (Christensen et al., Reference Christensen, Griffiths, MacKinnon and Brittliffe2006).
This paper presents the results of a web-based trial of a problem-solving intervention that has been developed based on the principles of perceptual control theory (PCT) (Powers, Reference Powers1973; Powers et al., Reference Powers, Clark and McFarland1960). PCT provides a unifying theory of everyday human functioning, including an integrative account of how psychological distress manifests and is maintained (Carey, Reference Carey2006; Higginson et al., Reference Higginson, Mansell and Wood2011; Mansell, Reference Mansell2005). According to PCT, all living things have a basic need for control, where the organism will act against any environmental disturbances in order to achieve and maintain control over its experience (Powers, Reference Powers1973; Powers et al., Reference Powers, Clark and McFarland1960). From a PCT perspective, problems in functioning can be understood as problems in this process of control.
Method of Levels therapy (MOL) (Carey, Reference Carey2006) has been developed based on the principles of PCT. MOL is transdiagnostic and uses open questioning techniques to focus on the process of thinking as opposed to the content of what the client says. MOL also does not depend on the use of homework, written formulations or other tools. In a series of pragmatic trials, MOL has been evaluated in treating a range of difficulties in primary care (Carey et al., Reference Carey, Carey, Mullan, Spratt and Spratt2009; Carey and Mullan, Reference Carey and Mullan2007, Reference Carey and Mullan2008). In those studies, patients receiving MOL reported improvements in depression, anxiety and stress. MOL aims to maximize client control, whereby clients determine the frequency and duration of therapy sessions. The provision of web-based tools would provide increased control over access to treatment, and MOL provides a particularly useful approach on which to base a web-based intervention thanks to its use of open questioning and its focus on processes over content.
A computerized version of MOL has been developed called Manage Your Life Online (MYLO), and a small, laboratory-based study has found it to be effective for helping healthy participants to address personal problems and reduce problem-related distress (Gaffney et al., Reference Gaffney, Mansell, Edwards and Wright2014). MYLO has been designed to simulate MOL questioning using an automated instant messenger interface. The next step in establishing the effectiveness of MYLO is to carry out a web-based randomized controlled trial where participants can access the programs via the web. The research in this paper therefore aimed to evaluate the usefulness of MYLO compared with an existing computer program (ELIZA; Weizenbaum, Reference Weizenbaum1966) for helping healthy people resolve mildly stressful long-standing problems. This study sought to establish the feasibility of administering MYLO over the internet, as well as replicating the results of a previous laboratory-based trial (Gaffney et al., Reference Gaffney, Mansell, Edwards and Wright2014) which found that both MYLO and ELIZA led to significant decreases in psychological distress immediately after using the programs. We also aimed to replicate the earlier finding of a small effect (d = 0.31) in the difference between the MYLO and ELIZA groups in this larger, more diverse sample. Specifically, it was hypothesized that:
• MYLO and ELIZA would both be associated with significant problem resolution and reductions in problem-related distress and psychopathology.
• MYLO would be rated as significantly more helpful than ELIZA.
• MYLO would be associated with greater improvement in problem distress and psychopathology than ELIZA.
Method
Participants
The study sample consisted of students and staff recruited from the University of Manchester and the University of Liverpool, UK. The study was advertised via the universities’ research volunteering websites and with recruitment posters. Undergraduate Psychology students completed the study in exchange for course credits. Advertisements asked for volunteers currently experiencing a problem that was causing distress. Figure 1 presents progression through the study for all participants.
Summary of recruitment and randomization
Measures
Depression, Anxiety and Stress Scale – Short form (DASS-21) (Lovibond and Lovibond, Reference Lovibond and Lovibond1995)
This is a 21-item version of the DASS scale with three 7-item subscales measuring depression, anxiety and stress over the past 7 days. Items are rated on a 4-point Likert scale from 0 (‘Did not apply to me at all’) to 3 (‘Applied to me most of the time’). Normative studies have found high internal consistency for the scale, with Cronbach alpha values of 0.91 (depression), 0.84 (anxiety) and 0.90 (stress) (Lovibond and Lovibond, Reference Lovibond and Lovibond1995). In our current sample reliability based on the available data (n = 173) was 0.76 (depression), 0.78 (anxiety) and 0.78 (stress).
Problem ratings
Participants were asked to provide their age and gender. They were also asked to briefly describe the problem they would be focusing on when utilizing the program. They were then asked to rate their current distress about the problem (on an 11-point scale from 0, ‘Not distressing at all’ to 10, ‘Highly distressing’) and how easily they believed it would be resolved (on an 11-point scale from 0, ‘Cannot be resolved’ to 10, ‘Easily resolved’). These ratings were repeated at follow-up, but the ease of resolution item was re-worded at follow-up to how much the participant felt the problem had been resolved. At the post-session and 2-week follow-ups participants were also asked ‘How well do you feel the session with the computer went for you?’, again on an 11-point Likert scale (0, ‘Not helpful at all’; 10, ‘Extremely helpful’).
A number of other measures were taken at baseline that are not reported in the current study.
Procedure
The study received ethical approval from the University of Manchester School Of Psychological Sciences Research Ethics Committee (reference number 11363).
Participation in the study was through an online web link, which could be accessed through any computer with an internet connection. Participants were required to click a button to indicate they had read the online information sheet and consent form before proceeding to complete baseline measures. At the start, participants were also asked to provide a username and an email address so that they could be followed up at 2 weeks, although this was optional. At the end of this baseline assessment participants were asked to click a button which randomly generated a link to the MYLO program or an active control programme, ‘ELIZA’.
Participants were free to use their assigned program for as long as they wished, with a suggested minimum of 15 minutes. Each program included a link for participants to end their session. This took them to a post-session questionnaire and feedback form. Two weeks after completing the study, participants who provided email addresses were followed up via email and asked to follow a link to another follow-up questionnaire and feedback form. Cases were matched using the unique username participants had created at the start of the study.
Computer programs
Examples of conversations with the two programs are provided in Table 1.
Examples of conversations with MYLO and ELIZA
MYLO conversation reproduced from Gaffney et al. (Reference Gaffney, Mansell, Edwards and Wright2014), p. 10; ELIZA conversation is an example provided in Weizenbaum (Reference Weizenbaum1966).
ELIZA (Weizenbaum, Reference Weizenbaum1966)
ELIZA is a text-based program designed to communicate in natural language. The program attempts to emulate a Rogerian psychotherapist (Rogers, Reference Rogers1957). ELIZA initiates a conversation by asking the user to tell her about their problem, and the user types messages in and submits using the ‘Enter’ key. The program then replies with questions that aim to get the participant to further discuss their problem. If the user thinks ELIZA has gone off topic, they are free to change topic.
Manage Your Life Online (MYLO; Gaffney et al., Reference Gaffney, Mansell, Edwards and Wright2014)
MYLO is a problem-solving program that, like ELIZA, also aims to facilitate a participant focusing on a problem. However, in addition, it simulates the type of questioning that might be helpful for a person to be asked to prompt thinking about their problems differently and to begin to solve them. The program aims to help people identify the cause of their problems and will pose questions to encourage them to approach the problem in different ways, as an attempt to assist problem solving. It is based on the principles of PCT and aims to aid participants in solving their problems by targeting the same processes as MOL (Carey, Reference Carey2006). MYLO is like most messenger-style applications. To begin, a person types in what is on their mind, and presses ‘OK’. MYLO then asks a question based on what has been typed.
Sample size calculation
Based on Gaffney et al. (Reference Gaffney, Mansell, Edwards and Wright2014), where a Cohen's d of 0.79 was found for the pre/post comparison of distress scores for those who used MYLO, power analysis revealed a minimum group size of 19 would be required for adequate power (0.8). However, comparison of improvements in distress between the MYLO and ELIZA groups in the same study revealed little difference between the groups (d = 0.31). Comparison of the two conditions would therefore require a minimum sample of 104.
The current study employed a mixed 2×3 ANOVA, with group (MYLO or ELIZA) as a between-participants factor and time (pre, post, follow-up) as a within-participant variable. Problem-related distress was the primary outcome variable, with DASS-21, helpfulness ratings, and problem resolution as secondary outcome measures. In order to achieve minimum power we aimed to recruit 120 participants in total, with 60 in each group.
Data analysis
Data were analysed using SPSS for Windows (version 20). The design of the study meant that participants who did not complete measures immediately after using the programs were not included in the analyses because it was impossible to know to which condition they had been randomized. It was therefore impossible to analyse data on an intention-to-treat basis. The main outcomes (problem distress and DASS scores) were analysed using per protocol analysis with the intention to follow-up, using the last observation carried forward as a conservative estimate of outcome when data were missing at follow-up. Only missing data at the 2-week follow-up were imputed. We provide means and number of participants without last observation carried forward for comparison.
The primary outcome measure for the study was problem-related distress. DASS-21, problem resolution, and helpfulness were secondary outcome measures. Primary outcome analyses used general linear modelling repeated measures with group (MYLO, ELIZA) as a between-participants factor and time (baseline, post-intervention, 2-week follow-up) as a within-participants factor. Ratings of helpfulness and problem resolution were compared at post-intervention and 2-week follow-up again using ANOVA.
Results
Sample characteristics
A total of 213 participants completed a baseline questionnaire, of which 171 completed the randomization process and took part in the intervention phase, providing immediate post-study data. Out of those who completed the study, 67 (32 MYLO and 35 ELIZA) provided data at all three time-points (baseline, post-intervention, and 2-week follow-up). Descriptive statistics for the sample with all three time-points are provided in Table 2. At baseline 188 provided information about their age. This ranged from 16 to 70 years, with a mean of 22.08 (SD 7.19). 196 participants provided information on gender and of these, 160 (81.6%) were female. For those who provided data at all time points the average age was 21.28 (SD 6.13) and 56 (90.3%) were female. Participants in the MYLO condition spent an average of 13 minutes using the program, while those in the ELIZA group spent an average of 5 minutes. Overall approximately 20% of sessions lasted longer than 15 minutes (40% MYLO; 7% ELIZA).
Means and standard deviations for all outcome measures at baseline, post-intervention and 2-week follow-up including both observed values and last observation carried forward (LOCF)
Rating of helpfulness
Mixed ANOVA was used to compare participants’ ratings of helpfulness at post-intervention and 2-week follow-up. Mean ratings for both conditions at the two time-points are shown in Fig. 2. We found a significant main effect of group, indicating that at both time-points MYLO participants rated the program as more helpful than ELIZA participants (F (1,60) = 12.98, p = .001, η2 = .18). There was also a significant main effect of time, with participants’ ratings in both groups increasing between post-intervention and follow-up (F (1,60) = 4.23, p < .05, η2 = .07). There was no interaction effect.
Mean helpfulness ratings for MYLO and ELIZA at post-intervention and follow-up
Rating of problem resolution
A 2×2 mixed methods analysis of variance (ANOVA) was conducted with the within-participants factor of time (post-intervention, follow-up) and the between-participants factor of group (MYLO, ELIZA) for participants’ ratings of problem resolution. The mean values for the two groups are presented in Fig. 3. There was a significant main effect for time (F (1,60) = 48.78, p < .001, η2 = .45), indicating that participants experienced further resolution of their problem between using the program and completing the follow-up. The main effect of program was not significant (F (1,60) = 2.49, n.s.), but there was a significant interaction effect (F (1,60) = 5.74, p < .05, η2 = .09), which indicated that immediately after the intervention, participants using MYLO reported significantly higher problem resolution (t (131.27) = 4.76, p < .001). Participants in the ELIZA group experienced further problem resolution between the study and follow-up than MYLO participants.
Mean ratings of problem resolution for participants using the two programs at immediate post-intervention follow-up and 2-week follow-up
At the start of the study participants were asked to rate how easy they expected their problem would be to resolve in order to compare the resolution ratings of those who expected the problem to be easy to resolve with those who expected it to be difficult. Participants who rated the ease of resolution as 5 (the median for the sample) or below were rated as expecting it to be difficult to resolve, with those scoring above 5 rated as expecting it to be easier to resolve. We did not find a significant difference in ratings of resolution between these groups (t (170) = 1.09, n.s.) post-intervention.
Problem-related distress
The primary aim of the study was to investigate the effectiveness of MYLO against a control condition for helping online users to reduce their problem-related distress. Figure 4 illustrates the reductions in problem-related distress over time for the two conditions. We carried out a mixed methods 2×3 ANOVA comparing scores for users of the two programs at baseline, post-study follow-up and 2-week follow-up. As expected, participants in both groups experienced significant improvement in distress over time (F (2,338) = 51.10, p < .001, η2 = .23). Contrasts revealed that improvement continued after using the program, with a significant decrease in scores from baseline to post-intervention (F (1,169) = 36.71, p < .001, η2 = .18), and another significant decrease from post-intervention to follow-up (F (1,169) = 21.93, p < .001, η2 = .16). These decreases were seen for participants in both groups (F (1,169) = .69, p = .41), and there was also no interaction effect of group and time (F (2,338) = 1.32, p = .27).
Mean distress ratings at all three time-points for participants using the two programs
Symptom measures
We explored the effect of the two interventions on total scores on the DASS-21 scale. A 2×3 mixed methods ANOVA was carried out. There was a significant effect of time (F (2,314) = 49.39, p < .001, η2 = .24), representing an overall reduction in symptom scores over the three time-points. Contrasts indicated significant reductions in scores from baseline to post-intervention (F (1,157) = 56.83, p < .001, η2 = .27) and from post-intervention to follow-up (F (1,157) = 9.87, p = .002, η2 = .06). The analysis did not find a main effect of group (F (1,157) = .16, n.s.), or an interaction effect between time and group (F (2,314) = .39, n.s.), indicating that the two programs did not have differential effects on symptom scores over time.Footnote 1
Discussion
The primary aim of the study presented in this paper was to investigate the efficacy of MYLO as a readily available web-based resource, and comparing it with ELIZA, which provided an active control condition with a similar interface. Analyses did not find MYLO to be more effective than ELIZA for bringing about reductions in psychological distress and symptoms of anxiety and depression, with modest but significant reductions observed in both conditions, including further improvement at 2-week follow-up. We found that participants rated MYLO as more effective in resolving their problem immediately after using it. MYLO was also found to be rated as significantly more helpful than ELIZA for resolving participants’ problems. Data on the amount of time participants spent using the programs also indicated that participants used MYLO for longer, which may indicate that this program was considered more acceptable.
These findings are consistent with those from the pilot, laboratory-based trial of MYLO (Gaffney et al., Reference Gaffney, Mansell, Edwards and Wright2014). While MYLO was not found to be more effective than ELIZA, these findings do provide support for the use of web-based interventions for helping individuals with mild to moderate distress. The results also revealed significant differences in symptom reduction between those who met criteria for moderate symptom severity and those who did not, with those meeting criteria for moderate severity experiencing greater improvements at both immediate follow-up and 2 weeks later. Our results provide evidence that web-based interventions using a ‘chatbot’ interface can help with resolving problems and reducing associated distress, even for those experiencing higher levels of distress at baseline. However, based on these findings the possibility that effects were due to natural recovery rather than either program cannot be ruled out. An important consideration for future research is therefore the inclusion of a no-treatment condition to compare the effects of MYLO with natural recovery.
The limitations of the present study's design limit what we can conclude from the results. Although the trial was randomized and included a control condition, the absence of a no-treatment group means we cannot be sure that changes in symptoms are due to the interventions or whether they simply reflect the symptoms improving over time (Posternak and Miller, Reference Posternak and Miller2001). Regression to the mean could explain the differences in change scores between high and low distress participants at the 2-week follow-up. However, results did reveal immediate reductions in distress and symptom scores after using the programs for an average of only 10 minutes, which are less likely to be due to regression to the mean. Despite its limitations the study demonstrates the feasibility of providing web-based interventions using this type of interface, and that such interventions can help to reduce distress in the short-term. Attrition at 2-week follow-up was quite high, and although the use of last observation carried forward provides a conservative estimate of missing observations (Engels and Diehr, Reference Engels and Diehr2003), the high attrition does limit the generalizability of the longitudinal effects found in this study. The design also meant that individuals who dropped out prior to accessing the programs could not be included in last observation carried forward analyses, as it was not known which condition they would have been randomized to. As a result, the data could not be analysed using intention to treat.
Attrition is a problem with unsupported online interventions more generally. Drop-out in the current study was comparable to previous research using unguided web-based CBT, where attrition was over 70% (Christensen et al., Reference Christensen, Griffiths, MacKinnon and Brittliffe2006). It is also possible in the current study that the characteristics of the study sample contributed to attrition, as the problems being discussed were quite short-term and conversations generally lasted less than 20 minutes. It is also possible that participants dropped out because of problems with the programs themselves. Future studies will need to evaluate the user-friendliness of MYLO and of its interface to ensure participants are not lost to follow-up due to problems with the software.
The primary outcome measure in this study was a non-validated scale first used by Gaffney et al. (Reference Gaffney, Mansell, Edwards and Wright2014). The use of non-validated instruments presents some potential limitations in terms of the quality of measurement and also for comparison with literature (Clarke, Reference Clarke2007). There is also evidence that using unpublished scales can lead to positive findings that would not have been obtained using validated instruments (Marshall et al., Reference Marshall, Lockwood, Bradley, Adams, Joy and Fenton2000). Given the number of outcome measures a potential limitation of the current study is the lack of correction for multiple comparisons, with the potential for Type I error.
Future studies will need to address the limitations of the current trial. The lack of differences between the two conditions will need to be addressed by including a no-treatment condition. Any future trials should also include validated measures for the primary outcomes. Recruiting participants already seeking help with problems, for example those seeking low-intensity treatment for anxiety and depression, may help with attrition and would potentially provide a clearer indication of MYLO's usefulness.
The use of an undergraduate student sample also raises limitations. The study advertisements asked for volunteers who were experiencing a current problem which was causing them some distress and which they felt unable to resolve. However, participants were not screened or assessed prior to taking part, raising the possibility that individuals participated who did not feel as distressed about their problem as others. Future studies should address this limitation by specifically recruiting individuals already seeking help with their problems.
Acknowledgements
The authors would like to thank the participants who took part in this study.
Funding: This study was conducted as part of a doctoral degree funded by an interdisciplinary studentship from the UK Economic and Social Research Council and Medical Research Council awarded to Timothy Bird.
Ethical statement: The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, and its most recent revision. The study presented in this paper was approved by the University of Manchester School Of Psychological Sciences Research Ethics Committee (reference number 11363).
Conflicts of interest: The authors have no conflicts of interest with respect to this publication.
Comments
No Comments have been published for this article.