Introduction
Increased access to psychological treatment has been aided by the escalating implementation of computers and the internet into healthcare, such as Beating the Blues (Proudfoot et al., Reference Proudfoot, Goldberg, Mann, Everitt, Marks and Gray2003) and FearFighter (Shaw, Marks and Toole, Reference Shaw, Marks and Toole1999). There is evidence that computerized therapy is an effective alternative to face-to-face therapy for the treatment of mild to moderate anxiety and depression (Proudfoot, Reference Proudfoot2004). However, the acceptability of computer-based self-help to patients remains unclear, and uptake and adherence can be low (Waller and Gilbody, Reference Waller and Gilbody2009). The majority of the evidence base for these interventions is disorder-specific, with the evidence then informing practice and access to therapy. Although cognitive and behavioural techniques can be adapted to address people's idiosyncratic problems, there are benefits from applying therapeutic principles in a transdiagnostic manner from the outset (Mansell, Harvey, Watkins and Shafran, Reference Mansell, Harvey, Watkins and Shafran2009).
One theoretical approach known as Perceptual Control Theory (PCT; Powers, Reference Powers1973) has been used increasingly as a unifying framework for understanding how psychological distress may be maintained across disorders (e.g. Carey, Reference Carey2006; Higginson, Mansell and Wood, Reference Higginson, Mansell and Wood2011; Mansell, Reference Mansell2005). It therefore has the capacity to integrate diverse approaches to understanding psychological distress. For example, a range of cognitive approaches advocate ameliorating loss of control through interventions such as attentional training and goal-setting. According to PCT, at the core of all living things is the dynamic process of “control” (Powers, Reference Powers1973). Since control is essential to living, it is proposed that any problems that occur are essentially problems of control (Carey, Reference Carey2008a). PCT also proposes a hierarchy of control, somewhat akin to the cognitive hierarchy from core beliefs through dysfunctional attitudes to automatic thoughts that are identified within traditional cognitive approaches. Yet, in PCT, problems are not caused by systematic distortions in these systems per se, but by the interference that occurs when a control system at a higher level sets incompatible goals for two differing lower level control systems. This process is called “conflict” in PCT (Carey, Reference Carey2008b) but has parallels in the ambivalence and indecision conceptualized within CBT. An example of a small segment of a control system hierarchy and the hypothesized areas of conflict for a client can be observed in Figure 1.
A representation of the control systems at different levels revealed from a MYLO conversation transcript from this study. Lightning bolts represent conflict between goals. Solid boxes represent explicitly stated goals, and dashed boxes represent an inferred goal
In order to resolve goal conflict, the individual needs to shift awareness to higher levels in their hierarchy in order to elicit a process of “reorganization” (Carey, Reference Carey2006), which is the basic learning mechanism within PCT. Facilitating the process of reorganization is the primary aim of a psychotherapy known as the Method of Levels (MOL; see Carey, Reference Carey2006). MOL is distinct from other therapies as it targets these processes using only open questioning techniques and does not utilize psychoeducation, homework, tools such as thought records, or written formulations. MOL has been tested empirically in treating a range of mental health problems (e.g. depression, anxiety, panic, eating disorder and comorbid presentations) and is associated with significant improvements in depression, anxiety and stress and self-reported level of distress when used as an intervention in primary and secondary care settings (Carey, Carey, Mullan, Spratt and Spratt, Reference Carey, Carey, Mullan, Spratt and Spratt2009). We propose that a computer-based self-help program that harnesses only the essential processes of psychological change identified within PCT would provide a more efficient treatment with tighter links between theory and practice, and increased flexibility for more complex cases.
A self-help computer software package called Manage Your Life Online (MYLO) was designed on the principles of MOL therapy and simulates an MOL-style conversation through an automated “instant messenger” interface. Due to the simplicity of techniques and client-centred nature of MOL, this psychotherapy lends itself well to automation and application to computer-based self-help. MYLO could potentially provide an intervention that is effective for any problem a client is “stuck” with. However, an important consideration is its acceptability to clients and the possible impact of client expectation on outcomes. Positive expectations regarding computer-based self-help are associated with improvement in depressed mood (De Graaf, Huibers, Riper, Gerhards and Arntz, Reference De Graaf, Huibers, Riper, Gerhards and Arntz2009) and clients who have lower expectations of the usefulness of computer-based self-help decline to utilize it (Murray et al., Reference Murray, Pombo-Carril, Bara-Carril, Grover, Reid and Langham2003).
Aims
First, the study attempted to establish the feasibility, acceptability and efficacy of MYLO through comparison with a natural language simulation program called ELIZA (Weizenbaum, Reference Weizenbaum1966). ELIZA is not designed to aid problem solving and controlled for the effects of time spent conversing with a computer. We hypothesized that MYLO would be a more effective problem solving aid than ELIZA. Second, the study investigated whether the mechanisms of psychological change identified within PCT are associated with psychological distress reduction and verify that it is through these processes that MYLO aids problem resolution. We hypothesized that greater awareness of conflict and higher level thinking processes would be associated with larger reductions in distress. Finally, we investigated whether client expectancy of computer-based problem solving affects the effectiveness of MYLO and, if so, whether this relationship was mediated by the prevalence of PCT processes within conversation transcripts. We hypothesized that clients with positive expectancies of computer-based problem solving would experience greater reductions in distress, greater problem resolution, and rate MYLO as more helpful. Significantly, these outcomes would be mediated by the degree of higher level awareness.
Method
Participants
Forty-eight student participants were recruited from the University of Manchester through an advertisement on the university SONA system and volunteering website, recruitment posters and word of mouth. Ethical approval was gained from the University of Manchester Ethics Committee, which conforms to the British Psychological Society (BPS) ethical guidelines. Participants were requested to have a problem that was currently causing them distress and that they would be willing to disclose to a computer program. Undergraduate psychology students received course credits for participation.
Materials
Problem ratings
Participants were asked to provide their age, gender and a concise description of their current problem (e.g. with workload, relationships, family). Participants then rated on separate 11-point scales the degree of distress the problem was causing, from 0 (not distressing at all) to 10 (highly distressing) and its solvability from 0 (cannot be resolved) to 10 (easily resolved). The distress rating was repeated at post-intervention and follow-up, whereas the Resolution Rating was reworded at these stages to inquire to what extent the participants had felt their problem had been solved, 0 (not resolved at all) to 10 (completely resolved). Additionally, at these stages, participants were asked “How well do you feel the sessions with the computer went for you?” on a scale of 0 (not helpful at all) to 10 (extremely helpful) to generate Helpfulness Ratings and space was also provided for qualitative responses.
Depression, Anxiety and Stress Scale (DASS-21; Lovibond and Lovibond, Reference Lovibond and Lovibond1995)
A 21- item scale that measures depression, anxiety and stress over the previous week on a 4-point scale from 0 (did not apply to me) to 3 (applied to me most of the time). Scores range from 0 to 42 in each of the three domains and are calculated by summing the scores of the seven items. The scale has high internal consistency with Chronbach's alpha values of Depression 0.91; Anxiety 0.84 and Stress 0.90 (Lovibond and Lovibond, Reference Lovibond and Lovibond1993).
Likelihood to use MYLO effectively scale (LIME)
This is a newly developed 24-item scale designed to measure client expectancies towards computer-based problem solving that was developed following earlier piloting and extraction of internally consistent items. There are two subscales: Positive Expectancy (14 items) measures attitudes thought to predict engagement with the computer-based problem solving (e.g. The computer is just another tool that can help to solve my problems) and Negative Expectancy (10 items) measures attitudes predictive of “disengagement” (negative expectancy) with computer-based problem solving (e.g. Talking to a computer is useless as it cannot offer me advice). Participants rate their belief in each statement on a scale of 0 (I don't believe this at all) to 100 (I believe this completely). An average score for each subscale is calculated and can range from 0 to 100. The scale has high internal consistency with Cronbach's alpha values of Positive Expectancy = .91 and Negative Expectancy = .85.
Computer programs
ELIZA (Weizenbaum, Reference Weizenbaum1966)
The version of ELIZA used for this study can be accessed online at http://www.cyberpsych.org/eliza. ELIZA was one of the first computer programs to attempt to simulate a conversation and operates using a script based on Rogerian psychotherapy (Rogers, Reference Rogers1957). ELIZA first asks the participant “Hello, let's talk.” The participant is asked to start the conversation by typing their problem into the response box and pressing the return key. ELIZA then replies with questions aimed at encouraging a conversation about the participant's problem.
Manage Your Life Online (MYLO)
MYLO can be accessed online at http://manageyourlifeonline.org. It is an automated computer-based self-help program that simulates a conversation between a client and therapist with the aim to aid participants in solving their problem by adhering to principles of MOL therapy (Carey, Reference Carey2006). Participants are required to create a login for MYLO to gain access to an introductory page explaining what MYLO is, what to expect, and how to get the most from the conversation.
MYLO operates by analysing the client's input of text for key terms and themes. It responds with questions about the problem aimed at encouraging conflict awareness and facilitating higher level awareness. Table 1 illustrates 14 of the 52 themes currently identified by MYLO and provides examples of matched terms and questions relevant to each theme. An extract of a MYLO conversation transcript with details of MYLO's working from this study can be seen in Table 2.
Example of 12 MYLO themes with corresponding matched terms or phrases and examples of consequent questions
Example of a dialogue in MYLO with the coding of client statements that informs the next question revealed
Prior to commencing a conversation with MYLO, participants were asked to rate their mood from 1 (very bad) to 9 (very good) and severity of their problem from 1 (mild) to 9 (severe). Next, the conversation screen appeared and MYLO asked the participant to describe their problem. When ready, the participant typed into the response box and pressed “ok”. The conversation history was displayed above in a history box. Every 4–8 responses (selected randomly), MYLO asked participants to rate their mood again and how helpful the conversation was since the last rating on the same 9-point scale. Finally, when the participant terminated the conversation, overall measures of mood, helpfulness, and problem severity were requested.
Procedure
Participants were randomly assigned by coin toss to either the MYLO or ELIZA experimental groups by a researcher prior to their initial 45-minute slot at an experimental cubicle. Researchers were therefore not blinded to treatment allocation. Participants read the participant information sheet, which included details of crisis contacts (university counselling services and Samaritans) should they require them. Participants then signed a consent form detailing that they consented to their anonymous text-based conversation being analysed by the researcher. Participants created unique usernames and were requested to write these at the top of each questionnaire to enable anonymous data matching. Participants in the MYLO condition were also asked to create a password in order to log in to the program.
Participants were then asked to complete the Problem Ratings, DASS-21, and LIME measures and the experimenter left the room for 10 minutes. Next, participants were asked to read usage instructions for MYLO or ELIZA, followed by a short demonstration of how to use their assigned program. Participants were then left alone to converse with their assigned program for up to 20 minutes. The average length of time spent using the MYLO program was 19.23 minutes (SD = 0.002) but data were unavailable for time spent using ELIZA. Finally, participants completed the DASS-21, LIME scores for a second time, and the Problem Ratings. Participants recorded their login details and web-address for their assigned program at the end of the session should they wish to use it outside of the session. Two weeks following this session, participants were asked to attend a 15-minute follow-up where they were asked to complete the DASS-21, LIME, and Problem Ratings again.
Qualitative coding instructions
Participant conversations with MYLO were coded and scored for evidence of three processes fundamental to psychological change according to PCT. Conversation transcripts are first read a number of times to familiarize with the data. Next, the script is re-read and scored on a table for the frequency of 1) conflict awareness, 2) awareness of higher order goals and 3) reorganization of higher order goals (detected as new insights, perspectives or appraisals). Finally, considering the frequencies of the three processes outlined above, each conversation transcript is given an overall rating between 1 (no awareness of conflict or higher level goals and does not consider new perspectives) and 6 (good awareness of conflict and higher level goals and many new perspectives gained on problem) which is called the Higher Level Awareness Score. A random sample of eight conversations were also scored by another researcher familiar with the fundamental principles of PCT and MOL. Pearson's correlational analyses were conducted to ascertain the inter-rater reliability. Frequency of statements regarding conflict were not significantly correlated between raters, r (7) = .48, p = .23. Frequency of new insights, perspectives or appraisals were significantly positively correlated, r (7) = .85, p < .01. Frequency of higher order goals or beliefs were significantly positively correlated, r (7) = 0.79, p < .05. Overall, Higher Level Awareness Scores of the sample were significantly positively correlated, r (7) = .95, p < .01.
Results
Six participants’ data were removed from analysis; four due to a malfunction with the server that hosted MYLO, one due to incomplete measures, and one participant did not attend follow-up (see Figure 2). In total, 42 participants (9 male) aged 18–32 (M = 21.4, SD = 3.1) were included in the analysis. There were 22 participants in the MYLO condition (7 male) and 20 participants in the ELIZA condition (2 male). All analyses were conducted using SPSS for Windows (Version 17). Descriptive statistics for distress, resolution, DASS-21 and LIME scores pre- and postintervention and follow-up are presented in Table 3. No participants reported accessing their assigned program in the interim between intervention and follow-up.
Means and standard deviations for measures taken pre and postintervention and follow-up
* These ratings represent how easy the problem would be to solve, rather than the degree to which the problem has been solved as measured at postintervention and follow-up.
Trial CONSORT diagram
The efficacy of MYLO compared with ELIZA was assessed using two two-way mixed analyses of variance (ANOVA) with the between subject factor of group (MYLO or ELIZA) and the within subject factor of time (postintervention and follow-up) for Helpfulness Ratings and Ratings of Resolution.
Helpfulness
There was a significant effect of group indicating that Helpfulness Ratings for MYLO were significantly higher than those for ELIZA, F (1, 40) = 8.22, p < .05. Figure 3 illustrates the mean Helpfulness Ratings for MYLO and ELIZA postintervention and at follow-up. There was no change from postintervention to follow-up, F (1, 40) = 2.38, p = .13, and a non-significant interaction between time and group, F (1, 40) = 2.94, p = .09.
Bar graph indicating mean Helpfulness Ratings for MYLO and ELIZA conditions at post-intervention and follow-up
Problem resolution
There was a significant effect of group, indicating that Ratings of Resolution for MYLO were higher than those for ELIZA, F (1, 40) = 5.47, p < .05. Ratings of Resolution significantly changed from postintervention to follow-up, F (1, 40) = 6.03, p < .05, suggesting that further resolution of their problem occurred after the session and prior to follow-up. There was a non-significant trend for an interaction between time and group, F (1, 40) = 3.62, p = .06. This may indicate that participants in the ELIZA condition experienced further resolution of their problem in the interim between their session with ELIZA and follow-up (see Figure 4).
Bar graph indicating mean Ratings of Resolution for MYLO and ELIZA conditions at post-intervention and follow-up
Self-reported distress
To investigate their effectiveness in reducing distress, two 2 × 3 mixed analyses of variance (ANOVA) were performed with the between subject factor of group (MYLO or ELIZA) and the within subject factor of time (pre- and postintervention and follow-up) for scores of self-reported distress on the PDRS. There was a significant main effect of time on distress, F (2, 80) = 27.51, p < .01. Contrasts revealed that distress score significantly reduced between preintervention and follow-up, F (1,40) = 50.16, p < .01 and distress score significantly decreased between postintervention and follow-up, F (1,40) = 8.65, p < .01. There was no significant effect of group, F (1, 40) = 2.44, p = .13, and no significant interaction effect between time and group, F (2, 80) = 1.00, p = .37, indicating that the specific computer program did not have a differential effect on change in distress scores over time.
Total DASS-21
There was a significant main effect of time on Total DASS-21 score, F (1.24, 49.57) = 6.19, p < .05. Contrasts revealed that Total DASS-21 score significantly reduced between preintervention and follow-up, F (1, 40) = 8.45, p < .01 but there was no significant decrease in depression scores between postintervention and follow-up, F (1, 40) = 4.06, p = .05. There was no significant effect of group, F (1, 40) = .87, p = .36 and no significant interaction effect between time and group, F (1.24, 49.57) = .50, p = .52, indicating that the specific computer program did not have a differential effect on change in Total DASS-21 scores over time.
The mechanism of psychological change
In order to investigate whether the mechanism of psychological change identified within PCT predicted psychological distress reduction, qualitative analysis of MYLO conversation transcripts were conducted. This analysis afforded participants in the MYLO condition with a Higher Level Awareness Score, which indicates the degree to which the proposed processes of psychological change were present in the participant's conversation with MYLO. First, Difference in Distress scores between preintervention and postintervention were calculated. Pearson's correlational analyses were conducted to investigate whether Higher Level Awareness Scores were associated with Difference in Distress score, Helpfulness Rating and Ratings of Resolution postintervention (Table 4). Greater higher level awareness throughout conversation with MYLO was correlated with greater distress reduction, larger problem resolution and more positive expectancies of MYLO. Furthermore, positive expectancy of MYLO was significantly correlated with higher helpfulness ratings and problem resolution, and negative expectancy reflected the opposite pattern.
Correlation matrix of Higher Level Awareness Score, Difference in Distress score and postintervention scores of Helpfulness Ratings, Ratings of Resolution, Positive and Negative Expectancy scores
*Correlation is significant at the 0.05 level (two-tailed).
**Correlation is significant at the 0.01 level (two-tailed).
The effect of client expectancy on the efficacy of MYLO
We wished to investigate the extent to which the two subscales of the expectancy measure predicted the degree of problem resolution. Therefore, a composite Expectancy Score was calculated by reverse scoring the Negative Expectancy items and calculating the average across both subscales, to reflect participants’ overall likelihood of engagement with MYLO. Rating of Resolution postintervention was entered as the outcome variable and Expectancy score entered as the predictor variable. The regression model was significant R 2 = .43, F (1, 20) = 15.29, p < .01 and Expectancy score accounted for 43% of the variance in Rating of Resolution. Expectancy score was a significant predictor of Rating of Resolution (beta = .66, t = 3.91, p < .01). In light of these results, we investigated whether the relationship between expectancy and problem resolution is mediated by the degree of higher level thinking processes and conflict awareness evident in participant's MYLO conversation transcripts. A mediation analysis using Expectancy scores was conducted following the 4-step method specified by Baron and Kenny Reference Baron and Kenny(1986) and is outlined below (see Figure 5).
Standardized regression coefficients for the relationship between Expectancy score and Rating of Resolution as partially mediated by Higher Level Awareness Score for each of the 4 paths specified in the Baron and Kenny method (Reference Baron and Kenny1986) *indicates significance at p < .05. ** indicates significance at p < .01.
First, three regression analyses were conducted. The first analysis was conducted with Higher Level Awareness Score as the outcome variable and Expectancy score postintervention as the predictor variable (Path A on Figure 5). The regression model was significant R2 = .21, F (1, 20) = 5.38, p < .05 and Expectancy score accounted for 21% of the variance in Higher Level Awareness Score. Expectancy score was a significant predictor of Higher Level Awareness Score (beta = .46, t = 2.32, p < .05). Therefore, participants with higher expectancy scores are more likely to consider higher level goals/beliefs and produce more new perspectives or appraisals on their problem during the session. The second analysis was conducted as above, with Rating of Resolution as the outcome variable and Expectancy score postintervention as the predictor variable (Path C). The regression model was significant R2 = .43, F (1, 20) = 15.29, p < .01 and Expectancy score accounted for 43% of the variance in Rating of Resolution. Expectancy score was a significant predictor of Rating of Resolution (beta = .66, t = 3.91, p < .01). The third analysis was conducted with Rating of Resolution as the outcome variable and Higher Level Awareness Score as the predictor variable (Path B). The regression model was significant R2 = .28, F (1, 20) = 7.84, p < .01 and Higher Level Awareness Score accounted for 28% of the variance in Rating of Resolution. Higher Level Awareness Score was a significant predictor of Rating of Resolution (beta = .53, t = 2.80, p < .01).
Finally, a fourth multiple regression analysis was conducted with Rating of Resolution as the outcome variable and both Higher Level Awareness Score and Expectancy score as the predictor variables (Path C’). This final analysis met the two requirements for a partial mediator effect: 1) the hypothesized mediator, Higher Level Awareness Score was a significant predictor of Rating of Resolution (beta = .53, t = 2.80, p < .01); 2) Expectancy score was less predictive of Rating of Resolution score (beta = .53, t = 2.87, p < .05). However, a Sobel test (Sobel, Reference Sobel and Leinhardt1982) indicated that this only represented a trend towards statistical significance (z = 1.75, p = 0.08). Therefore, the non-significant reduction in direct association between Expectancy score and Rating of Resolution when Higher Level Awareness Score was in the model partially supported the hypothesis that conflict awareness, new insights, perspectives or appraisals, and awareness of higher order goals is at least one of the mediators in the relationship between attitudes towards computer-based problem solving and self-reported problem resolution (see Figure 5). Note that the results were very similar when the analysis was also conducted with the expectancy subscales entered separately.
Discussion
MYLO was rated as significantly more helpful and facilitated significantly larger reductions in self-reported problem resolution than ELIZA and gains were sustained at follow-up. Both MYLO and ELIZA elicited significant reductions in distress, depression, anxiety and stress and there was no effect of group on outcomes. The findings provide initial support for the acceptability of the MYLO program and the utilization of an automated text-based conversation method of self-help intervention. However, the efficacy of the MYLO program in reducing psychological distress beyond the effects of a control condition were not observed and will require further evaluation in a larger trial, or using a “no treatment” control group.
Greater awareness of conflict and higher level processes during conversation with MYLO predicted greater distress reduction, larger problem resolution and higher helpfulness ratings of MYLO. The findings offer tentative support for the proposal that reorganization is a core element to psychological change and that reorganization follows awareness (Carey, Reference Carey2006; Powers, Reference Powers1973). Crucially, the study has provided a qualitative, within-session account of psychological adjustment from the perspective of the client that is consistent with the quantitative findings and the theoretical tenets of PCT.
Expectancy was not significantly related to Difference in Distress score between pre- and postintervention, suggesting that attitudes towards computer-based therapy do not impact upon the efficacy of MYLO as an aid to problem solving. However, the results of multiple regression analyses indicated that participants who were more willing to engage with MYLO experienced greater problem resolution and rate the MYLO program as being more helpful than participants who were less positive about computer-based problem solving. Additionally, participants who were likely to disengage with MYLO experienced less resolution of their problem than those who were less pessimistic about computer-based problem solving.
The results of a mediation analysis indicated that participants who have greater positive expectancies and lower negative expectancies towards computer-based problem solving experienced significantly larger problem resolution and this was partially due to greater experience of the psychological mechanisms essential to psychotherapeutic change, according to PCT.
These results provide initial support for the relationship between client expectations and clinical outcome (e.g. De Graaf et al., Reference De Graaf, Huibers, Riper, Gerhards and Arntz2009). Specifically, clients who express greater positive expectancies and fewer negative expectancies regarding computer-based problem solving are more likely to be co-operative with MYLO by answering questions in an engaged and comprehensive manner. Consequently, it is argued that these clients gain greater awareness of areas of conflict, experience higher level awareness more frequently, and reappraise goals more readily than those who are less positive and more negative about engaging with computer-based problem solving. This in turn affects the extent to which their problem is resolved. These findings are particularly relevant for practice as, if attitudes towards computer-based problem solving affect engagement and outcome, it is important to consider the views of clients in order to increase the likelihood that treatment will be acceptable and effective.
In terms of limitations, the sample size is small and selective; the majority of participants were female, White British undergraduate students who received course credits for participation. The randomization procedure of coin-tossing would be corrected by a computational system. Coding of the MYLO conversation transcripts would benefit from further testing to verify the procedure as a robust measure of the prevalence of higher level awareness. The results gained with an initial session of a maximum of 20 minutes with MYLO are encouraging but need to be extended in future. Moreover, measuring the total number of exchanges between clients and MYLO in future trials may give an empirical measure of the client's degree of engagement with MYLO. Finally, the Likelihood to use MYLO Effectively Scale (LIME) could be further validated and shortened.
Leading on from this study, MYLO needs to be rigorously tested using a large randomized control trial. The coding system used to rate conversation transcripts for evidence of key processes of psychological change according to PCT could be used in further qualitative research investigating the mechanisms of psychological change during a regular MOL therapy session or client interview. Finally, further research into the acceptability of computer-based self-help programs is required as the effectiveness of this medium relies at least to some extent on the client's engagement and commitment. Telephone or video-based communication with a person rather than a computer may be preferable for some people.
To conclude, the findings provide initial support for the acceptability of the MYLO program in a non-clinical sample although its efficacy as an innovative computer-based aid to problem solving remains unclear. Nevertheless, the findings provide early support for the mechanisms of psychological change identified within PCT and highlight the importance of client expectations on predicting engagement in computer-based self-help.
Acknowledgements
Thank you to Yu Li, and the ESRC Grant RES 060-25-0044.
Comments
No Comments have been published for this article.