Introduction
Reduced approach behaviour has long been implicated in mood-related psychopathology (Lewinsohn and Graf, Reference Lewinsohn and Graf1973). A likely neurobiological substrate of such behaviour is dopaminergic (DA) projections in frontostriatal circuits (Schultz et al., Reference Schultz, Tremblay and Hollerman2000; Hamid et al., Reference Hamid, Pettibone, Mabrouk, Hetrick, Schmidt, Vander Weele, Kennedy, Aragona and Berke2016). A relevant DA signal is the reward prediction error (RPE) – an increase in neuronal firing after unexpected rewards (Bayer and Glimcher, Reference Bayer and Glimcher2005) – which is thought to regulate learning about which cues predict future rewards, leading to cue-motivated approach behaviour.
To measure this signal in humans, laboratory-based reward learning (RL) paradigms can be administered in which certain cues are probabilistically associated with rewards. The RPE signal can then be modelled using a computational approach (Sutton and Barto, Reference Sutton and Barto1998) to examine, with functional Magnetic Resonance Imaging (fMRI), wherein the brain activation correlates with this signal [model-based fMRI (O'Doherty et al., Reference O'Doherty, Hampton and Kim2007)]. Across many studies in healthy controls, RPE signals have been found in the ventral and dorsal striatum (Chase et al., Reference Chase, Kumar, Eickhoff and Dombrovski2015). Interestingly, studies by Kumar et al. (Reference Kumar, Waiter, Ahearn, Milders, Reid and Steele2008) and Gradin et al. (Reference Gradin, Kumar, Waiter, Ahearn, Stickle, Milders, Reid, Hall and Steele2011) report this signal to be blunted in medicated individuals diagnosed with major depressive disorder (MDD). However, Rothkirch et al. (Reference Rothkirch, Tonn, Kohler and Sterzer2017) did not replicate this finding in unmedicated MDD patients. Hence, it remains unknown whether blunted striatal RPE signals underlie MDD, or whether previous findings were confounded by medication use.
A complementary approach to investigating reward-related behaviour comes from the Experience-Sampling Method (ESM) (Csikszentmihalyi and Larson, Reference Csikszentmihalyi and Larson1987). This method samples affect and behaviours frequently and in the context of one's daily life, thereby revealing how moment-to-moment experiences and behavioural patterns can interact differently at different stages of psychopathology (Myin-Germeys et al., Reference Myin-Germeys, Oorschot, Collip, Lataster, Delespau and van Os2009; Wichers, Reference Wichers2014). Interestingly, despite the finding that individuals with depressive symptoms reported a similar increase in Positive Affect (PA) following a pleasant event (Bylsma et al., Reference Bylsma, Taylor-Clift and Rottenberg2011; Thompson et al., Reference Thompson, Mata, Jaeggi, Buschkuehl, Jonides and Gotlib2012, van Roekel et al., Reference van Roekel, Bennik, Bastiaansen, Verhagen, Ormel, Engels and Oldehinkel2016), they reported relatively fewer pleasant events (van Roekel et al., Reference van Roekel, Bennik, Bastiaansen, Verhagen, Ormel, Engels and Oldehinkel2016), indicating reduced reward approach behaviour despite intact hedonic reactivity. This latter finding was extended by Bakker et al. (Reference Bakker, Goossens, Lange, Michielse, Schruers, Lieverse, Marcelis, van Amelsvoort, van Os, Myin-Germeys and Wichers2017), who found depressive symptoms to be associated with a reduced active behavioural response following reports of anticipating a pleasurable situation, despite similar PA responses to active behaviour.
Importantly, even though both neuroimaging and ESM literature refer to reward-related behaviour and its relevance in MDD research, the relationship between the two different approaches has rarely been investigated. Combining ESM and neuroimaging techniques can illuminate how neural abnormalities might be manifested in real-world behaviours as well as enrich the construct validity of ESM measures (Wilson et al., Reference Wilson, Smyth and MacLean2013). Heller et al. (Reference Heller, Fox, Wing, McQuisition, Vack and Davidson2015) showed that the duration of ventral striatal reactivity to winning money in a fMRI scanner was associated with the duration of PA increase in response to winning money in a task performed in the field, showing that neural processes – unfolding over seconds – and the experience of emotions – lasting minutes – might be traced back to common pathways. In the context of RL and real-world behaviour; a prior Positron Emission Tomography (PET) study in healthy controls found RL-induced striatal DA release to be associated with daily-life reward-oriented behaviour (Kasanova et al., Reference Kasanova, Ceccarini, Frank, Amelsvoort, Booij, Heinzel, Mottaghy and Myin-Germeys2017). However, the PET paradigm used in Kasanova et al. (Reference Kasanova, Ceccarini, Frank, Amelsvoort, Booij, Heinzel, Mottaghy and Myin-Germeys2017) could not separate DA responses to unexpected rewards (e.g. RPE) and reward-predicting stimuli. Hence, the current study will be the first to investigate how model-based striatal RPE responses relate to behaviour in a real-world context.
In doing so, we focused on an ESM measure hypothesized to capture a daily-life process related to RPE. Specifically, since the RPE involves a clear anticipatory component, it was examined how a momentary report of looking forward to a situation in the future (i.e. the anticipation of something pleasant) was related to the subsequent report of enjoyment of being engaged in an activity (~90 min later). A higher association between these time-lagged measures could be an indication that one is better able to predict a pleasurable activity sometime later, or that one is better able to change behaviour towards pleasant activities following the anticipation of reward. Both processes are possible results from intact stimulus-response learning processes, which is assumed to be driven by adequate RPE signals.
In sum, reduced Positive Valence System function, and specifically impaired model-based fMRI RPE signals, could be important in MDD. In this context, the current study aims to add insight into two unanswered questions; first, it is unclear how RPE-related neural activity might be disrupted along the continuum of depressive symptoms in the absence of antidepressant treatments. Second, little is known about how RPE brain signals translate into daily-life reward dynamics. Hence, the current study examined – in a sample of non-help-seeking and unmedicated individuals along the depression continuum (up to moderate severity) – the association between RPE brain activity and depressive symptoms as well as a relevant ESM measure.
Methods
Participants
A total of 132 young adults (age 16–25) were included between September 2013 and January 2017 as part of the SMARTSCAN study – a randomized controlled trial examining the effect of psychotherapeutic training on daily life functioning and neurobiological correlates, conducted at Maastricht University Medical Centre (Dutch Trial Register nr.: NTR3808). Since the current paper focuses on a small part of the SMARTSCAN baseline, only procedures relevant to the current analyses will be discussed.
Participants reporting symptoms along the depression continuum were recruited from the general population via advertisements in public places and social media. Individuals with sub-threshold symptoms of depression [Mild/Moderate Depression (m/m-D) group: n = 81] were oversampled by specifically including those with a Montgomery–Åsberg Depression Rating Scale [MADRS (Montgomery and Åsberg, Reference Montgomery and Åsberg1979)] total score of 10 or higher, unless they received current psychiatric/psychological treatment or had a significant need for care as assessed by a psychiatrist. Participants with a MADRS score below 10 [No/Low Depression (n/l-D) group: n = 51] were included, unless they reported current and/or lifetime psychological/psychiatric treatment or a psychiatric diagnosis [as assessed with the Mini International Neuropsychiatric Interview – MINI, (Overbeek et al., Reference Overbeek, Schruers and Griez1999)]Footnote †Footnote 1. Overall, participants were excluded based on MRI contraindications (e.g. metal implants, pregnancy, claustrophobia, neurological disorders) or being left-handed.
The Medical Ethics Committee of Maastricht University Medical Centre approved all study procedures (protocol number: NL41929.068.12 / METC 12-2-072), and all participants provided written informed consent (additionally signed by their proxy if age<18).
Procedure
Study procedures are summarized in Fig. 1a. Depressive symptomatology was assessed with an interviewer-rated [MADRS (Montgomery and Åsberg, Reference Montgomery and Åsberg1979)] and a self-report [Inventory of Depressive Symptomatology (IDS-SR); Rush et al., Reference Rush, Gullion, Basco, Jarrett and Trivedi2009)] scale, since it has been established that each can measure unique aspects of depression (Uher et al., Reference Uher, Farmer, Maier, Rietschel, Hauser, Marusic, Mors, Elkin, Williamson, Schmael, Henigsberg, Perez, Mendlewicz, Janzing, Zobel, Skibinska, Kozel, Stamp, Bajs, Placentino, Barreto, McGuffin and Aitchison2007; Uher et al., Reference Uher, Perlis, Placentino, Dernovšek, Henigsberg, Mors, Maier, McGuffin and Farmer2012). MADRS score was measured both at the inclusion (first) and scan (second) sessions, and IDS-SR was collected only at the scanning session.
Study and task procedures. (a) Study procedure. (b) Task procedure, adapted from Gold et al. (Reference Gold, Waltz, Matveeva, Kasanova, Strauss, Herbener, Collins and Frank2012). MADRS, Montgomery–Åsberg Depression Rating Scale (Montgomery and Åsberg, Reference Montgomery and Åsberg1979); MINI, Mini International Neuropsychiatric Interview (Overbeek et al., Reference Overbeek, Schruers and Griez1999); IDS-SR, Inventory of Depressive Symptomatology (Self-Report) (Rush et al., 2009); RL, Reinforcement Learning; fMRI, functional Magnetic Resonance Imaging; ESM, Experience Sampling Method, m/m-D, participants with moderate symptoms of depression; n/l-D, participants with low symptoms of depression. 1One participant only carried the device for 7 days since initially included in another arm of the SMARTSCAN study.
Reinforcement learning task
The task was adapted from Gold et al. (Reference Gold, Waltz, Matveeva, Kasanova, Strauss, Herbener, Collins and Frank2012, see Fig. 1b). On each trial, participants had to select one stimulus in one of three pairs of stimuli (gain, loss or neutral; 80 trials per condition). Participants were instructed to find out, within each pair, which stimulus was ‘better’ most of the time based on the feedback received after each selection (Gain:+€0.20 v. no gain; Loss: no loss v. −€0.20; Neutral: no monetary outcome but written feedback, ‘Correct’ v. ‘Incorrect’). Selection of the better picture (optimal choice) resulted in a better outcome in 70% of trials; the other stimulus of the same pair associated with the optimal outcome in 30% of trials. Stimuli were presented side by side; the position of the stimuli was counterbalanced across trials. Conditions were pseudo-randomly ordered within 4 blocks of 60 trials (i.e. 20 trials per condition per block, total duration: ~24 min). The current paper focuses only on gain trials, but the other two conditions were modelled in the imaging analyses (see below).
MRI data acquisition and processing
Data acquisition
MRI scans were acquired using a 3T Siemens Magnetom Prisma scanner at the Scannexus facilities, Maastricht, the Netherlands. See the Supplemental Materials for details.
Processing
For each individual, functional data were slice-time corrected, realigned to the mean image, co-registered to the anatomical image and normalized to 2 × 2 × 2 MNI space and smoothed with a 4 mm Gaussian kernel using the Statistical Mapping software package (SPM12). See Supplemental Materials for a more detailed description.
ESM procedure and measures
Procedure
Participants carried a dedicated device (the PsyMate©), which was programmed to emit a signal (‘beep’) at semi-random moments, within 90 min blocks, between 07:30 and 22:30 (i.e. 10 beeps a day). At each beep, participants completed a brief beep-questionnaire on the PsyMate© including reports on current (positive and negative) mood, (social and physical) context, daily events and activities, on a seven-point Likert-scale. Participants were able to fill in the questionnaire up to 10 min after the initial beep but were asked to fill in the questionnaires as soon as possible in order to reduce recall bias. All but one participant carried the device for 15 days (receiving 150 beeps); one participant who was initially included in another arm of SMARTSCAN but fulfilled criteria for the current sample, did so for 7 days (receiving 70 beeps). Following earlier work (Delespaul, Reference Delespaul1995), participants who filled in less than 30% of received beeps were excluded from analyses (excluded: 1 n/l-D, 1 m/m-D). In addition, subjects who filled in beep-questionnaires on less than 6 days were also excluded (excluded: 1 m/m-D).
Measures
Two ESM items (see below) were combined in a statistical modelFootnote 2 in order to form one ESM reward measure, by predicting the level of activity pleasantness at time t with the level of reward anticipation at time t-1, thereby indicating whether higher levels of reward anticipation were followed by higher levels of activity pleasantness, both actualized as follows:
Reward anticipation was assessed in two steps: (1) ‘Think about what you consider to be the most important situation in the next hour…’ and (2) ‘How much are you looking forward to this situation?’.
Activity pleasantness was assessed in two steps: (1) ‘Think about what you were doing right before the beep-signal…’ and (2) ‘I enjoy doing this’.
Analyses
RL task – behaviour
A standard Q-learning algorithm was used to calculate the expected value of, and prediction error after, each choice (Sutton and Barto, Reference Sutton and Barto1998). Additionally, it was tested how well the RL model fitted the observed data relative to chance (see Supplement for more detailed information).
Associations with depressive symptoms were examined by regressing total MADRS and IDS-SR scores with a model-fit measure (pseudoR2) and learning parameters.
RL task – imaging
First-level general linear models included 19 regressors which were convolved with a hemodynamic response function: cue and feedback onsets of gain, loss and neutral trials (6 regressors); parametric modulation of cue and feedback onset times (only gain and loss condition) by model-derived expected value and prediction error respectively (4 regressors); cue and feedback onsets of non-response trials (2 regressors); six motion realignment parameters and a constant term (7 regressors). A linear contrast of the regression coefficient of RPE was computed at the individual level and resulting contrast images were taken to conduct within-subject t test. Whole brain analyses across all subjects were run with a cluster-extent based threshold of p < 0.05 FWE with p < 0.001 as the initial cluster-forming threshold.
ROI analyses. Anatomically constrained bilateral putamen and nucleus accumbens (NAcc) were extracted from a recent meta-analysis of RPE studies in healthy controls (Chase et al., Reference Chase, Kumar, Eickhoff and Dombrovski2015; Fig. 2b). These anatomical masks were created from the FSL Harvard-Oxford subcortical atlas using 40% probability threshold. RPE parameter estimates from these four ROIs were derived using the REX toolbox (https://www.nitrc.org/projects/rex/).
fMRI results and correlations between RPE and self-reported depressive symptoms. (a) Active clusters during Reward Prediction Error (average over the whole group). (b) Regions of Interest, extracted from Chase et al. (Reference Chase, Kumar, Eickhoff and Dombrovski2015). (c) Mean activation per Region of Interest for each participant. (d) Scatterplots of RPE beta weight and IDS-SR total score for the right Nucleus Accumbens and Putamen. L, left; R, right; NAcc, Nucleus Accumbens; RPE, reward prediction error; IDS-SR, Inventory of Depressive Symptomatology (Self-Report).
Association of Reward learning (RPE) signals with depressive symptoms and a measure of real-life motivated behaviour (measured using ESM).
Depressive symptoms. As our aim was to investigate RPE-related neural activity along the continuum of depressive symptoms, we adopted a dimensional approach. RPE signals from each ROI were, therefore, linearly regressed with total MADRS score and total IDS-SR score using the REGRESS command in STATA 13.1 (StataCorp, 2013).
ESM. It was examined whether RPE moderated the association between reward anticipation at time t-1 and activity pleasantness at time t. ESM data have a hierarchical structure (multiple observations – Level 1 – are clustered within participants – Level 2). Multilevel (mixed effects) linear regression analyses take the variability associated with each level of nesting into account (Snijders and Bosker, Reference Snijders and Bosker1999). For these analyses, the MIXED command in STATA 13.1 (StataCorp, 2013) was used. To control for potential time trends in the ESM data, analyses controlled also for sampling time (in days, including two decimals, starting from midnight of the first sampling day increasing up to the last completed beep questionnaire) as well as the time within a day (in minutes, starting from midnight that day). Since we were mostly interested in within-person effects, predictors of the analyses with lagged variables were person-mean-centred. These models additionally included the lagged version of the outcome variable in order to control the effect of the other lagged predictors for the autocorrelation in the dependent variable. The structure of the residual errors within the lowest-level groups was left to default (i.e. independent and identically distributed with one common variance)Footnote 3. Lastly, since not all models in which the variance-covariance matrix of the random effects was set to unstructured (thereby allowing estimation of all variances and covariances separately) converged, this structure was set to an identity matrix in which covariances were assumed to be 0. In addition to a standard maximum likelihood approach (using the observed information matrix to estimate the variance-covariance matrix of the fixed effects), we ran analyses using the robust Huber–White sandwich estimator (using subjects as the clustering variable), which is robust to some types of misspecification of the variance-covariance matrix (Rogers, Reference Rogers1993). For a complete description of the models that were tested, see Supplemental Materials.
All analyses controlled for age (in years) and gender. For each outcome, analyses were corrected for multiple testing of the four ROIs following the Holm method (Holm, Reference Holm1979; Aickin and Gensler, Reference Aickin and Gensler1996).
Results
Participants
Of the 132 included participants, 114 provided complete data. After quality checks of behavioural, imaging and ESM data, 87 participants were analyzed (16 excluded for RL task performance, 8 for MRI-related artifacts; 3 for ESM-related inconsistencies, see online Supplementary Fig. S1). There were no significant differences in age, gender, total MADRS score at inclusion and IDS-SR total scoreFootnote 4 between analyzed and not analyzed participants (all ps>0.2). Table 1a summarizes sociodemographic and clinical characteristics of the final sample (n = 87; n/l-D = 37 and m/m-D = 50). The groups did not differ significantly in age, gender or education (all ps>0.10). As expected, groups differed significantly in their depression severity as measured by interview (MADRS, both at inclusion (p < 0.001) and scan (p < 0.001) sessions) and self-report (IDS-SR, at scan session, p < 0.001), as well as average levels of daily-life reward anticipation (p < 0.001) and activity pleasantness (p < 0.001), with the m/m-D group reporting more symptoms and lower levels of the ESM items.
Demographic and clinical descriptive information (a); Information regarding behavioural performance on the Reward Learning task (b); and computational modelling output (c)
s.d., standard deviation; s.d._b, between subjects standard deviation; s.d._w, within subjects standard deviation; n, number of participants; MADRS, Montgomery-Åsberg Depression Scale; IDS-SR, Inventory of Depressive Symptomatology – Self-Report; RL, Reward Learning; RT, reaction time, ms, miliseconds; m/m-D, group of participants included based on the report of moderate symptom severity; n/l-D, group of participants included based on the report of low depressive symptom severity, CI, confidence interval, LLH, log-likelihood.
*p < 0.001.
a AIC: Akaike Information Criterion, calculated as −2 × LLH + 2 × k, where k is the number of parameters (k = 2).
b BIC: Bayesian Information Criterion, calculated as −2 × LLH + log(t) × k, where t is the number of trials (depends on how many responses a subject gave on the task) and k is the number of parameters (k = 2).
c PseudoR: calculated as -(LLHmodel-LLHchance)/LLHchance. LLHchance = log(0.5) × t, where t is the number of trials.
RL task performance and model fitting
Participants with optimal choice accuracy on the gain trials ⩽55% were excluded from further analyses since it could not be ascertained whether they were engaged in the task (excluded: 6 n/l-D, 10 m/m-D). Depressive symptoms (interview and self-report) were not associated with average reaction time, the number of non-responses, average optimal choice accuracy or the total amount of money won on gain trials (all ps>0.2; Table 1b).
Additionally, depressive symptoms (interview and self-report) were not associated with model-fit-measures nor learning parameters (all ps>0.2; Table 1c). See online Supplementary Fig. 2 for choice probabilities predicted by the computational model. For the fMRI analyses, a fixed alpha was chosen and a learning model was fitted with a single set of parameters (Pessiglione et al., Reference Pessiglione, Seymour, Flandin, Dolan and Frith2006; Daw, Reference Daw, Delgado, Phelps and Robbins2011). Specifically, we used averaged estimates of alpha calculated across all subjects during reward (alpha: 0.2).
fMRI
Replicating previous results, across all subjects significant RPE clusters were found in the dorsal and ventral striatum extending to the amygdala, cingulate gyrus, frontal cortex, inferior parietal lobule (Garrison et al., Reference Garrison, Erdeniz and Done2013; Chase et al., Reference Chase, Kumar, Eickhoff and Dombrovski2015) as well as in the hippocampus and precuneus (see Fig. 2a and online Supplementary Table S1).
ROI analyses indicated that average activity in all four regions (bilateral NAcc and putamen) was significantly correlated with model-based RPE signals (see online Supplementary Table S3 and Fig. 2c) across all subjects.
Associations of RPE from each ROI with depression symptoms
None of the associations between RPE signals in each of the ROIs and MADRS total score were significant (all ps>0.20). However, the association between RPE and IDS-SR total score was significant for the right putamen (b = −0.13, p = 0.009) and right NAcc (b = −0.017, p = 0.010), indicating decreased RPE-related striatal activity in participants with higher self-reported depression scores (see Table 2 and Fig. 2d).
Associations between mean activation in R NAcc and Putamen and reward-related ESM measures (controlled for age and gender, and the ESM analyses also for overall time of measurement, time of measurement within the day)
All ESM predictors were person-mean-centered. Models with time-varying predictors include random slopes of those predictors (incl. quadratic variables). Models with lagged predictors include the lagged outcome as a predictor. L, left; R, right; NAcc, Nuccleus Accumbens; MADRS, Montgomery–Åsberg Depression Scale; IDS-SR, Inventory of Depressive Symptomatology – Self-Report; s.e., standard error; p, p value; s.e.ML, standard error estimated using standard log-likelihood procedure; s.e.rob, standard error estimated using the robust Huber–White sandwich estimation procedure; pML, p value of effect based on the log-likelihood standard error estimation; prob, p value of effect based on robust Huber–White sandwich estimation; qua, quadratic (referring to time-lagged quadratic effect of reward anticipation on activity pleasantness; lin, linear (referring to time-lagged linear effect of reward anticipation on activity pleasantness).
*Indicates a significant p value after correction for multiple testing following the Holm method (Holm, Reference Holm1979; Aickin and Gensler, Reference Aickin and Gensler1996), correction performed within each row of the table.
Associations of RPE from each ROI with the ESM reward measure
See Supplemental Materials for details on beep questionnaire response rates.
ROI RPEs did not significantly moderate the linear relation between reward anticipation and activity pleasantness (all ps>0.10). However, after visual inspection of the data, it seemed that a non-linear relation between reward anticipation and activity pleasantness might be a better fit. Hence, post-hoc analyses were performed examining the moderating role of ROI RPE on the quadratic association between reward anticipation and activity pleasantness, yielding significant results for the left NAcc (b = 0.031, p = 0.010), right NAcc (b = 0.026, p = 0.011) and right putamen (b = 0.047, p < 0.001). In other words, the association between reward anticipation (squared) at time t-1 and activity pleasantness at time t (on average 90 min later) depended on how well brain activity followed the model-derived RPE signal, with an increased association between these time-lagged ESM variables for higher brain RPE-related activity levels. These results remained significant when using a robust – but more rigorous – estimation of the standard errors of the effects (see Table 2 and Fig. 3).
Simple slopes to illustrate significant moderating effects of RPE beta weight of three ROI's on the time-lagged association between Reward anticipation and Activity pleasantness, at three levels of the RPE beta weight (group average, and 2 standard deviations below/above this group average). R, right; L, left; NAcc, Nucleus Accumbens; EMM, estimated marginal mean; WP, within-person; s.d., standard deviation; RPE, Reward Prediction Error; ROI, Region of Interest
Exploratory analyses examined whether ROI RPE mean activations were associated with the individual ESM items that comprised the ESM measure of interest (reward anticipation and activity pleasantness). No significant associations were found for any of the ROIs (all ps>0.05, see Table 2).
Discussion
The current study found that RPE signals in the right putamen and nucleus accumbens correlated with self-reported depression severity suggesting that individuals with higher depression severity scores exhibit lower striatal RL signals. Second, although RPE-related brain activity in these areas did not moderate the lagged linear association between reward anticipation and activity pleasantness, quadratic effects emerged. Specifically, RPE-related brain activity in the putamen (right) and NAcc (left and right) moderated the lagged relation between quadratic reward anticipation and activity pleasantness. This suggests that a lower striatal RPE signal during a laboratory-based RL task was associated with increased decoupling between real-life enjoyment of activities and previous anticipation of pleasure (possibly akin to a real-life prediction error). Collectively, these findings reveal an association between how the brain processes RPEs under abstract and controlled circumstances and how much people change their behaviour towards pleasant activities following the anticipation of reward during day-to-day life.
Striatal RPE and depressive symptomatology
The current paper used a dimensional approach, with which we found significantly reduced striatal activation during unexpected rewards with increasing self-reported depression severity, which complements previous case-control analyses (Kumar et al., Reference Kumar, Waiter, Ahearn, Milders, Reid and Steele2008; Gradin et al., Reference Gradin, Kumar, Waiter, Ahearn, Stickle, Milders, Reid, Hall and Steele2011; Robinson et al., Reference Robinson, Cools, Carlisi, Sahakian and Drevets2012). This is of interest since there is an on-going debate regarding the discreteness of MDD (Hankin et al., Reference Hankin, Fraley, Lahey and Waldman2005; Prisciandaro and Roberts, Reference Prisciandaro and Roberts2009; Ruscio et al., Reference Ruscio, Brown and Meron Ruscio2009). Our findings indicate that striatal RPE is blunted with increasing depression severity, and is not a discontinuous, abrupt process emerging with an MDD diagnosis.
A recent study suggested that earlier findings regarding altered RPE signals in MDD might have been due to medication use (Rothkirch et al., Reference Rothkirch, Tonn, Kohler and Sterzer2017). However, this cannot explain the dimensional association with depressive symptoms we found in our non-help-seeking and unmedicated sample. Since MDD is a very heterogeneous disorder (Sharpley and Bitsika, Reference Sharpley and Bitsika2013; Zimmerman et al., Reference Zimmerman, Ellison, Young, Chelminski and Dalrymple2015) and different symptoms are present during different stages of the disorder (Rakofsky et al., Reference Rakofsky, Schettler, Kinkead, Frank, Judd, Kupfer, Rush, Thase, Yonkers and Rapaport2013), it is possible that clinical heterogeneity is partially responsible for inconsistent findings. We, therefore, advocate for better sample characterization and report sample averages per symptom in the supplementary materials.
It is important to note that we found a significant association between striatal RPE signal and depressive symptoms, only when the latter was assessed with a self-report measure (IDS-SR) and not with an interview-administered scale (MADRS). This discrepancy likely stems from the fact that self-report and interview assessments in general measure slightly different things (Uher et al., Reference Uher, Farmer, Maier, Rietschel, Hauser, Marusic, Mors, Elkin, Williamson, Schmael, Henigsberg, Perez, Mendlewicz, Janzing, Zobel, Skibinska, Kozel, Stamp, Bajs, Placentino, Barreto, McGuffin and Aitchison2007; Fantino and Moore, Reference Fantino and Moore2009; Uher et al., Reference Uher, Perlis, Placentino, Dernovšek, Henigsberg, Mors, Maier, McGuffin and Farmer2012). However, the scales also differ in their contents. For example, whereas the MADRS includes only two items related to the Positive Valence System (PVS; ‘Inability to feel’ and ‘Lassitude’), the IDS-SR assesses similar concepts in four items (‘Response of mood to good/desired events’, ‘General interest’, ‘Energy level’, ‘Capacity for pleasure/enjoyment’). The latter might, therefore, be more sensitive to variations in specific sub-domains of PVS functioning. Future research should include measures developed to specifically measure different PVS domains [e.g., the Temporal Experience of Pleasure Scale (Gard et al., Reference Gard, Gard, Kring and John2006)].
Lastly, the reduced striatal activation during unexpected rewards with increasing self-reported depression severity was not accompanied by a decreased average learning performance. That is, average optimal choice accuracy was not associated with depression severity. This discrepancy in associations of neural and behavioural data of the same task could be due to several reasons. One option we explored is that individuals with more symptoms of depression (and reduced neural PE-related striatal activation) learn slower, but that the task was long enough for these individuals to ‘catch up’. Indeed, a post-hoc analysis examining whether the cumulative choice accuracy (i.e. per trial, the percentage of optimal choices of all trials up to that trial) differs for different depression levels resulted in a significant quadratic effect (p < 0.001) characterized by a less steep (quadratic) learning curve for increasing depression severity (online Supplementary Fig. 4). Again, this effect was only visible for IDS-SR and not for MADRS scores.
Striatal RPE and a measure of real-life reward-related behaviour
The current study fits within the relatively recent approach to combine neuroimaging techniques with real-world psychological experiences. However, it is the first to examine a computational model-based brain signal in this context. Previous work on reward-processing in depression showed that striatal activation during anticipation and receipt of rewards was associated with a daily life measure of PA in a group of depressed adolescents (Forbes et al., Reference Forbes, Hariri, Martin, Silk, Moyles, Fisher, Brown, Ryan, Birmaher, Axelson and Dahl2009). This finding was replicated in healthy controls (Forbes et al., Reference Forbes, Ryan, Phillips, Manuck, Worthman, Moyles, Tarr, Sciarrillo and Dahl2010). These reports added significant ecological validity to other imaging studies reporting reduced striatal activity during reward anticipation (Smoski et al., Reference Smoski, Felder, Bizzell, Green, Ernst, Lynch and Dichter2009; Gotlib et al., Reference Gotlib, Hamilton, Cooney, Singh, Henry and Joormann2010; Olino et al., Reference Olino, McMakin, Morgan, Silk, Birmaher, Axelson, Williamson, Dahl, Ryan and Forbes2014) and reward receipt (McCabe et al., Reference McCabe, Cowen and Harmer2009; Pizzagalli et al., Reference Pizzagalli, Holmes, Dillon, Goetz, Birk, Ryan, Dougherty, Iosifescu, Rauch and Fava2009; McCabe et al., Reference McCabe, Woffindale, Harmer and Cowen2012; Sharp et al., Reference Sharp, Kim, Herman, Pane, Reuter and Strathearn2014) in individuals, and offspring of individuals, currently or previously diagnosed with MDD.
Recent research suggests that, in the context of mood-related psychopathology, neural RPE processing might be a relevant measure in addition to these more static reward-processing measures (Rutledge et al., Reference Rutledge, Skandali, Dayan and Dolan2014; Eldar et al., Reference Eldar, Rutledge, Dolan and Niv2016). By investigating a dynamic ESM measure, i.e., how two variables are related over time, we were able to gain insight into which real-life processes are related to the striatal RPE signal. Evolutionary theories pose that goals for behaviour are specified by reward and punishment evaluation (Rolls, Reference Rolls1999), hence forming the basis for flexible (approach and avoidance) behaviours. The assumption is that flexible approach behaviour requires learning to recognize stimuli signalling potential rewards, followed by a cost-benefit analysis, possibly leading to a change in behaviour. Our ESM data show that, on average, people do report taking more pleasure in their activity when earlier they reported looking forward to a future situation.
However, in our sample, this was more so the case in individuals whose striatal activation more closely followed the model-derived RPE signal. Interestingly, the relevant interaction was only found with a quadratic association between daily life reward anticipation and activity pleasantness. This was characterized by lower striatal PE-related activity being associated with more decoupling between the two constructs only at higher levels of reward anticipation. This could indicate that these individuals might be less able to change their behaviour following the recognition of a cue highly predictive of reward, which would be in line with experimental findings by Sherdell et al. (Reference Sherdell, Waugh and Gotlib2012) indicating that liking and motivation are dissociated in individuals with MDD. Another option for this quadratic effect could be that these individuals are limited by reduced reward sensitivity and thus, even though they might have changed their behaviour following the highly predictive cues, they were not able to like the activity as much as individuals with less blunted striatal PE errors. Since mood affects reward sensitivity (Eldar and Niv, Reference Eldar and Niv2015), the role of mood in the brain-daily life associations reported in the current paper will have to be investigated more closely in the future.
Gaining more insight into real-life dynamics could ultimately lead to the refinement of psychotherapeutic interventions aiming at normalizing PVS dysfunction, [e.g., Craske et al. (Reference Craske, Meuret, Ritz, Treanor and Dour2016)]. Our results suggest that a focus on increasing the coupling of reward anticipation and engagement in pleasing activities might be a relevant target, for example with the use of experience sampling interventions. On an individualized basis, individuals could be made aware of what disrupts this coupling by reviewing, with the help of a professional, their own data after a week of ESM (similar to Kramer et al., Reference Kramer, Simons, Hartmann, Menne-Lothmann, Viechtbauer, Peeters, Schruers, Bemmel, Myin-Germeys, Delespaul, Os and Wichers2014). More advanced technology could additionally be used to provide real-life prompts to teach people to reduce this decoupling, for example by providing motivating messages following the report of reward anticipation.
Limitations
The results should be interpreted in the context of certain limitations. First, the current ESM design could not ascertain whether anticipated (at t-1) and reported (at t) pleasure referred to the same activities/situations [as was done in (Wu et al., Reference Wu, Mata, Furman, Whitmer, Gotlib and Thompson2016)]. This was a conscious choice as otherwise we would influence the participant's mood and/or behaviour (asking whether one actually performed what one was anticipating could make one aware of the failure, or change behaviour in order to avoid this failure). As our goal was to investigate whether, in general, an increase in activity pleasantness is preceded by an increase in reward anticipation this issue did not affect our aims.
Second, even though studies show that depression might be dimensional in nature (Liu, Reference Liu2016), it cannot be assumed that our ESM-fMRI association results generalize to clinical samples. Additionally, sub-threshold symptoms are known to be diffuse (van Os, Reference van Os2013), hence sub-threshold comorbidities are a concern.
Third, the current probabilistic RL task was adapted from a purely behavioural version. It, therefore, has several shortcomings, including the fact that the timings of the response and feedback overlapped, possibly confounding the RPE signal with motor-related activity. However, the pattern of whole-brain RPE results was comparable with previous meta-analyses (Garrison et al., Reference Garrison, Erdeniz and Done2013; Chase et al., Reference Chase, Kumar, Eickhoff and Dombrovski2015), which provides reassurance. In addition, after a win or a loss the total amount of winnings was updated (so that people could keep track of their progress), but only at the start of the next trial. This prevented us from analysing the neural correlates of model-based expected value – which might be relevant in MDD (Gradin et al., Reference Gradin, Kumar, Waiter, Ahearn, Stickle, Milders, Reid, Hall and Steele2011) – as these signals may be confounded by the effects of feedback.
Fourth, the cut-off for performing at chance level was set at 55%, which is an arbitrary limit. This cut-off was decided a priori to ensure the inclusion of compliant participants. Only three participants had accuracy scores between 50 and 55%, and their inclusion in the analyses did not affect the significance of the results. Additionally, the task was rather long (~24 min) and was part of a larger protocol including additional tasks. Hence, it is unclear whether participants performing below chance did so because of alterations in neural RPE mechanisms, fatigue or general reluctance to comply with instructions in order to finish more quickly. Owing to this design, we had to exclude a substantial number of participants due to potential task non-compliance. However, included and excluded participants did not differ in demographic or clinical variables. Future studies would benefit from including a post-scan questionnaire probing how motivated participants felt during the scan-session.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291718003446
Author ORCIDs
Jindra M. Bakker, Liesbet Goossens, Poornima Kumar, Iris M.J. Lange, Stijn Michielse, Jojanneke A. Bastiaansen, Jim van Os, Inez Myin-Germeys, Diego A. Pizzagalli, Marieke Wichers 0000-0002-5270-0581, 0000-0003-0368-2732, 0000-0002-9548-3281, 0000-0002-6251-8978, 0000-003-3930-8646, 0000-0002-7865-5694, 0000-0002-7245-1586, 0000-0002-3731-4930, 0000-0002-7772-1143,0000-0001-5024-9064.
Acknowledgements
This research was supported by grants from the Weijerhorst Foundation (J.v.O.); the Brain Foundation of the Netherlands (M.W., grant number 2012(1)-03); and a Boehringer Ingelheim Foundation travel grant (J.M.B., 2016). DAP was partially supported by R37 MH068376 from the National Institute of Mental Health (NIMH). PK was partially supported by a NARSAD Young Investigator award and by a R21MH105775 from NIMH.
Conflict of interest
Over the past 3 years, Dr Pizzagalli has received consulting fees from Akili Interactive Labs, BlackThorn Therapeutics, Boehringer Ingelheim, Pfizer and Posit Science, for activities unrelated to the current research. All other authors report no biomedical financial interests.
Ethical standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
