2.1 Examining insight in novice judges

According to some researchers, the reason for this poor insight is that judges learn how to make their judgments in an implicit manner (e.g., Evans et al. Reference Evans, Clibbens, Cattani, Harris and Dennis2003), and these processes are thus inaccessible to introspection. Testing such an account in established experts is of course difficult because the relevant learning has already been accomplished. Thus researchers have studied the acquisition of judgment policies in laboratory analogues of typical real-world judgment tasks.

An illustrative study is that of Evans et al. (Reference Evans, Clibbens, Cattani, Harris and Dennis2003) who asked participants to predict the suitability of fictional job candidates for an unspecified job on the basis of “ability tests.” The complexity of the task was manipulated by varying the ratio of relevant and irrelevant ability tests. Relevant tests contributed a constant value (+1 or −1) to the linear model that determined feedback; irrelevant tests contributed zero. Participants learned over a period of 80 to 100 trials with corrective feedback and were then given 40 test trials in which no feedback was provided. At the end of the test, participants rated each test on a scale from 1 (less relevant) to 7 (more relevant).

Evans et al. (Reference Evans, Clibbens, Cattani, Harris and Dennis2003) assessed “implicit” knowledge by measuring participants' revealed beta weights from test judgments and “explicit” knowledge by calculating a difference score between ratings given to relevant and irrelevant cues. In their second experiment, Evans et al. claimed to find a dissociation between these two measures of knowledge. Cue polarity (positive/negative) and absolute cue number (4 or 6) had large effects on the self-insight and performance scores (correlations between the criterion and prediction – labeled “achievement” in Fig. 1) but no effect on the explicit knowledge scores. Moreover, the difference between ratings for relevant and irrelevant predictors only differed from zero for one of three prediction tasks. This pattern of results led Evans et al. to conclude: “we have compelling evidence that performance … was largely mediated by implicit learning” (p. 615).

There are, however, reasons to question such a strong conclusion. Participants were faced with different job tasks in each experimental session, each one involving a different relevant/irrelevant cue ratio and different numbers of positive and negative predictors. Self-ratings of cue relevance were made at the end of each task, thereby failing the immediacy criterion for assessment (see Table 1). The sensitivity of the measures can also be questioned: There were 40 intervening test trials without feedback before ratings were made, and there were three different tasks per session, all with common labels for cues (A–F). Both of these factors could have increased the chance for cross-task confusion, making the low levels of explicit knowledge rather unsurprising.

In a recent study Rolison et al. (Reference Rolison, Evans, Walsh and Dennis2011) used similar methods to investigate the role of working memory capacity (WMC) in multicue judgment. They found that WMC correlated with performance when tasks involved negative predictors, but not when all relevant cues were positive predictors. Rolison et al. interpreted this pattern as evidence for reliance on deliberative processes in tasks with negative cues, and on implicit processes in tasks with exclusively positive cues. However, their data also showed the same associations and lack of associations between WMC and explicit knowledge of the underlying task structure. Thus a plausible alternative explanation is that performance was mediated by explicit knowledge in all tasks, but that the latter sometimes is and sometimes is not related to WMC.

Taken together, these illustrative experiments provide little evidence that unconscious processes influence multicue judgment. The dominant pattern across the experiments in both the Evans et al. (Reference Evans, Clibbens, Cattani, Harris and Dennis2003) and Rolison et al. (Reference Rolison, Evans, Walsh and Dennis2011) studies was of significant positive correlations between measures of performance and explicit knowledge of cue relevance/usage. In those instances where such correlations were absent, procedural artifacts (e.g., timing of awareness assessment) may have been responsible.

In recognition of the problems of retrospective interrogation of explicit knowledge, Lagnado et al. (Reference Lagnado, Newell, Kahan and Shanks2006) used an approach in which participants learning a multiple-cue judgment task were probed throughout training trials for the explicit basis of each prediction. On each trial participants were asked to rate how much they had relied on each cue in making their prediction. The “explicit” cue ratings were then compared with the “implicit” weights derived from running “rolling” regressions (a series of regressions from predictions to cues across a moving window of consecutive trials; cf. Kelley & Friedman Reference Kelley and Friedman2002).

The take-home message from the analysis of these data was that participants clearly distinguished between strong and weak predictors on both the implicit and explicit measures of cue reliance. This ability occurred fairly early in the task and was maintained or increased across training. Lagnado et al. (Reference Lagnado, Newell, Kahan and Shanks2006) also reported strong positive correlations between individuals' cue reliance ratings and implicit regression weights. The overall pattern strongly suggested that people had access to the internal states underlying their behavior and that this access drove both online predictions and explicit reliance ratings. Note that it is unlikely that the requirement to make online ratings altered participants' judgment strategies, as an additional experiment demonstrated that overall accuracy in the task was unaffected by the inclusion of the online ratings. In a recent study, Speekenbrink and Shanks (Reference Speekenbrink and Shanks2010) extended this approach by using a “dynamic lens model” to assess participants' insight in an environment in which cue validities changed across the course of an experiment. Consistent with Lagnado et al. (Reference Lagnado, Newell, Kahan and Shanks2006), Speekenbrink and Shanks found little evidence for the contribution of implicit processes: Participants learned to adapt to changes in the environment, and their reports of how they changed their reliance on cues reflected their actual reliance on those cues as evidenced by their predictions.

2.2 Assessing expert knowledge

Much of the work examining expert judgment has focused on the necessary antecedent conditions for the development of “intuitive” expertise (e.g., Hogarth Reference Hogarth2001; Kahneman & Klein Reference Kahneman and Klein2009; Shanteau Reference Shanteau1992) and the relative accuracy of expert and statistical judgment (Dawes et al. Reference Dawes, Faust and Meehl1989; Meehl Reference Meehl1954; Vrieze & Grove Reference Vrieze and Grove2009). Our focus here is somewhat different; we are interested in the rather smaller literature that has examined the extent and nature of experts' self-insight into the cues they use in real-world judgment tasks.

Slovic and Lichtenstein (Reference Slovic and Lichtenstein1971) were early to note that there were “serious discrepancies” (p. 49) between the explicit weights provided post hoc by judges and the implicit weights they placed on cues as evidenced by regression modeling. One source of this discrepancy was judges' tendency to overestimate the importance placed on minor cues and to underestimate their reliance on major cues. For example, Slovic et al. (Reference Slovic, Fleissner and Bauman1972) reported a correlation of only 0.34 between the implicit and explicit weights of 13 professional stockbrokers performing a stock selection task. The low correlation was attributed to the variance of explicit weights across the individuals: Each of the eight predictor variables was rated as most important by at least one judge, and some variables were rated subjectively more important than the regression analysis warranted.

The “serious discrepancies” identified by Slovic et al. (Reference Slovic, Fleissner and Bauman1972) and many others (e.g., Balzer et al. Reference Balzer, Rohrbaugh and Murphy1983; Phelps & Shanteau Reference Phelps and Shanteau1978) seem problematic for the view that we have access to the information influencing our behavior. These results would seem to suggest that there are indeed unconscious influences on the process of weighting and integrating cue information (see Fig. 1, Point D). However, the strength with which such conclusions can be drawn depends crucially on the methods used to elicit the importance ratings. It is quite possible that judges have good insight, but that experimenters have not provided them with sufficient opportunities to report the knowledge that they possess. It is also possible that judges confuse questions about the “importance” of cues for the task environment (i.e., ecological validities; see Fig. 1, Point B) with their “importance” for their own judgment process (i.e., cue utilizations; see Fig. 1, Point D) (cf. Lagnado et al. Reference Lagnado, Newell, Kahan and Shanks2006; Speekenbrink & Shanks Reference Speekenbrink and Shanks2010; Surber Reference Surber1985). As we shall see, there is considerable justification for these concerns.

2.3 Insight through policy recognition

In an influential brace of articles, Reilly and Doherty (Reference Reilly and Doherty1989; Reference Reilly and Doherty1992) examined an alternative way of assessing insight and drew significantly more optimistic conclusions about experts' knowledge of their judgment policies. Their novel procedure used a policy selection or “recognition” test that involved identifying one's own policy (described by normalized cue utilization indices) from an array of possible policies. In both articles, across a variety of hypothetical judgment tasks, this policy recognition method of assessing insight revealed much higher levels of self-insight into implicit and explicit policy profiles than indicated in previous research.

Harries et al. (Reference Harries, Evans and Dennis2000) extended the policy recognition approach by assessing self-insight in medical general practitioners. The doctors had taken part in a policy-capturing study 10 months prior to the insight assessment. They had been asked to make prescription decisions (e.g., whether to prescribe lipid-lowering drugs) for 130 hypothetical patients, each described by 13 cues (e.g., hypertension, cholesterol level, age), and to rate the importance of each cue for their judgments. In the follow-up, the doctors were presented with two arrays each containing 12 bar charts. The first array displayed implicit policy profiles (regression weights), and the second explicit profiles (importance ratings) both on standard bar charts. The 12 charts included the participant's own policy and 11 others randomly selected from the total pool of 32 participants. Their task was to rank the three policies in each set that they thought were closest to their own.

Consistent with Reilly and Doherty (Reference Reilly and Doherty1989; Reference Reilly and Doherty1992), the doctors were significantly above chance at picking both types of policies. The average hit rate (having one's own policy in the three selected) was 0.48 for implicit and 0.50 for explicit policy recognition. This level of performance is clearly far from perfect but it is considerably better than the 0.25 hit rate expected by chance. This replication is important because it not only demonstrates self-insight in genuine domain experts (instead of undergraduate students), but also rules out one possible explanation for Reilly and Doherty's findings. In their study some participants mentioned selecting explicit policies on the basis of explicit memory for the particular numbers of points they had distributed to individual cues (e.g., “I know I used 2.5 for one attribute”). Such memory for specifics, rather than insight into the actual policy, is less likely to have been a contributing factor in the Harries et al. (Reference Harries, Evans and Dennis2000) study, given that policy recognition was conducted 10 months after the judgment task and importance ratings were represented as bar charts. Note that although the test used in these studies does not meet the immediacy criterion for awareness assessment (see Table 1), the use of recognition rather than free recall makes it a more sensitive and arguably relevant test of insight.

The recognition measures used in the Reilly and Doherty studies revealed an “astonishing degree of insight” (Reilly & Doherty Reference Reilly and Doherty1989, p. 125), but the standard measures (e.g., correlations between implicit and explicit policy weights) showed the same poor to moderate levels as seen in many previous experiments. Furthermore, in both studies predictions on hold-out samples of judgments (i.e., cross-validation) demonstrated that models using implicit weights were superior to those using explicit weights in almost 100% of cases. Thus there appears to be “something else” captured in the implicit policies that participants are unable to communicate in their explicit policies.

However, the lower predictive accuracy of explicit weights and the tendency for people to state that they have relied on more cues than are apparent from their judgments (e.g., Slovic et al. Reference Slovic, Fleissner and Bauman1972) might also be partially artifactual. Harries et al. (Reference Harries, Evans and Dennis2000) pointed out that explicit weight estimates are based on a sample size of one – that is, they are made once, at the end of a series of (often) hundreds of judgments. As such they fail the immediacy, sensitivity, and reliability criteria for awareness outlined in Table 1. In contrast, the implicit weights are calculated from all trials and are thus more likely to capture patterns of cue use. Thus the low correlation between the two types may be due to the weakness of the cue importance measure.

The mismatch between stated and actual cue use could also be attributable to another aspect of typical experimental designs: the use of orthogonal cue sets (cf., Harries et al. Reference Harries, Evans and Dennis2000; Reilly & Doherty Reference Reilly and Doherty1992). Policy-capturing studies aim to discover reliance on particular cues; this is very difficult to do if a stimulus set contains highly intercorrelated cues, and so experimenters take pains to develop orthogonal cue sets. However, this can lead to problems if a judge uses cues inconsistently across cases.

Harries et al. (Reference Harries, Evans and Dennis2000) cited the example of a doctor using overweight or blood pressure interchangeably in making a decision about hypertension (because the two cues are highly correlated in reality). If the doctor was then presented with hypothetical cases in which these cues were orthogonal, he or she might still switch between them in his or her judgments but rate them both highly important at the end of the task. The regression analysis would then reveal equal but only moderate reliance on the cues, which would mismatch with the high importance ratings. In support of this possibility, Reilly and Doherty (Reference Reilly and Doherty1992) reported higher correlations between explicit and implicit weights in the representative conditions of their experiments (in which existing cue intercorrelations were maintained) than in their orthogonal conditions (in which they were reduced/eliminated; see Dhami et al. [Reference Dhami, Hertwig and Hoffrage2004] for further discussion of the important impact of representative designs and Beckstead [Reference Beckstead2007] for an illuminating treatment of the statistical methods for assessing policy recognition tests).

2.4 Summary and conclusions

The multiple-cue judgment literature presents a rich source of information about the potential role of unconscious influences. Although the received wisdom in studies of both novice and expert judges suggests poor insight into the factors underlying judgment, a close analysis of the data reveals a somewhat more optimistic picture. Our critique also highlights the importance of distinguishing genuine self-insight (or lack thereof) from artifacts that are inherent in the methods used to assess judgment.

One possible solution to this problem is to adopt a verbal policy-capturing method in which structured interviews are used to elicit explicit policies. Ikomi and Guion (Reference Ikomi and Guion2000) used such a technique with flight instructors and found that their “declared” policies were more accurate in predicting judgments than implicit weights for 12 of their 19 participants. An alternative approach is to reconsider the model underlying judgment. Policy-capturing studies are wedded to the idea that judgments involve the weighting and adding of individual cues (i.e., a linear additive model), but people might be using similarity to previously encountered instances (Brooks et al. Reference Brooks, Norman and Allen1991), or applying sequential heuristics (Gigerenzer Reference Gigerenzer2007) in making their judgments. These judgments might well be consciously mediated but would appear unconscious if participants were asked to explain what they were doing in terms of attribute weights, yielding inadvertent failure to meet the relevance criterion.

More research using various ways of assessing explicit knowledge is required before strong conclusions can be drawn, but at the very least we can say that many studies have revealed reliable access by participants into the thoughts underlying their judgments.

3.1 Reliability of the unconscious-thought effect

Demonstration of the benefit of unconscious thought on choice requires two criteria to be satisfied. First, choices following distraction need to be significantly better than those following deliberation, and, second, they need to be better than those following an immediate decision. In view of the amount that has been written about the merits of unconscious thought, it is surprising how rarely these criteria have been satisfied in one experiment. Both criteria are important. Demonstrating that distraction leads to better choices than deliberation could either mean that distraction is beneficial or that deliberation is detrimental. The latter conclusion is less surprising, especially if the conditions for deliberation are suboptimal (cf. Mamede et al. Reference Mamede, Schmidt, Rikers, Custers, Splinter and van Saase2010; Newell et al. Reference Newell, Wong, Cheung and Rakow2009; Payne et al. Reference Payne, Samper, Bettman and Luce2008; Shanks Reference Shanks2006; Wilson & Schooler Reference Wilson and Schooler1991). The second criterion is thus a crucial prerequisite for drawing any conclusions about the added benefit of unconscious thought.

In the first published work on unconscious thought, Dijksterhuis (Reference Dijksterhuis2004) reported three experiments that compared attitude ratings and/or choices following distraction, deliberation, and immediate processing. None of these experiments satisfied the two criteria outlined above. Moreover there were troubling (and unexplained) patterns in the data. For example, in Experiments 1 and 3 significant differences between some conditions were only found for males who constituted the clear minority in the sample. Thus even in this foundational study the evidence for unconscious influences was rather flimsy. It appears that when it comes to the role of unconscious processes, once an (intuitive) idea has taken hold, a momentum appears to build that is belied by the strength of the existing data. But despite this rocky start, it is now clear that there are several demonstrations of the effect – both in terms of improvements relative to conscious thought and immediate thought (see Strick et al. Reference Strick, Dijksterhuis, Bos, Sjoerdma, van Baaren and Nordgren2011, for a meta-analysis), although experiments in which all three conditions are tested and significant differences are found between each are still the exception rather than the rule (e.g., Dijksterhuis et al. [Reference Dijksterhuis, Bos, van der Leij and van Baaren2009] and Lerouge [Reference Lerouge2009] – but see González-Vallejo & Phillips [Reference González-Vallejo and Phillips2010] for a re-evaluation of the former).

These positive findings are, however, tempered by several studies that have compared all three thought conditions in a single experiment and failed to demonstrate any advantage of unconscious thought over conscious and/or immediate decisions (Acker Reference Acker2008; Calvillo & Penaloza Reference Calvillo and Penaloza2009; Huizenga et al. Reference Huizenga, Wetzels, Van Ravenzwaaij and Wagenmakers2012; Mamede et al. Reference Mamede, Schmidt, Rikers, Custers, Splinter and van Saase2010; Newell et al. Reference Newell, Wong, Cheung and Rakow2009; Payne et al. Reference Payne, Samper, Bettman and Luce2008; Rey et al. Reference Rey, Goldstein and Perruchet2009; Thorsteinson & Withrow Reference Thorsteinson and Withrow2009; Waroquier et al. Reference Waroquier, Marchiori, Klein and Cleeremans2010). The reliability of the effect is also questioned by an earlier meta-analysis of the unconscious-thought literature. Acker (Reference Acker2008) found that across 17 data sets there was “little evidence” (p. 292) for an advantage of unconscious thought. He also found that the largest unconscious thought effects were in the studies with the smallest sample sizes. Note that this is exactly the pattern predicted if one adopts exploratory rather than confirmatory research practices (Simmons et al. Reference Simmons, Nelson and Simonsohn2011; Wagenmakers et al. Reference Wagenmakers, Wetzels, Borsboom and van der Maas2011) and is also consistent with a publication bias operating (i.e., preferential publication of statistically significant effects – Renkewitz et al. Reference Renkewitz, Fuchs and Fiedler2011).Footnote ⁵ In line with these conclusions, Newell and Rakow (Reference Newell and Rakow2011) presented a Bayesian analysis of 16 unconscious-thought experiments from their laboratories (including both published and unpublished studies) and found overwhelming evidence in support of the null hypothesis of no difference between conscious and unconscious thought.

A charitable interpretation is that it is too early to draw strong conclusions about the robustness of the effect (cf. Hogarth Reference Hogarth2010). Vagaries of procedures, experimental instructions, differences in population samples, and differences in stimulus materials are all likely to contribute noise and hamper interpretation. But what about those cases where an effect is found? Do such results necessitate the involvement of an intelligent unconscious?

3.2 Explanations of the deliberation-without-attention effect

Proponents of the unconscious-thought theory (UTT) argue that deliberation without attention works because of the increased capacity and superior information-weighting ability of unconscious relative to conscious thought (Dijksterhuis & Nordgren Reference Dijksterhuis and Nordgren2006). However, substantiating these claims has proved somewhat problematic on both theoretical and empirical grounds (for a wide-ranging critique of the “capacity principle” of UTT, see, e.g., González-Vallejo et al. Reference González-Vallejo, Lassiter, Bellezza and Lindberg2008). With regard to superior weighting of information, the experimental evidence is equivocal at best. In the standard paradigm described above, participants' own subjective attribute weightings are ignored because the importance of attributes is predefined by the experimenter (e.g., Nordgren et al. Reference Nordgren, Bos and Dijksterhuis2011). Often this is done in an implausible manner. For example, in Dijksterhuis et al.'s (Reference Dijksterhuis, Bos, Nordgren and van Baaren2006b) study the number of cup holders in a car was deemed as important as the fuel economy (obviously cup holders are far more important): Both were given the same single-unit weight in the calculation of the best and worst cars. With these experimenter-defined weighting schemes, it is impossible to know whether the best choice is indeed the one favored by all participants.

Newell et al. (Reference Newell, Wong, Cheung and Rakow2009) examined this issue by asking participants, after choices had been made, for importance ratings for each attribute (e.g., How important are cup holders?). In so doing, Newell et al. were able to determine, retrospectively, the best option for each participant and then see how often participants chose the option predicted by their idiosyncratic weights. The results were clear: Regardless of the condition (conscious, unconscious, or immediate), the majority of participants chose the option predicted by their own idiosyncratic weights. In a similar vein, Dijksterhuis (Reference Dijksterhuis2004) reported that conscious and unconscious thinkers did not differ significantly in terms of the correlations between their idiosyncratic attribute weightings and attitudes toward options.

This last finding was echoed in a recent study by Bos et al. (Reference Bos, Dijksterhuis and van Baaren2011), who demonstrated that participants in both an immediate and an unconscious-thought condition were able to differentiate between cars that had a high number of “important” positive attributes (quality cars) from those that had several “unimportant” positive attributes (frequency cars) (a conscious thought condition was not included). While unconscious thinkers were significantly better at this differentiation (their difference scores were larger), there was no significant difference in the extent to which participants obeyed their own weighting schemes. Moreover, because a conscious thought comparison group was not run, we do not know if it was the operation of some active unconscious process that improved weighting or simply the additional time between presentation of the alternatives and the elicitation of the decision.

A study by Usher et al. (Reference Usher, Russo, Weyers, Brauner and Zakay2011) sheds further light on the weighting issue. They asked participants to rate the set of attributes from which the objects were composed before the decision task. A unique set of objects was then created, via computer software, to ensure that one object was the best for each individual participant, one the worst, and two others in-between. The standard decision task was then conducted with conscious- and unconscious-thought groups (no immediate group was included). Almost 70% of the distracted participants chose the best option, while fewer than 30% of those asked to think carefully did so. This is a compelling result suggesting more optimal weighting in unconscious than conscious thought, but without the immediate group for comparison, the Usher et al. results (on their own) do not satisfy our earlier criteria: The added value of unconscious processing, relative to an immediate judgment, cannot be assessed.

Several authors have asked whether the deliberation-without-attention effect is due to disadvantages conferred on conscious thought via particular experimental procedures rather than any hypothesized advantages of unconscious thought. For example, Payne et al. (Reference Payne, Samper, Bettman and Luce2008) examined whether conscious thinkers did poorly in the standard experimental task because they were forced to think about the problem for too long. Such persistence could, according to Payne et al., lead to a shift in attention toward less relevant information (cf., Rey et al. Reference Rey, Goldstein and Perruchet2009; Wilson & Schooler Reference Wilson and Schooler1991). To test this idea, Payne et al. compared participants in the standard conscious- and unconscious-thought conditions with a “self-paced” conscious thought condition in which participants were told they would have as much time as they liked to deliberate and decide.” The results were clear-cut: Participants in the unconscious and self-paced conditions outperformed those in the conscious condition but did not differ from each other. Payne et al. interpreted this combination of findings as evidence for poor performance of inappropriately constrained conscious thought rather than for superiority of unconscious thought.

A second re-interpretation of the unconscious-thought effect focuses on the possibility that participants make their decisions before entering the deliberation or distraction periods. The notion is that because attribute information is presented serially (and often randomly) about each option, participants engage in on-line processing, updating their impression of each option as subsequent pieces of information are presented (e.g., Lassiter et al. Reference Lassiter, Lindberg, González-Vallejo, Bellezza and Phillips2009; Newell et al. Reference Newell, Wong, Cheung and Rakow2009; cf. Hastie & Park Reference Hastie and Park1986). In the distraction condition, where post-information-acquisition processing is prevented (or discouraged), participants default to these on-line impressions when asked to make their final decision. In contrast, those given the opportunity to deliberate attempt to integrate the large amount of attribute information into a single “memory-based judgment” (Hastie & Park Reference Hastie and Park1986; Lassiter et al. Reference Lassiter, Lindberg, González-Vallejo, Bellezza and Phillips2009). The result is that the retrieved on-line judgments (or first impressions) are sometimes superior because conscious thinkers are hampered by fragmentary and poorly organized memory for the attributes (cf. Shanks Reference Shanks2006). Even authors who have challenged this interpretation (e.g., Strick et al. Reference Strick, Dijksterhuis and van Baaren2010) reported that 60% of their participants made decisions on-line. If this proportion is representative, then it provides a serious challenge to many previous studies that have argued that participants deliberate (either consciously or unconsciously) after information has been presented (for similar arguments, see also Newell & Rakow Reference Newell and Rakow2011).

Usher et al. (Reference Usher, Russo, Weyers, Brauner and Zakay2011) attempted to counter these problems by using a novel procedure in which multiple periods of distraction/deliberation were interpolated between the presentations of attribute information. They argued that this interpolation reduced the likelihood of participants deciding before being exposed to the thought manipulation. Under these conditions a small advantage for unconscious thought was still found. This result is particularly striking because the conditions for deliberative thinking were more suitable – there was less chance that attribute information could have been forgotten, and there were fewer pieces of information to think about at each “thinking interval.” Why filling these intervals with distraction (anagram solving) led to improvements in judgment remains a challenge to both the made-the-decision-before and the poor-conditions-for-deliberation alternative interpretations. However, even Usher et al. did not take this result as unequivocal evidence for active unconscious processes (p. 10).

3.3 Summary and conclusions

The notion that “sleeping on it,” in the sense of allowing a passage of time to elapse during which one is distracted, improves our decisions is enduring, appealing, and in line with anecdotal experience. Dijksterhuis and colleagues have struck a chord in the research community (and the public imagination) with an experimental paradigm that appears, to some extent, to provide empirical evidence for the soundness of the deliberation-without-attention recommendation. What is very clear, however, from our review is that the robustness and explanation of the deliberation-without-attention effect is far from settled (cf. Hogarth Reference Hogarth2010). Given this state of affairs, suggestions to rely on the unconscious in applied domains such as legal reasoning (Ham et al. Reference Ham, van den Bos and van Doorn2009) seem extremely premature.

One noteworthy feature of the vast majority of unconscious-thought research on decision making is that it has been done with students making inconsequential, hypothetical choices about situations that they may not have much experience with – for example, buying cars. Indeed, one of the few studies that examined the influence of distraction and deliberation in experts drew rather sobering conclusions for proponents of UTT. Mamede et al. (Reference Mamede, Schmidt, Rikers, Custers, Splinter and van Saase2010) showed that expert doctors given a structured diagnosis-elicitation-tool during the deliberation period produced more accurate diagnoses in complex cases than when they were distracted or made an immediate diagnosis. In fact, conscious deliberation gave rise to a 50% gain in diagnostic accuracy over an immediate diagnosis. This result illustrates that experts given appropriate conditions for deliberation can access relevant knowledge and improve their reasoning. Interestingly, in the same study novice doctors made poorer diagnoses in complex cases following deliberation compared to an immediate judgment (the accuracy of deliberative and distracted diagnoses did not differ) – suggesting that the period of structured deliberation is only useful if particular key facts are already part of one's knowledge base (Mamede et al. Reference Mamede, Schmidt, Rikers, Custers, Splinter and van Saase2010).

In summary, although the deliberation-without-attention effect has spurred welcome debate, ultimately, even if the effect can be reliably obtained, its existence falls well short of providing unequivocal evidence for the involvement of active unconscious processes in the construction of cue utilizations (Fig. 1, Point D).

4.1 The Iowa Gambling Task

Consider the choice between decks of cards where each card turned from each deck yields some reward or penalty, but nothing is known at the outset about the distribution of these outcomes. Someone playing this game has the opportunity to learn that the long-run payoffs of the decks differ and hence can adapt their sampling of the decks to reflect the payoffs. This essential structure describes the Iowa Gambling Task (IGT), devised by Bechara et al. (Reference Bechara, Damasio, Damasio and Anderson1994). In the years since it was first described and studied, a vast literature has grown up around this simple choice task (see Dunn et al. Reference Dunn, Dalgleish and Lawrence2006).

The conventional structure of the task employs four card decks and 100 card selections. Two of the decks yield positive payoffs of $100 for each card, and the remaining two decks yield payoffs of $50. However, some of the cards yield simultaneous losses. These are programmed to be more substantial on the decks that yield $100 payoffs such that in the long run these decks are disadvantageous and yield average net losses (equal to −$25), while the decks with $50 payoffs are advantageous and yield positive average net payoffs (equal to +$25). Within each pair of decks, one has larger but less frequent punishments, but the average payoff is equal. Thus in the long run the best strategy is to select cards from one or both of the advantageous decks and avoid the disadvantageous ones.

In addition to assessing choice behavior in this task, Bechara et al. (Reference Bechara, Damasio, Tranel and Damasio1997) probed participants' awareness of the task structure. After the first 20 trials and then after every additional 10 trials, participants were asked to describe what they knew and felt about the task. The majority of participants eventually reached a “conceptual” period in which they were able to describe with confidence which were the good and bad decks, and in this period they unsurprisingly selected from the good decks on the majority of trials. Prior to the conceptual period was a “hunch” period, described by Bechara et al. (Reference Bechara, Damasio, Tranel and Damasio1997) as involving a reported liking for the good over the bad decks, but with low confidence and reports of guessing. In the phase before this (the “prehunch” phase) participants “professed no notion of what was happening in the game” (Bechara et al. Reference Bechara, Damasio, Tranel and Damasio1997, p. 1294). Crucially, then, the question is whether awareness correlated with card selections or whether, in contrast, participants selected from the good decks in the prehunch phase before being aware of the differences between the decks in terms of their average payoff. It is this latter outcome that Bechara et al. (Reference Bechara, Damasio, Tranel and Damasio1997) claimed to observe in their data, concluding that “normals began to choose advantageously before they realized which strategy worked best” and that “in normal individuals, nonconscious biases guide behavior before conscious knowledge does” (p. 1293). Elsewhere, it has been claimed that “this biasing effect occurs even before the subject becomes aware of the goodness or badness of the choice s/he is about to make” (Bechara et al. Reference Bechara, Damasio and Damasio2000, p. 301).

Studies employing the IGT have a very natural interpretation within the lens model framework of Fig. 1. The decks can be conceived of as the cues, and their relationships to reward and punishment (the criterion) are captured by their ecological validities. The participant's goal is to judge the likely payoff for choosing each deck and to make a decision accordingly. If participants indeed learn to make advantageous deck selections, then their utilizations are appropriately tuned to the validities, yielding high achievement. Inability to report which are the good or bad decks is unawareness located at Point B in Fig. 1.

In view of the enormous amount written about the IGT and this pioneering study, it is remarkable to note that the key behavioral observation with regard to normal participants – more selections from good than bad decks in the prehunch period – was not in fact statistically significant in the Bechara et al. (Reference Bechara, Damasio, Tranel and Damasio1997) study. Preference for cards from the good decks was significant in the hunch and conceptual periods, but by that stage, of course, the participants possessed some conscious knowledge that could be guiding their choices. And the failure of this preference for the good decks in the prehunch period to reach significance is unlikely to be due simply to low power, because in two direct replications, with the same assessment of awareness, Maia and McClelland (Reference Maia and McClelland2004) and Wagar and Dixon (Reference Wagar and Dixon2006) did not even observe a numerical preference for the good decks in the prehunch period.

In addition to their replication of the original study, Maia and McClelland (Reference Maia and McClelland2004) tested another group of participants but employed a much more careful assessment of their awareness of the nature of the task at regular intervals. This careful assessment satisfied the criteria listed in Table 1. Rather than simply recording responses to open-ended questions regarding what they thought and felt about the task, Maia and McClelland required their participants to rate each deck on a numerical scale, to explain their numerical ratings, to report in detail what they thought the average net winnings or losses would be if 10 cards were selected from each deck, and to state which deck they would choose if they could only select from one deck for the remainder of the game. Answers to these questions provided a range of assessments of awareness against which actual card selections could be compared. In addition, Maia and McClelland ensured that the classification of decks as good or bad was based on the actual payoffs experienced by the individual participant to that point. Bechara et al. (Reference Bechara, Damasio, Tranel and Damasio1997) fixed the sequence of payoffs from each deck in the same way for each participant and scheduled very few penalties on the bad decks across the early trials. Thus a participant selecting early on from the bad decks might actually be making good choices, because the penalties that ultimately make such decks bad have not yet been experienced. Plainly, it is crucial to classify selections as good or bad in relation to what the participant has actually experienced, not in relation to the long-term but unknown average.

When card selections were compared with reported awareness under Maia and McClelland's (Reference Maia and McClelland2004) improved method, it was apparent that awareness if anything was more finely tuned to the payoffs than the overt selections were. Far from observing selections from the good decks in participants who could not report which were the good decks, Maia and McClelland found that conscious reports about the decks were more reliable than overt behavior. This might indicate that participants were still exploring the task and acquiring further information about the decks, but it clearly provides no support for the claim that nonconscious biases occur before individuals have relevant conscious knowledge. Maia and McClelland's results were replicated by Wagar and Dixon (Reference Wagar and Dixon2006), and similar outcomes were obtained by Evans et al. (Reference Evans, Bowman and Turnbull2005), Bowman et al. (Reference Bowman, Evans and Turnbull2005), and Cella et al. (Reference Cella, Dymond, Cooper and Turnbull2007), who in three separate experiments found that preferential awareness ratings for the good over the bad decks emerged before the point at which preferential card selections favored the good decks. By the time behavioral choice revealed a preference for the good decks, awareness was sharply discriminating.

Maia and McClelland's (Reference Maia and McClelland2004) study provides a particularly striking illustration of the dangers of employing an unreliable or insensitive test of awareness. In the Bechara et al. (Reference Bechara, Damasio, Tranel and Damasio1997) study, normal participants were reported to progress from the prehunch (no relevant awareness for discriminating the good and bad decks) to the hunch (some awareness that two of the decks were better than the others) phases at trial 50 on average, with no participant making this transition prior to trial 30. In their replication using the Bechara et al. (Reference Bechara, Damasio, Tranel and Damasio1997) awareness questions, but with a more careful algorithm for making the awareness classification, Maia and McClelland located the average transition at about the same point. Yet in their second group of participants, in whom awareness was measured via numerical judgments, participants were clearly aware of the difference between the good and bad decks by the first assessment at trial 20, and the onset of awareness has been located at a similar point in other studies (Evans et al. Reference Evans, Bowman and Turnbull2005; Wagar & Dixon Reference Wagar and Dixon2006). At this point, for example, 80% of Maia and McClelland's participants gave a good deck their highest numerical rating, and 85% of participants indicated one of the good decks when asked which deck they would choose if they could only select from one deck for the rest of the game. Thus on the Maia and McClelland assessment method, most participants had discriminative awareness by trial 20 (and possibly before then), whereas on the Bechara et al. (Reference Bechara, Damasio, Tranel and Damasio1997) method, none had such awareness prior to trial 30. The open-ended questions Bechara et al. (Reference Bechara, Damasio, Tranel and Damasio1997) used (“tell me all you know about what is going on in this game” and “tell me how you feel about this game”), together with their classification procedure for participants' responses to these questions, clearly did not make a sufficiently reliable and/or sensitive instrument for measuring awareness.

As noted earlier in this article, there has been much discussion about how best to measure awareness. Although they have attracted considerable controversy, alternatives to verbal report have been explored. Here we describe data from one study of decision making in the IGT that attempted to assess awareness without recourse to reports. Persaud et al. (Reference Persaud, McLeod and Cowey2007) required their participants not only to make a deck selection on each trial, but also to wager on the payoff for that trial. The wager could either be a high (£20) or a low (£10) amount. The reward from the advantageous decks was equal to the amount wagered, while that from the disadvantageous decks was twice the amount wagered, with occasional penalties being larger on the disadvantageous decks. The point of the task is that wagering is assumed to provide a probe of the participant's awareness. If the participant has some awareness that his or her decision is a good one, then he or she should be willing to bet higher on that choice in order to obtain a higher payoff. Thus Persaud et al. speculated that choices from the good decks should tend to be accompanied by high wagers and choices from the bad decks by low wagers, if the participant has some awareness of the difference between the decks.

In a group of participants tested under these circumstances, the good decks began to be reliably selected by around trial 40, but wagering did not begin to show a bias until trial 70. On the basis of this outcome, Persaud et al. (Reference Persaud, McLeod and Cowey2007) argued that the initial preference for the good decks must be based on unconscious information.

There are, however, some substantial difficulties with this set of conclusions. First, to locate the onset of awareness at around trial 70 in the IGT is to run radically counter to the data obtained in other IGT studies when the first set of test questions is administered at trial 20. Several studies (as noted above) have found that the vast majority of participants give higher numerical estimates for the good compared to the bad decks the first time they are questioned (Persaud et al. (Reference Persaud, McLeod and Cowey2007) did not report their own results from these awareness questions). Since the onset of a choice preference for the good decks is similar in the Persaud et al. study to that found elsewhere (around trial 40), it seems implausible to argue that the wagering component made the task harder overall and therefore delayed the onset of learning and awareness. Instead, it seems reasonable to speculate that wagering was measuring something other than awareness, or that it was measuring awareness insensitively or unreliably. This latter possibility is consistent with a second problem facing the wagering method of assessing awareness: Participants may have an aversion to risk or loss and hence may choose to make low wagers even when they have some degree of awareness. Evidence that this is not just a theoretical speculation but also an empirical reality has been reported by Dienes and Seth (Reference Dienes and Seth2010), and Konstantinidis and Shanks (Reference Konstantinidis and Shanks2013) have found that when loss aversion is avoided, wagering very closely matches deck selections.

4.2 Covert emotions in decisions under uncertainty

The review in this section thus far has considered Bechara et al.'s (Reference Bechara, Damasio, Tranel and Damasio1997) behavioral evidence concerning unconscious biases in decision making. However, that research is influential for a further reason: Physiological markers of emotion were measured at the same time as card selections. Specifically, Bechara et al. (Reference Bechara, Tranel, Damasio and Damasio1996; Reference Bechara, Damasio, Tranel and Damasio1997) measured their participants' skin conductance responses (SCRs) prior to each choice. In normal participants, these responses, commonly assumed to measure bodily states of arousal and emotion, were found to be substantial after both rewards and punishments. Most importantly, though, they began to emerge during the course of the task in anticipation of card choices, in particular becoming larger before selections from bad than from good decks. Bechara et al. (Reference Bechara, Tranel, Damasio and Damasio1996; Reference Bechara, Damasio, Tranel and Damasio1997) took these SCRs to be “somatic markers,” or covert emotional reactions capable of influencing behavior unconsciously, suggesting that “a negative somatic state as hallmarked by an anticipatory SCR, would nonconsciously “advise” the avoidance of the disadvantageous decks, while helping bring on line, cognitively the reasons for making the avoidance explicit” (Bechara et al. Reference Bechara, Tranel, Damasio and Damasio1996, p. 224).

Of course, the evidence described above that participants' awareness in the IGT is quite extensive raises considerable doubt over the inference that these somatic markers are in any sense covert. On the contrary, they may be the effect rather than the cause of conscious thought, and indeed there is evidence in favor of this viewpoint. Gutbrod et al. (Reference Gutbrod, Kroužel, Hofer, Müri, Perrig and Ptak2006) measured SCRs as well as card choices and found that anticipatory SCRs did not begin to discriminate between good and bad decks until about trial 80, yet card selections favored the good decks as early as trial 40. In fact, this sequence is evident in Bechara et al.'s (Reference Bechara, Damasio, Tranel and Damasio1997) data too: Whereas significantly more cards were selected from good than from bad decks in the hunch period, anticipatory SCRs measured during that period were not significantly different for good versus bad decks. As Gutbrod et al. noted, this early development of a behavioral preference for the good decks cannot have been driven by the somatic markers measured in anticipatory SCRs. It could, on the other hand, have been driven by differential awareness which, as discussed above, emerges very early in the task. This temporal sequence – awareness → differential choice → differential SCRs – seems to fit the data across these experiments well, with awareness being evident by around trial 20, advantageous card selections by trial 40, and differential anticipatory SCRs by around trial 80.

The only recent study to provide support for the possibility that anticipatory SCRs precede the development of card selections is that of Wagar and Dixon (Reference Wagar and Dixon2006). These authors obtained the typical finding of advantageous card selections emerging at around trial 40, but in their data differential SCRs were evident by around trial 30. Although these results suggest that more work is needed before we fully understand the relative timing of and causal relationship between anticipatory SCRs and card selections, even Wagar and Dixon themselves did not take any of their results as evidence of unconscious influences on decision making. Their participants showed awareness at least as early as they showed a preference for the good decks.

Moreover, there is a major concern surrounding the interpretation of somatic markers. On Bechara et al.'s (Reference Bechara, Damasio, Tranel and Damasio1997) interpretation, they provide anticipatory information about the value of a particular choice option, especially for negative outcomes. Specifically, they are assumed to encode information about the negative emotions that were previously triggered by a stimulus or choice outcome, and then covertly guide subsequent decisions. On this account, whatever the individual's report may state, his or her decision is actually driven at least in part by an emotional marker of the valence of the choice outcome, a marker that is related to previous (especially negative) experiences independently of subjective belief. In contrast to this account, recent findings suggest that SCRs code the uncertainty associated with the participant's decision, not the outcome (Davis et al. Reference Davis, Love and Maddox2009; Tomb et al. Reference Tomb, Hauser, Deldin and Caramazza2002). For example, Tomb et al. showed that when the IGT was modified so that it was the good rather than the bad decks that were associated with large payoffs and losses, SCRs tended to precede selections from the good decks. This strongly challenges the claim of the somatic marker hypothesis that such markers provide biasing signals for choice, because SCRs precede those choices (of bad decks) that are eventually eliminated in the standard IGT and precede those (of good decks) that eventually dominate in Tomb et al.'s modified version. Although it is possible that there are psychologically distinct somatic markers of positive and negative outcomes, it is plain that they cannot be distinguished by conventional SCR measurement.

4.3 Summary and conclusions

Of all the experimental methods used in recent years to study the role of awareness in decision making, the IGT and its variants have probably been studied more intensively than any others. The task lends itself quite naturally to a variety of awareness assessments and a range of behavioral indices, such as card choices and SCRs. While questions remain about important issues such as the suitability of using wagering as a means of gauging awareness, the evidence (particularly from Maia & McClelland's [Reference Maia and McClelland2004] major study) is clear in showing that participants acquire detailed conscious knowledge about the payoff structure at an early point during the task. This awareness emerges at least as early as behavioral differentiation itself, and there is little convincing evidence that decision making in the IGT is dissociable from awareness.

5.1 Subliminal perception

Subliminal perception is the controversial phenomenon whereby invisible stimuli may influence some aspect of behavior (see Fig. 1, Point C). It is intriguing that in the wake of a comprehensive methodological debate about 25 years ago (see Holender Reference Holender1986), subliminal processing was afforded a rather modest role in most theoretical debates about the causation of behavior. Yet in recent years there has been a wealth of claims, based on subliminal perception experiments, concerning the importance of the unconscious in behavior including some striking reports of subliminal priming on decision making (e.g., Winkielman et al. Reference Winkielman, Berridge and Wilbarger2005). Here we do not attempt to review this extensive literature. We do, however, briefly comment on the pervasive methodological problems that plague interpretation of results in this field (Dixon Reference Dixon1971; Holender Reference Holender1986; Miller Reference Miller2000; Pratte & Rouder Reference Pratte and Rouder2009), and we illustrate these problems with reference to a prominent and typical recent claim about subliminal influences on decision making.

In a striking illustration, Hassin et al. (Reference Hassin, Ferguson, Shidlovski and Gross2007) primed their participants with a brief (16-ms) masked presentation of either the Israeli flag or a scrambled version of the flag, prior to each of several questions about political attitudes (e.g., “Do you support the formation of a Palestinian state?”) and voting intentions. Not only did the subliminal primes influence responses to these questions, but they also affected subsequent voting decisions in the Israeli general elections. Key evidence that the primes were invisible came from a test in which participants were shown the masked images and asked directly to indicate for each whether it was a flag or scrambled flag, which revealed chance-level performance.

There are substantial problems with this kind of inference. For instance, the form of awareness check employed by Hassin et al. (Reference Hassin, Ferguson, Shidlovski and Gross2007) is susceptible to bias if the participant's confidence about seeing the flag is low. On some occasions on which they actually see the flag, they may nonetheless respond “scrambled flag” because their judgment is uncertain and they adopt a conservative decision criterion.

Worse still, Pratte and Rouder (Reference Pratte and Rouder2009) have shown that typical tests used to measure awareness in subliminal perception experiments (such as that used by Hassin et al. Reference Hassin, Ferguson, Shidlovski and Gross2007) may significantly underestimate conscious perception as a result of task difficulty. Because tests assessing perception of near-threshold stimuli are very difficult, participants may lose motivation. In their experiments, Pratte and Rouder maintained participants' motivation by intermixing above-threshold and near-threshold stimuli and found that identification of the near-threshold stimuli increased reliably. Thus brief stimulus presentations that would have been regarded as subliminal in a conventional awareness test were found to be supraliminal in a modified test designed to be more closely equated to the main priming test in terms of difficulty. Until subliminal priming experiments are able to rule out such artifacts, their conclusions will remain in doubt. Recent methodological advances (e.g., Rouder et al. Reference Rouder, Morey, Speckman and Pratte2007) offer the promise of more clear-cut tests of subliminal perception in the future.

5.2 Primes-to-behavior

Other striking studies, largely emerging from social cognition, describe apparent influence of primes on behavior where the prime, but not its influence, is consciously perceived (Fig. 1, Point D). A number of instances have been reported in recent years, such as that individuals can be induced to act socially or unsocially, walk faster or slower, behave more or less intelligently, or perceive accurately or inaccurately as a result of subtle priming influences of which they are unaware. In Bargh et al.'s (Reference Bargh, Chen and Burrows1996) famous experiment, for example, participants read sentences containing words related to the concept old age and, as a consequence, a few minutes later walked more slowly down a corridor. Although few of these studies relate specifically to decision making, they are provocative illustrations of possible unconscious influences on behavior.Footnote ⁶

Significant question marks exist concerning behavioral priming studies, particularly in regard to their assessment of awareness. The methods used for assessing awareness have generally been weak and fail the criteria described in Table 1. Bargh et al. (Reference Bargh, Chen and Burrows1996), for example, reported an experiment specifically designed to evaluate whether their participants were aware of the potential influence of the prime.

Bargh et al. (Reference Bargh, Chen and Burrows1996) reported that only 1 of 19 participants showed any awareness of a relationship between the stimulus words and the elderly stereotype.

This experiment leaves a number of questions unresolved. For example, was there any difference between the two groups in their responses to any of the questions? No actual data were reported at all, let alone broken down by group. Why were questions about whether the purpose of the experiment might have been different from what the experimenter had explained, and about whether the words had any relation to one another, included in the awareness test? These issues are irrelevant to the critical issue, namely, whether the participant was conscious of the activation of the age concept. The only relevant question is the final one, whether the participant had noticed any relation between the scrambled sentences and the concept of age. All the other questions are irrelevant, and their inclusion simply adds noise to the overall score. Put differently, the groups may have differed on their answers to this question, but that difference might well have been submerged in the random variance added by the other questions. Worse still, Doyen et al. (Reference Doyen, Klein, Pichon and Cleeremans2012) used the same walking speed task but with more careful awareness debriefing: Participants were required to choose among four pictures representing categories that could have been used as primes (athletic person, Arabic person, handicapped person, elderly). Doyen et al. found that primed participants had significantly greater awareness of the prime on this test than unprimed participants.

Unfortunately, weak methods are still being employed. In Ackerman et al.'s (Reference Ackerman, Nocera and Bargh2010) recent report that various social judgments can be nonconsciously influenced by haptic sensations, the only supporting evidence regarding awareness was that “Only one participant (in Experiment 5) reported awareness of the hypotheses, and so this person was removed from the analyses” (supplementary materials). How participants were probed about the influence of the primes on their behavior is not described, and whether or not they would have reported awareness if the criteria described in Table 1 had been satisfied (e.g., using sensitive methods such as rating scales) is unknown.

Another major problem is that the replicability of many of these priming effects has yet to be established. Dijksterhuis et al. (Reference Dijksterhuis, Spears, Postmes, Stapel, Koomen, van Knippenberg and Scheepers1998, study 2), Doyen et al. (Reference Doyen, Klein, Pichon and Cleeremans2012), and Pashler et al. (Reference Pashler, Harris and Coburn2011) all failed to replicate Bargh et al.'s (Reference Bargh, Chen and Burrows1996) finding that priming the stereotype of elderly people can affect walking speed. In another priming situation, Bhalla and Proffitt (Reference Bhalla and Proffitt1999) reported that participants judged a hill as steeper when they were wearing a heavy backpack, but results from Durgin et al. (Reference Durgin, Baird, Greenburg, Russell, Shaughnessy and Waymouth2009) found evidence that this priming effect is an artifact of compliance by participants to the perceived experimental hypothesis. In yet another example, Zhong and Liljenquist (Reference Zhong and Liljenquist2006) reported that asking participants to recall an unethical act from their past increased the accessibility of cleansing-related words and the likelihood of taking antiseptic wipes, yet the only published attempt to replicate these findings yielded four failures (Gámez et al. Reference Gámez, Díaz and Marrero2011). Until clear replications of these priming effects are reported, using more sophisticated assessments of awareness, it is premature to conclude that these studies provide robust evidence of unconscious influences on behavior.

5.3 Summary and conclusions

Few topics in psychology excite as much attention in the media as research on priming effects with subtle but unnoticed or outright subliminal stimuli. Yet research in this field yields remarkably few effects that convincingly demonstrate unconscious influences. The claim that briefly presented primes fall outside consciousness is difficult to establish without extremely rigorous methods, but such methods are employed insufficiently often. The primes-to-behavior literature has also largely employed weak methods to assess awareness, there are question marks over the replicability of some of its most prominent findings, and selective publication bias and file drawer effects (Renkewitz et al. Reference Renkewitz, Fuchs and Fiedler2011) may be clouding the overall picture.