Newell & Shanks (N&S) convincingly argue that past research has severely overstated the importance of conscious processes in decision making. We agree with N&S on many counts, but here we focus on what is perhaps our sole source of dissention. N&S conclude that healthy participants who perform the Iowa Gambling Task (IGT) learn to differentiate between the good and bad decks, and that this behavioral differentiation is even reflected in conscious knowledge about the payoff structure. We believe this conclusion may be premature: Several pitfalls in IGT data analysis methods frustrate a fair interpretation of IGT data, and several behavioral findings go against the authors' conclusion.
The first pitfall is that the traditional way of analyzing IGT data is incomplete and potentially misleading because it collapses choice behavior over the two good decks and over the two bad decks. This procedure hides the impact of the frequency of losses (bad deck B and good deck D yield rare losses, whereas bad deck A and good deck C yield frequent losses) and potentially obscures diagnostic information. For example, consider the data of Fridberg et al.'s (Reference Fridberg, Queller, Ahn, Kim, Bishara, Busemeyer, Porrino and Stout2010) healthy participants. Fridberg et al. plot the mean proportion of choices from the good and bad decks as a function of trial number, replotted here in the left panel of Figure 1. This panel suggests that participants learn to prefer the good decks. However, Fridberg et al. also plot the mean proportion of choices from each deck separately, replotted here in the right panel of Figure 1. This panel shows that, across all trials, participants prefer the decks with infrequent losses (B & D).
Figure 1.
Choice behavior of healthy participants in Fridberg et al. (2010), once for the good and bad decks (left panel) and once for each deck separately (right panel). Each block contains 20 trials, except the last block (15 trials)
A similar problem is evident in work that assesses conscious knowledge about the IGT either with subjective experience ratings (C+D) – (A+B) (Bowman et al. Reference Bowman, Evans and Turnbull2005; Cella et al. Reference Cella, Dymond, Cooper and Turnbull2007), or by determining whether participants have conscious knowledge that would encourage them to choose one of the two best decks (Maia & McClelland Reference Maia and McClelland2004). However, participants who consider “one of the best decks as the best deck” do not necessarily understand that there are two best decks and that both bad decks should be avoided. To investigate whether participants understand that there are two good decks, participants should identify the best and second-best deck on each trial.
The final pitfall concerns the way in which IGT studies typically assess the learning process, namely by applying an analysis of variance to assess whether participants' preference for the good decks (i.e., (C+D) – (A+B)) increases over blocks of trials (main effect of block). A significant effect of block is typically taken as evidence that participants learned to discriminate between the good and bad decks. However, when the main effect of block is significant, this does not imply it is also substantial. For example, consider the data of Bowman et al. (Reference Bowman, Evans and Turnbull2005), who tested three groups of healthy participants that differed in whether they obtained a manual or computerized IGT combined with or without a 6-second delay. The only significant effect was a main effect of block. However, even in the last block (i.e., the final 20 trials), the three groups showed at most a weak preference for the good decks, as (C+D) – (A+B) ranged from about 3 to about 6.5. A value of 3 corresponds to an average of 11.5 out of 20 choices from the good decks, and a value of 6.5 corresponds to an average of 13.25 out of 20 choices from the good decks. Similar unconvincing results were evident from subjective ratings of how positive each deck was experienced. These findings suggest that neither participants' behavioral preference for the good decks nor their conscious preference for the good decks is substantial. Cella et al. (Reference Cella, Dymond, Cooper and Turnbull2007) reported similar findings.
Next to the above mentioned pitfalls, several behavioral findings contradict the conclusion from N&S. First, a detailed re-analysis of eight data sets showed that healthy participants learn to prefer the good decks in only one data set (see Steingroever et al. Reference Steingroever, Wetzels, Horstmann, Neumann and Wagenmakers2013, and references therein). In the remaining seven data sets, participants either only learn to avoid bad deck A (frequent losses) or prefer the decks with infrequent losses (decks B & D). Such a preference for the decks with infrequent losses – the frequency-of-losses effect – has been reported by many studies. The empirical evidence for the frequency-of-losses effect contradicts the assumption that healthy participants learn to prefer the good decks.
Second, Steingroever et al. (Reference Steingroever, Wetzels, Horstmann, Neumann and Wagenmakers2013) showed that participants have a tendency to switch frequently throughout the entire task. This is counterintuitive because one expects a strong decrease in the mean number of switches once participants learned to prefer the good decks. The frequent switches suggest that participants do not learn to systematically differentiate between the good and bad decks, a suggestion that is illustrated by deck selection profiles of 394 participants (Steingroever et al. Reference Steingroever, Wetzels, Horstmann, Neumann and Wagenmakers2013; see https://dl.dropbox.com/u/12798592/DeckSelectionProfiles.zip for the deck selection profiles); each participant has a highly idiosyncratic choice pattern, and for most participants it is impossible to identify a point where they realized that the good decks should be preferred.
In sum, detailed analyses of IGT data have shown that even healthy participants are unable to discriminate the good decks from the bad decks, a finding that suggests a lack of both conscious and unconscious knowledge in this task.
Newell & Shanks (N&S) convincingly argue that past research has severely overstated the importance of conscious processes in decision making. We agree with N&S on many counts, but here we focus on what is perhaps our sole source of dissention. N&S conclude that healthy participants who perform the Iowa Gambling Task (IGT) learn to differentiate between the good and bad decks, and that this behavioral differentiation is even reflected in conscious knowledge about the payoff structure. We believe this conclusion may be premature: Several pitfalls in IGT data analysis methods frustrate a fair interpretation of IGT data, and several behavioral findings go against the authors' conclusion.
The first pitfall is that the traditional way of analyzing IGT data is incomplete and potentially misleading because it collapses choice behavior over the two good decks and over the two bad decks. This procedure hides the impact of the frequency of losses (bad deck B and good deck D yield rare losses, whereas bad deck A and good deck C yield frequent losses) and potentially obscures diagnostic information. For example, consider the data of Fridberg et al.'s (Reference Fridberg, Queller, Ahn, Kim, Bishara, Busemeyer, Porrino and Stout2010) healthy participants. Fridberg et al. plot the mean proportion of choices from the good and bad decks as a function of trial number, replotted here in the left panel of Figure 1. This panel suggests that participants learn to prefer the good decks. However, Fridberg et al. also plot the mean proportion of choices from each deck separately, replotted here in the right panel of Figure 1. This panel shows that, across all trials, participants prefer the decks with infrequent losses (B & D).
Choice behavior of healthy participants in Fridberg et al. (2010), once for the good and bad decks (left panel) and once for each deck separately (right panel). Each block contains 20 trials, except the last block (15 trials)
A similar problem is evident in work that assesses conscious knowledge about the IGT either with subjective experience ratings (C+D) – (A+B) (Bowman et al. Reference Bowman, Evans and Turnbull2005; Cella et al. Reference Cella, Dymond, Cooper and Turnbull2007), or by determining whether participants have conscious knowledge that would encourage them to choose one of the two best decks (Maia & McClelland Reference Maia and McClelland2004). However, participants who consider “one of the best decks as the best deck” do not necessarily understand that there are two best decks and that both bad decks should be avoided. To investigate whether participants understand that there are two good decks, participants should identify the best and second-best deck on each trial.
The final pitfall concerns the way in which IGT studies typically assess the learning process, namely by applying an analysis of variance to assess whether participants' preference for the good decks (i.e., (C+D) – (A+B)) increases over blocks of trials (main effect of block). A significant effect of block is typically taken as evidence that participants learned to discriminate between the good and bad decks. However, when the main effect of block is significant, this does not imply it is also substantial. For example, consider the data of Bowman et al. (Reference Bowman, Evans and Turnbull2005), who tested three groups of healthy participants that differed in whether they obtained a manual or computerized IGT combined with or without a 6-second delay. The only significant effect was a main effect of block. However, even in the last block (i.e., the final 20 trials), the three groups showed at most a weak preference for the good decks, as (C+D) – (A+B) ranged from about 3 to about 6.5. A value of 3 corresponds to an average of 11.5 out of 20 choices from the good decks, and a value of 6.5 corresponds to an average of 13.25 out of 20 choices from the good decks. Similar unconvincing results were evident from subjective ratings of how positive each deck was experienced. These findings suggest that neither participants' behavioral preference for the good decks nor their conscious preference for the good decks is substantial. Cella et al. (Reference Cella, Dymond, Cooper and Turnbull2007) reported similar findings.
Next to the above mentioned pitfalls, several behavioral findings contradict the conclusion from N&S. First, a detailed re-analysis of eight data sets showed that healthy participants learn to prefer the good decks in only one data set (see Steingroever et al. Reference Steingroever, Wetzels, Horstmann, Neumann and Wagenmakers2013, and references therein). In the remaining seven data sets, participants either only learn to avoid bad deck A (frequent losses) or prefer the decks with infrequent losses (decks B & D). Such a preference for the decks with infrequent losses – the frequency-of-losses effect – has been reported by many studies. The empirical evidence for the frequency-of-losses effect contradicts the assumption that healthy participants learn to prefer the good decks.
Second, Steingroever et al. (Reference Steingroever, Wetzels, Horstmann, Neumann and Wagenmakers2013) showed that participants have a tendency to switch frequently throughout the entire task. This is counterintuitive because one expects a strong decrease in the mean number of switches once participants learned to prefer the good decks. The frequent switches suggest that participants do not learn to systematically differentiate between the good and bad decks, a suggestion that is illustrated by deck selection profiles of 394 participants (Steingroever et al. Reference Steingroever, Wetzels, Horstmann, Neumann and Wagenmakers2013; see https://dl.dropbox.com/u/12798592/DeckSelectionProfiles.zip for the deck selection profiles); each participant has a highly idiosyncratic choice pattern, and for most participants it is impossible to identify a point where they realized that the good decks should be preferred.
In sum, detailed analyses of IGT data have shown that even healthy participants are unable to discriminate the good decks from the bad decks, a finding that suggests a lack of both conscious and unconscious knowledge in this task.