Slovic and Lichtenstein's (Reference Slovic and Lichtenstein1971) extensive review of research on policy capturing for multiple-cue judgment (see Newell & Shanks [N&S], sect. 2.2) was one of the first papers that I read on human judgment; so one of the first things I learned about human judgment was that people exhibit poor insight into their cue use, overestimating the number of cues used and misestimating the relative weights applied to different cues. N&S have usefully reminded us that this conclusion is dependent upon a particular choice of methods, and that the picture changes if alternative methods for investigating self-insight are employed (e.g., Reilly & Doherty Reference Reilly and Doherty1989; Reference Reilly and Doherty1992). When considering methods for assessing self-insight for multiple-cue judgment, it is worth noting that such investigations are an exercise in model recovery. In fact, two model recoveries are required: (1) statistical techniques (typically regression) are used in an attempt to recover the judge's policy for making the original case judgments (the implicit policy); and (2) the judge is also asked to recover his or her strategy for making the original judgments (the explicit policy) – often by assigning a points value to indicate each cue's weight. Conceptualising self-insight research as “double model recovery” emphasises some limitations in previous work that bear on N&S's relevance and sensitivity criteria for the adequate assessment of awareness (target article, Table 1), and points to avenues for developing better methods for assessing self-insight.
First, this double-model-recovery framework highlights a critical question: why – as seems standard – instantiate the implicit policy (from statistical model recovery) as the “correct” model, and therefore assume that any discrepancy between implicit and explicit policies represents the judge's failure to recover the “true” model? Model-recovery exercises in cognitive science usually consider multiple “families” of candidate model. Typically, lens model research considers a single family of models: compensatory linear rules that integrate a fixed number of cues – though it does consider different family members, which differ according to the number of cues used. Alternative families of non-linear (configural) or non-compensatory judgment models are less frequently considered in the multiple-cue judgment literature, even though several alternatives can be modelled, such as judgments made according to the similarity of each case to a prototype, judgments made following a non-exhaustive lexicographic search through cues, and judgments where cues are selected probabilistically and therefore different cues are used for different cases. In contrast, some research on multi-attribute choice does consider different families of models: for instance, comparing alternative models reflecting whether a compensatory or lexicographic decision rule is being applied (e.g., Bröder Reference Bröder2003). Additionally, this research on recovering choice processes highlights that different models reflecting quite distinct processes often fit the data similarly well. Therefore, even when a compensatory linear model fits the data, the judge may nonetheless have followed a quite different process in making his or her judgments. In such cases, any elicitation procedure that presupposes the compensatory linear combination of a fixed number of cues fails N&S's relevance criterion because the behaviours being probed are not those that drove the judgment. This is liable to generate a poor match between the implicit and explicit policies. Thus, by following a restricted approach to modelling the judge to dictate the constraints of that judge's self-description, we create an insensitive assessment of awareness and may misattribute poor modelling as poor self-insight.
Second, a double-model-recovery framework emphasises the potential for mis-recovery of the original judgment process by either recovery technique (statistical or human). As many others have done, I have pitted human judges against statistical rules in multiple-cue judgments and – as is typical – have found that “statistical judges” outperform their human competitors (Dawes et al. Reference Dawes, Faust and Meehl1989). However, in one investigation (see Rakow et al. Reference Rakow, Harvey and Finer2003), our statistical judge showed the same apparent lack of self-insight as its human counterparts. A seven-cue predictive model derived using logistic regression generated predicted probabilities (that an applicant would be offered a place at a given university) for a series of cases, each defined by multiple cues. Human judges also provided the same set of judgments. Using the same linear regression analysis applied to the human participants, the implicit policy for the statistical judge declared only five cues to be used reliably (i.e., significant). Thus the statistical judge showed the typical pattern of limited self-insight that human judges display, apparently overestimating the number of cues that it used! Thus, just as assessments of awareness may fail N&S's sensitivity criterion, so too, insufficiently sensitive model recovery via linear regression could contribute to an inappropriate conclusion of “limited self-insight” (for a technical discussion of this problem, see Beckstead Reference Beckstead2007).
Third, we can consider strategies for assisting human judges in recovering (describing) their judgment policies, which may, also, influence the candidate models for the statistical element of the double recovery exercise. In a recent study, we asked mental health practitioners to self-identify with descriptions of alternative families of judgment models – (non-)compensatory and (non-)exhaustive – which drew on analogies to common decision aids such as “balance sheets” and “trouble-shooting guides.” Many of our assessors selected those options that implied contingent information search or non-compensatory information integration in their own (triage) judgments. Thus, if required to describe themselves in terms of a compensatory model always using a fixed number of cues (as per most self-insight research – though arguably failing the relevance criterion), inevitably some participants were forced to misrepresent their policy. It would therefore be unsurprising if judges displayed “poor self-insight.” Much work has been done on alternative strategies for eliciting the subjective weights for compensatory linear judgment policies (e.g., Cook & Stewart Reference Cook and Stewart1975). However, we need improved (i.e., relevant and sensitive) elicitation methods that allow for a wider range of information search and integration processes to be identified when judges are asked to describe their judgment policies. Fair assessment of a judge's self-insight requires that both the statistical exercise of deriving the implicit judgment policy and the elicitation exercise whereby the judge describes his or her own judgments allow – as far as possible – recovery of the processes by which the original judgments were made.
Slovic and Lichtenstein's (Reference Slovic and Lichtenstein1971) extensive review of research on policy capturing for multiple-cue judgment (see Newell & Shanks [N&S], sect. 2.2) was one of the first papers that I read on human judgment; so one of the first things I learned about human judgment was that people exhibit poor insight into their cue use, overestimating the number of cues used and misestimating the relative weights applied to different cues. N&S have usefully reminded us that this conclusion is dependent upon a particular choice of methods, and that the picture changes if alternative methods for investigating self-insight are employed (e.g., Reilly & Doherty Reference Reilly and Doherty1989; Reference Reilly and Doherty1992). When considering methods for assessing self-insight for multiple-cue judgment, it is worth noting that such investigations are an exercise in model recovery. In fact, two model recoveries are required: (1) statistical techniques (typically regression) are used in an attempt to recover the judge's policy for making the original case judgments (the implicit policy); and (2) the judge is also asked to recover his or her strategy for making the original judgments (the explicit policy) – often by assigning a points value to indicate each cue's weight. Conceptualising self-insight research as “double model recovery” emphasises some limitations in previous work that bear on N&S's relevance and sensitivity criteria for the adequate assessment of awareness (target article, Table 1), and points to avenues for developing better methods for assessing self-insight.
First, this double-model-recovery framework highlights a critical question: why – as seems standard – instantiate the implicit policy (from statistical model recovery) as the “correct” model, and therefore assume that any discrepancy between implicit and explicit policies represents the judge's failure to recover the “true” model? Model-recovery exercises in cognitive science usually consider multiple “families” of candidate model. Typically, lens model research considers a single family of models: compensatory linear rules that integrate a fixed number of cues – though it does consider different family members, which differ according to the number of cues used. Alternative families of non-linear (configural) or non-compensatory judgment models are less frequently considered in the multiple-cue judgment literature, even though several alternatives can be modelled, such as judgments made according to the similarity of each case to a prototype, judgments made following a non-exhaustive lexicographic search through cues, and judgments where cues are selected probabilistically and therefore different cues are used for different cases. In contrast, some research on multi-attribute choice does consider different families of models: for instance, comparing alternative models reflecting whether a compensatory or lexicographic decision rule is being applied (e.g., Bröder Reference Bröder2003). Additionally, this research on recovering choice processes highlights that different models reflecting quite distinct processes often fit the data similarly well. Therefore, even when a compensatory linear model fits the data, the judge may nonetheless have followed a quite different process in making his or her judgments. In such cases, any elicitation procedure that presupposes the compensatory linear combination of a fixed number of cues fails N&S's relevance criterion because the behaviours being probed are not those that drove the judgment. This is liable to generate a poor match between the implicit and explicit policies. Thus, by following a restricted approach to modelling the judge to dictate the constraints of that judge's self-description, we create an insensitive assessment of awareness and may misattribute poor modelling as poor self-insight.
Second, a double-model-recovery framework emphasises the potential for mis-recovery of the original judgment process by either recovery technique (statistical or human). As many others have done, I have pitted human judges against statistical rules in multiple-cue judgments and – as is typical – have found that “statistical judges” outperform their human competitors (Dawes et al. Reference Dawes, Faust and Meehl1989). However, in one investigation (see Rakow et al. Reference Rakow, Harvey and Finer2003), our statistical judge showed the same apparent lack of self-insight as its human counterparts. A seven-cue predictive model derived using logistic regression generated predicted probabilities (that an applicant would be offered a place at a given university) for a series of cases, each defined by multiple cues. Human judges also provided the same set of judgments. Using the same linear regression analysis applied to the human participants, the implicit policy for the statistical judge declared only five cues to be used reliably (i.e., significant). Thus the statistical judge showed the typical pattern of limited self-insight that human judges display, apparently overestimating the number of cues that it used! Thus, just as assessments of awareness may fail N&S's sensitivity criterion, so too, insufficiently sensitive model recovery via linear regression could contribute to an inappropriate conclusion of “limited self-insight” (for a technical discussion of this problem, see Beckstead Reference Beckstead2007).
Third, we can consider strategies for assisting human judges in recovering (describing) their judgment policies, which may, also, influence the candidate models for the statistical element of the double recovery exercise. In a recent study, we asked mental health practitioners to self-identify with descriptions of alternative families of judgment models – (non-)compensatory and (non-)exhaustive – which drew on analogies to common decision aids such as “balance sheets” and “trouble-shooting guides.” Many of our assessors selected those options that implied contingent information search or non-compensatory information integration in their own (triage) judgments. Thus, if required to describe themselves in terms of a compensatory model always using a fixed number of cues (as per most self-insight research – though arguably failing the relevance criterion), inevitably some participants were forced to misrepresent their policy. It would therefore be unsurprising if judges displayed “poor self-insight.” Much work has been done on alternative strategies for eliciting the subjective weights for compensatory linear judgment policies (e.g., Cook & Stewart Reference Cook and Stewart1975). However, we need improved (i.e., relevant and sensitive) elicitation methods that allow for a wider range of information search and integration processes to be identified when judges are asked to describe their judgment policies. Fair assessment of a judge's self-insight requires that both the statistical exercise of deriving the implicit judgment policy and the elicitation exercise whereby the judge describes his or her own judgments allow – as far as possible – recovery of the processes by which the original judgments were made.