Why SJTs Capture Situational Judgment Even Though It Is Not Explicitly Measured

We agree that relevant knowledge is important when test takers in a usual SJT have to make judgments about the effectiveness of different response options—or about their potential behavioral intentions. However, we doubt that knowledge about the general effectiveness of certain behaviors or of expressing certain traits is sufficient to answer an SJT effectively because different individuals might interpret the situations that are described in the different items very differently, even when they have similar work experience. This is where test takers’ situational judgment—or more generally their interpretation of the situations—becomes relevant.

For test takers, the situations that are described in SJT items are often similar to the situations that interviewees face in questions in a selection interview or that participants face in exercises in an AC or in other kinds of work simulations. Especially, when these questions or exercises deal with interpersonal situations, then there is often more than an obvious one and only way to handle them—or said differently, many of these situations are not “strong situations” that are interpreted in the same uniform manner and that allow only a single, uniform behavioral response (Mischel, Reference Mischel1973), but they are often ambiguous so that different individuals might interpret them differently. Furthermore, based on their respective perceptions of a situation, different individuals might choose very different responses to deal with this situation in addition to different responses that are related to differences in individuals’ procedural knowledge.

In line with this, measures of situational judgment (or of situation assessment or of individuals’ ability to identify evaluative criteria, ATIC) have been found to be related to performance in selection interviews (e.g., Melchers et al., Reference Melchers, Klehe, Richter, Kleinmann, König and Lievens2009), ACs (e.g., Jansen et al., Reference Jansen, Melchers, Lievens, Kleinmann, Brändli, Fraefel and König2013), and other selection simulations (e.g., Oostrom, Melchers, Ingold, & Kleinmann, in press). In the present context, situational interviews are particularly relevant, because they are similar to SJTs in many regards: They describe hypothetical work-related situations and ask interviewees what they would do in these situations (Maurer, Sue-Chan, & Latham, Reference Maurer, Sue-Chan, Latham, Eder and Harris1999). Furthermore, similar to SJTs, interviews and ACs have been challenged by evidence concerning their construct-related validity. This research raised concerns about whether it is indeed possible to measure the targeted performance constructs that they were designed to measure (e.g., Lance, Reference Lance2008; Macan, Reference Macan2009) even though this has been a less severe issue in the interview domain because interviews are often not developed in a construct-oriented way. Thus, similar to the available evidence that situation assessment correlates with interview or AC performance (cf. Kleinmann et al., Reference Kleinmann, Ingold, Lievens, König, Melchers and Jansen2011), we would consider the findings by Rockstuhl et al. (Reference Rockstuhl, Ang, Ng, Lievens and Van Dyne2015) as support for the suggestion that SJT scores that are based on effectiveness judgments nevertheless capture systematic variance that is related to situational judgment.

Why Situational Judgment Is Relevant for the Criterion-Related Validity of SJTs

The ability to correctly interpret a social situation is relevant not only when applicants have to take an SJT, an interview, or an AC but also later on the job when they face ambiguous social situations. In line with this, Rockstuhl et al. (Reference Rockstuhl, Ang, Ng, Lievens and Van Dyne2015) found that situational judgment was a valid predictor of task performance. Similarly, previous research also found that situation assessment in interviews and ACs was a valid predictor of task performance and that the criterion-related validity of these selection procedures was higher when the variance related to situation assessment was not partialed out (e.g., Ingold, Kleinmann, König, Melchers, & Van Iddekinge, Reference Ingold, Kleinmann, König, Melchers and Van Iddekinge2015; Jansen et al., Reference Jansen, Melchers, Lievens, Kleinmann, Brändli, Fraefel and König2013; Oostrom et al., in press). Although this research used a different approach to measure situation assessment by focusing on candidates’ ability to correctly interpret performance criteria, it nevertheless stressed the importance of situation assessment even in samples that were relatively homogeneous in their work and educational background.

In line with the argument that situational judgment plays an important role, we echo Lievens and Motowidlo's suggestion that it might be valuable to collect situational judgments in addition to judgments concerning the effectiveness of different response options. However, we are skeptical about whether SJTs should always “be designed to determine whether people know that actions that express high levels of the relevant compound trait in a particular job are effective and that actions that express low and polar opposite levels of the compound trait are ineffective” (Lievens & Motowidlo, p. 16).

The reason for our skepticism is that this suggestion seems to be based on the assumption that actions related to a given trait are always effective. However, in many jobs that involve a broad range of social interactions—like, for example, in leadership positions—different actions that are related to different traits might be necessary to be successful across different situations: In some circumstances, it might be more important to be sympathetic or to show cooperative behavior, whereas other situations might require more assertive behavior. Thus, good situational judgment is necessary to correctly interpret these situations and to decide about the appropriate actions that need to be taken. In line with this, evidence from the AC domain suggests that good situational judgment is reflected in the ability to also discern targeted performance requirements across exercises that put very different demands on assessees (Speer, Christiansen, Melchers, König, & Kleinmann, Reference Speer, Christiansen, Melchers, König and Kleinmann2014).

Why Does Situational Judgment Matter Even Without a Description of the Situation?

Lievens and Motowidlo also state that the results of a series of studies by Krumm et al. (Reference Krumm, Lievens, Hüffmeier, Lipnevich, Bendels and Hertel2015) question the assumption that SJTs tap into situational judgment. In these studies, test takers had to choose the most effective response option on the basis of these response options alone because the usual item stems were deleted. Results from verbal protocol analyses were taken as support for the notion that reliance on general domain knowledge was related to better performance in the SJT.

At this point, we would still argue that test takers try to understand the situation to which the response options are related, with the only difference that an understanding of this situation is (at least somewhat) more difficult in comparison with when an item stem with a description of the situation is included. Furthermore, the evidence provided by Krumm et al. (Reference Krumm, Lievens, Hüffmeier, Lipnevich, Bendels and Hertel2015) does not rule out this suggestion. Even though Krumm et al. (Reference Krumm, Lievens, Hüffmeier, Lipnevich, Bendels and Hertel2015) write that “use of general knowledge” was related to a higher likelihood of choosing the correct response option (p. 410), many of the examples in their article suggest to us that this knowledge was related to the correct interpretation of the situation in the first step. Thus, test takers, for example, first had to discern that the situation required effective team performance before they could make a judgment about the most effective behavior to ensure effective team performance in the second step.

How Shall Situational Judgment Be Taken Into Account in the Development of SJTs?

One way, concerning the development of more construct valid SJTs, is to target only a single construct in an SJT. In line with this, the overview by Christian, Edwards, and Bradley (Reference Christian, Edwards and Bradley2010) showed that many SJTs focus on only one construct—or at least on only a single construct category or domain. A potential drawback with this approach is that the targeted kind of effective behavior might become too obvious across a set of SJT items that are always related to the same trait and requiring the same kind of action. In such a case, an SJT might become too transparent for test takers, and it might possibly be that this might ultimately lead to an impairment of the SJT's criterion-related validity. Thus, if the degree to which SJTs require situational judgment becomes too limited, systematic variance that is related to performance in the SJT as well as on the job might be lost. In line with this, evidence from ACs has shown that criterion-related validity diminished once the targeted AC dimensions were made very transparent to participants (Ingold, Kleinmann, König, & Melchers, in press).

An approach that would alleviate the risk to make SJTs too transparent is to capture more than a single construct. Similarly, as suggested by Lievens and Motowidlo, if one targets compound traits, which consist of relatively different facets that are relevant for effective performance, then the risk of a harmful increase in transparency might be prevented.

Another option that might be used—at least when an SJT is not too transparent—would be to make use of situational judgments in addition to judgments of response effectiveness. On the basis of the SJT research by Rockstuhl et al. (Reference Rockstuhl, Ang, Ng, Lievens and Van Dyne2015) as well as on results from the interview and the AC domains (e.g., Ingold et al., Reference Ingold, Kleinmann, König, Melchers and Van Iddekinge2015; Jansen et al., Reference Jansen, Melchers, Lievens, Kleinmann, Brändli, Fraefel and König2013), one could try to additionally collect situational judgments and to use them for making selection decisions. However, in this case, more research would be necessary to determine the acceptability among recruiters as well as applicants of the different ways in which such judgments might be collected.