Alternate Formats: Video and Virtual SJTs

In both video-based and virtual SJTs, multimedia technology is used to present scenarios and response options in a filmed or virtually animated format (Lievens & Sackett, Reference Lievens and Sackett2006; McHenry & Schmitt, Reference McHenry, Schmitt, Rumsey, Walker and Harris1994; Olson-Buchanan & Drasgow, Reference Olson-Buchanan, Drasgow, Weekley and Ployhart2006). Recent meta-analytic results demonstrate that multimedia SJTs show stronger criterion-related validity results than written text SJTs for predicting interpersonal skills (Christian, Edwards, & Bradley, Reference Christian, Edwards and Bradley2010). Multimedia SJTs form a richer administration medium than written SJTs because they can present many more social cues, including verbal, nonverbal, and paralinguistic information.

According to media richness theory, the expression of multiple cues or the use of multiple channels in exchanging information benefits the delivery of a certain message (Daft & Lengel, Reference Daft, Lengel, Staw and Cummings1984; Potosky, Reference Potosky2008; Trevino, Lengel, & Daft, Reference Trevino, Lengel and Daft1987). In this way, it may be beneficial to present not only verbal cues that give away information about the content of an emotionally loaded situation but also nonverbal and paralinguistic cues that allow the perception of facial and vocal emotional expressions and subtle behavioral nuances that reflect domains such as leadership, integrity, and conscientiousness. The relevance of adding different information channels is also in line with the theory of cumulative contextual knowledge, which states that cumulating extra sources of information in social interaction contexts is important “because it can make a unique contribution to the perceiver's accuracy in ‘reading’ the target person's thoughts and feelings, that is, one that goes beyond the contribution made by the target's immediate verbal and non-verbal cues” (Gesn & Ickes, Reference Gesn and Ickes1999, p. 746).

Multimedia technology also serves to enhance the degree to which the test paradigms mirror the real environment (i.e., the extent to which the assessment task and context mirror those actually present on the job; Callinan & Robertson, Reference Callinan and Robertson2000). Because multimedia SJT items display a more detailed representation of the situation, a close resemblance with real leadership, teamwork, or other varying situations is possible.

Video-based SJTs have been used increasingly in business as an alternative to traditional text-based SJTs. Tuzinski, Drew, Lee, and Coughlin (Reference Tuzinski, Drew, Lee and Coughlin2012) highlight several benefits of video-based SJTs, such as improving candidate engagement, producing more favorable candidate reactions, and providing more realistic job previews than text-based SJTs. Another option involves developing virtual SJTs using generative photorealistic animated formats as a lower cost alternative to video-based SJTs. Generative formats can allow for new situations and items to be created at a much lower cost than refilming videos while keeping the advantages of the more realistically presented scenarios of video-based SJTs (although more research is needed to confirm that these advantages are retained). Tuzinski et al. (Reference Tuzinski, Drew, Lee and Coughlin2012) reiterate the benefits of animated formats over filmed video formats, including the ability to change aspects of both the setting and the actor (e.g., switching from office to home settings; modifying an actor's clothing, ethnic background or language) without refilming costs and with a lower overall investment.

Alternate Formats: Implications for Measuring Domain Knowledge

Given these varying format types and respective advantages, it seems appropriate to explore and consider how general domain knowledge assessment might differ across format type. Video SJT formats, for example, contain audio and visual cues that add depth and media richness to situation portrayals, but would this information be considered construct-irrelevant variance when conceptualizing SJTs as measures of general domain knowledge? Is general domain knowledge a construct that is distinct or independent of the construct of situation perception skill?

Lievens and Motowidlo aptly highlight Krumm and colleagues’ (Reference Krumm, Lievens, Hüffmeier, Lipnevich, Bendels and Hertel2015) recent study, which revealed situational stems account for only modest variability in SJT scores, to buttress their assertion that SJTs are largely domain-general measures that rely very little on their namesake, “situational judgment.” Krumm et al. (Reference Krumm, Lievens, Hüffmeier, Lipnevich, Bendels and Hertel2015) exclusively used written SJTs in their study, as Lievens and Motowidlo also noted, allowing for situational details such as the race, gender, and age of the people mentioned in the item, as well as the item setting, to possibly go unmentioned within the item. Details such as these would almost surely be unavoidable in a video-based SJT that would make use of actors to convey, if not the situational scenario composing the item stem, the behavioral response options. As several studies have now shown that video-based SJTs are higher in realism and fidelity (Funke & Schuler, Reference Funke and Schuler1998), yield higher predictive validity (e.g., Lievens & Sackett, Reference Lievens and Sackett2006), have higher face validity, and have decreased group differences (Chan & Schmitt, Reference Chan and Schmitt1997) compared with their text-based counterparts, it is important to consider how the inclusion of specific contextual information in video-based formats may impact the generality of SJTs.

For instance, video-based SJTs almost always present actors’ gender, age, and race to test takers. To minimize the effects of these specific pieces of information, which may introduce construct-irrelevant variance in SJT scores, and the extent to which these pieces of information interact with test takers’ own gender, age, and race, should these factors be systematically varied in order to produce the most “general” type of test? Conversely, should these factors be held constant across items? Or, are these factors merely sources of error variance that should be best avoided by employing less realistic methodologies such as avatars or cartoons that may be free of gender, race, and age characteristics?

Moreover, video-based SJTs employ human actors who may utilize both verbal and nonverbal cues that are not typically addressed explicitly within text-based SJT items. It is conceivable that this feature of video-based SJTs may allow the situational scenario composing the SJT item stem to have a greater impact on test takers’ scores than traditional, text-based multiple-response methods. For example, a text-based multiple-response SJT may describe a focal situation to a respondent as, “A customer is asking for a product that the customer saw on the store's website. However, it is currently out of stock. The customer is disappointed and upset” (item provided on www.jobtestprep.co.uk). Such an item explicitly informs the test taker that the customer is disappointed and upset rather than requiring the test taker to observe the customer and infer from verbal/nonverbal cues and behavior how the customer feels. Alternatively, a video-based SJT would merely portray the customer's reaction to the item being out of stock, requiring the respondent to use the customer's language, vocal cues, and body language to make an interpretation of the customer's feelings, which should be perceived as disappointment, for the item to be answered correctly. Paralleling similar processes underlying cascading models of emotional intelligence (cf. Joseph & Newman, Reference Joseph and Newman2010), individuals may first have to possess the ability to accurately perceive and understand what is happening in a given situation before they are able to correctly identify what the best course of action is in response to that situation. As text-based SJTs have been described as “low fidelity simulations” (Motowidlo, Dunnette, & Carter, Reference Motowidlo, Dunnette and Carter1990) of real-life interactions, the requirement that a test taker accurately perceive what is taking place in a video-based SJT may be what drives its higher predictive validity over text-based SJTs, as this added perceptual step may more closely mirror the processes underlying the behavioral content that pervades much of the criteria of interest in organizational psychology. The study by Rockstuhl, Ang, Ng, Lievens, and Van Dyne (Reference Rockstuhl, Ang, Ng, Lievens and Van Dyne2015), mentioned by Lievens and Motowidlo, suggests this may be the case, as this study revealed that asking participants to explicitly make judgments based on their construal of the situations accounted for incremental validity over judgments of response effectiveness. Lievens and colleagues’ (Lievens & Patterson, Reference Lievens and Patterson2011; Lievens & Sackett, Reference Lievens and Sackett2006) findings that scores derived from higher fidelity assessments (e.g., assessment centers, video SJTs) account for validity in job performance beyond SJT scores further support this point.

Perhaps the requirement that test takers accurately perceive and interpret a situation or behavior depicted in a video-based SJT is also a part of the general domain knowledge typically measured by SJTs, even if this facet of knowledge is not typically captured by most text-based multiple-response SJTs, as Krumm et al. (Reference Krumm, Lievens, Hüffmeier, Lipnevich, Bendels and Hertel2015) imply. If this ability to accurately perceive and understand human behavior and its implications for subsequent action on the part of the observer is indeed a precursor for general domain knowledge about behavioral effectiveness, this begs many questions about how this ability is best measured in a way that generalizes across different contexts. Should verbal and nonverbal cues be systematically varied across items, such that the more cues are portrayed, the better and more generalizable the measure? Some research has indicated the importance of perceiving negative information in order to interact effectively (e.g., McArthur & Baron, Reference McArthur and Baron1983). Does this mean that certain behavioral cues (for example, those that suggest problematic or negative traits and emotions [e.g., negative affect, embarrassment]) are more critical to portray than others (such as positive traits or inconsequential emotions [e.g., agreeableness, boredom])? Future research may more carefully examine how systematically varying SJT content affects SJT scores and their predictive validity. Such research could facilitate a better understanding of how the domain-general knowledge is best measured and shed light on the factors that contribute to how this knowledge may be acquired over time.

Situational Perception: Concluding Thoughts

It is clear that the use of multimedia SJTs presents an important issue for consideration when casting SJTs as measures of general domain knowledge. We contend that media-rich video or virtual SJTs present a case in which the situational content of SJTs matters, as these SJT formats may compensate for the construct underrepresentation (Messick, Reference Messick1995) of text-based SJTs by measuring test takers’ ability to accurately perceive situations. In comparison with text-based SJTs that frequently flatly describe the details of a given situation or scenario (e.g., “your boss angrily tosses the report into the trash can”), video and virtual SJTs require the respondent to synthesize verbal and nonverbal information and infer traits or emotional states that are expressed in the presented situation. In this way, video and virtual SJTs can be said to measure at least two constructs: situational perception and the ability to identify effective behavioral responses. The extent to which both of these constructs are incorporated as a measure of general domain knowledge is a question that deserves further exploration. Perhaps future studies can examine the degree to which latent classes emerge in response data reflecting differing perceptions of a situation and situational cues or accurate identification of effective behavioral responses to that situation. Scoring for “general domain knowledge” in this case would reflect not only accurate appraisal of the situation (possibly measured against subject matter expert ratings if rationally scored) but also accurate selection or rating of effective responses to the situation. The idea of SJTs as measures of general domain knowledge is an intriguing one for further research, and examining the way in which this idea extends across multimedia SJTs can help clarify not only implications across formats but also the construct of “general domain knowledge” itself.