A Model of Response Behavior

The argument that technology can improve measurement is not new. The primary focus of the research has, as Morelli et al. (Reference Morelli, Potosky, Arthur and Tippins2017) note, been a reactive one in which “the primary concern with using a new assessment medium is that the technology introduces error” (p. 638) in measurement, which is subsequently investigated using tests of measurement invariance. The focal article authors argue that conceptual models of how technology interacts with the test taker, the test itself, the testing environment, or all three (i.e., the four questions on p. 639) can help preemptively answer questions regarding from where measurement error results. By integrating components of technology into existing models of response behavior, researchers and practitioners can leverage technological innovations to a priori minimize measurement errors. Furthermore, by linking technological design features to the various parts of the response process, I-O psychologists can actually help develop technological innovations with specific goals in mind rather than simply taking advantage of what others have developed.

Figure 1 presents a model of the cognitive processes that underlie responding to a survey item (Sudman, Bradburn, & Schwarz, Reference Sudman, Bradburn and Schwarz1996). Although Sudman et al. (Reference Sudman, Bradburn and Schwarz1996) presented this model with respect to survey items, it is readily applicable to any question format insofar as the steps outlined in Figure 1 are necessary for responding to any question posed.

Figure 1.

Sudman, Bradburn, and Schwarz's (1996) model of survey responding.

The first step in responding to a question is accurately interpreting the question. Measurement can be adversely impacted if the questions asked to respondents are not interpreted the same way because of wording differences or differences in the way the question was understood (Schwarz, Reference Schwarz1999). Technology can influence how test takers interpret questions. Research on training design in the cognitive and educational psychology literatures suggests that design efforts to highlight critical information and reduce extraneous information will enhance recognition and rehearsal of important points (Mayer, Reference Mayer2008). As Arthur's (2009) model suggests, technology has the potential to increase cognitive load, which could subsequently lead to overwhelming individuals’ attentional capacities (Randall, Oswald, & Beier, Reference Randall, Oswald and Beier2014), making it less likely that test takers will detect detail and comprehend questions adequately. Thus, to facilitate question interpretation, technology could direct attention to important instructions and item components without distracting individuals and muddling comprehension with irrelevant, distracting features.

After interpreting the question, respondents have to retrieve information from memory to provide an answer to the question. Measurement can be adversely impacted if respondents’ retrieval of information is limited or selective therein responding to the question without fully considering all aspects of what was asked, such as when supervisors make performance ratings based on whatever they can recall about an employee's performance at the time he/she fills out a performance evaluation (e.g., Borman, Reference Borman1978). With the assistance of technology at the first stage, mental effort can be conserved that will assist with information retrieval. Furthermore, technologies that utilize multiple media methods (e.g., pairing text with visuals), and those that provide job-relevant contextual cues and examples can facilitate the cognitive processes of rehearsal and retrieval (Mayer, Reference Mayer2008; Salas & Cannon-Bowers, Reference Salas and Cannon-Bowers2001). Therefore, technology that does not overburden attentional capacity and that uses rich, relevant media should facilitate information retrieval.

After retrieving the information, the respondent must engage in a judgment process wherein he/she evaluates the retrieved information to provide an initial answer to the posed question. Measurement issues can arise from forming a judgment from selectively recalled information. For example, the aforementioned supervisor's recollection of an employee's performance may be downwardly biased given that negative information can loom larger than positive information (Skowronski & Carlston, Reference Skowronski and Carlston1989). This negative bias can result in improperly weighting information when forming a judgment. Cognitive feedback can improve performance judgments by making decision makers aware of how they are utilizing the information at their disposal (see Balzer, Doherty, & O'Connor, Reference Balzer, Doherty and O'Connor1989). Measurement error stemming from the judgment process could be mitigated with technology that facilitates real-time cognitive feedback.

After forming this initial judgment, the response has to be formatted to fit the type of response requested. Measurement differences occur with different response formats (Schwarz, 1999). To continue the performance rating example, forced ranking scales require a different response formatting than absolute rating scales, which can impact greatly the overall measurement of performance (Goffin, Gellatly, Paunonen, Jackson, & Meyer, Reference Goffin, Gellatly, Paunonen, Jackson and Meyer1996). In personnel selection, different response formats (e.g., self-report, performance-based) also introduce different sources of error (e.g., faking vs. cheating, retest contamination; Lievens, Reeve, & Heggestad, Reference Lievens, Reeve and Heggestad2007; Villado, Randall, & Zimmer, 2017). Technological advances that allow for a more direct mapping of judgments to response options could mitigate errors arising from this stage of responding. The computer adaptive rating scales (CARS; Borman et al., Reference Borman, Buck, Hanson, Motowidlo, Stark and Drasgow2001) are one example of leveraging technology (i.e., computer adaptive testing) with scale development (i.e., ratee-targeted behavioral statements) to help respondents map judgments onto responses and minimize measurement error.

Finally, before submitting their response, respondents have the opportunity to edit their initial responses. These edits can be active attempts to present a more positive image (e.g., Dalal & Hakel, Reference Dalal and Hakel2016) and faking in personnel selection (e.g., Griffith, Lee, Peterson, & Zickar, Reference Griffith, Lee, Peterson and Zickar2011). Importantly, response editing can be expected for even non-self-report scales, such as when performance raters might allow political considerations to cloud their ratings (Levy & Williams, Reference Levy and Williams2004). The issues with respect to measurement quality in this last stage are evident: Actively misrepresenting one's standing on a construct limits the validity of the measurement. The use of technology may mitigate these intentional distortions. We might expect, for example, that the display of information, or transparency features, as highlighted in Potosky's (2008) framework, might influence response editing. High structural transparency, wherein it is easy for test takers to review multiple answers at one time, might lead to different editing patterns appearing more consistent or desirable than if the technology design was low in transparency, making it is more difficult for test takers to review many items at once. Thus, to the extent that technology enables test takers to review past answers or multiple answers at once, we might expect to see more response distortion and measurement error.

Moving Beyond Cognition and Behavior

Theory and research must incorporate both affective and motivational processes in addition to cognition and behavior in order to capture additional potential sources of measurement error due to technology. Ignoring how affect and motivation influence and are influenced by technology would leave models and theories of technology and I-O psychology incomplete. Focusing exclusively on cognitive and behavioral aspects of technology misses other crucial processes.

Our review of the Morelli et al. (Reference Morelli, Potosky, Arthur and Tippins2017) focal article, and admittedly our own treatment of technology design in stages of response behavior, tended to emphasize the cognitive and behavioral aspects underlying human behavior at the expense of the role of affective and motivational pathways. For example, Arthur, Doverspike, Kinney, and O'Connell's (2017) structural characteristics and information processing model focuses on cognitive traits and processes that predict cognitive load; likewise, Sudman et al.’s (1996) model in Figure 1 also focuses on the cognitive underpinnings of respondent behavior. However, we also know that affect and motivation are important considerations in the measurement process. For example, affective reactions and emotions experienced during the application process and test taking produce applicant reactions that can influence job choice, psychological contracts, and broader perceptions of organizational justice and commitment (e.g., Brief & Weiss, Reference Brief and Weiss2002; Hausknecht, Day, & Thomas, Reference Hausknecht, Day and Thomas2004; Morrison & Robinson, Reference Morrison and Robinson1997). Moreover, test-taker motivation can interact with cognition to predict applicant performance based on features such as reward valence, expectancy, and task difficulty (e.g., Kanfer & Ackerman, Reference Kanfer and Ackerman1989). Thus, motivation has the potential to influence test performance by introducing construct-irrelevant variance when it leads to increased effort, persistence, preparation, or even unethical behaviors like cheating or faking (Randall & Villado, Reference Randall and Villado2017).

Therefore, future research that incorporates technology directly into theoretical models should account for the effects of technology on behavior, not just through cognitive and behavioral pathways, but through affect and motivation as well. The third technology perspective that Morelli et al. (Reference Morelli, Potosky, Arthur and Tippins2017) highlight—sociomateriality—allows for the incorporation of motivation and affect as part of the purpose and typical use of technological tools utilized for selection, and may be a good area to investigate some of the interactive effects of these pathways to behavior on measurement error. Drawing from research in e-learning, affective reactions and engagement become increasingly important predictors of success in technology-enabled environments (Brown, Reference Brown2005; Sitzmann, Brown, Casper, Ely, & Zimmerman, Reference Sitzmann, Brown, Casper, Ely and Zimmerman2008), so these pathways may similarly be more deserving of emphasis in technology-based assessment. Test takers need to interact successfully with the technology platform in order to persist and complete the assessment, hopefully with minimal contamination from interfering emotions (e.g., frustration, anger) and motives (e.g., low tech self-efficacy, boredom). Indeed, similar arguments have been made with respect to candidate experience as part of the recruitment and selection process (e.g., Schmit & Ryan, Reference Schmit and Ryan1997). Thus, future research should more directly acknowledge how technology design and use can impact not only cognition and behaviors, but also emotional and motivational routes to behavior throughout the selection process. Similar to the considerations for the behavioral response model we addressed previously, acknowledging the impact of technological tools and design choices on affective reactions and motivational processes will inform technology use and implementation.