Theory Meets Practice

At the start of the millennium, it was a selling point if a selection vendor sold assessments that could be administered online. These days it is an expectation. Although many I-O psychologists had concerns at the time, the market moved forward, and expectations for assessments now to be mobile compatible are increasingly urgent. Although many of the questions raised by the authors in the focal article are academically interesting, by the time new research or theory development hits press, it is often too late for the early applications of the new technology. Instead, we believe that peer-reviewed theory development often serves the important role of supporting decisions that have already been made in practice. Theory can also support revisions, additions, or modifications to our existing design and scoring approaches. It is our experience that the rapid evolution of technology in our space means the research studying new technology can rarely serve the role of leading advancements. In addition, many of the factors examined by the three conceptual frameworks are used to understand and explain issues that are, and always have been, beyond our control. We agree that new technology introduces potentially new sources of error, but as these are rarely within our control, theories speaking to them may have limited practical value.

One of the biggest modern concerns in our industry is increasing accessibility. Each year more and more people are taking our assessments on mobile devices. We also know that several protected classes are more likely to be using mobile devices as their primary or only source of connectivity when completing an assessment (Aon Hewitt, Reference Hewitt2016). There may come a day when not offering a mobile-optimized assessment will be considered a discriminatory act itself. We recognize that new technology brings new threats to the validity and equivalency of our tests, but we must balance the purity of our assessments with the accessibility needs of our applicants. We went down this road when the field was first confronted with the concept of unproctored Internet testing (UIT), a concept that is an expectation for high-volume pre-hire assessment today. It is valuable to consider the implications of technology on our field, but as practitioners looking to lead the path forward, we cannot await the development and dissemination of new theory before taking practical action toward adopting new modalities for test delivery.

To be clear, our intent is not to discredit the role of theory development in informing good practice. In fact, theoretical considerations drive many of the basic design elements of our assessments. But we believe the most pragmatic way to remain relevant and on the cutting edge of assessment development is to take a more holistic, multifaceted approach, which allows for faster product development and implementation. We believe that an evolving practical guide to best practices would be tremendously helpful. Such a guide could draw upon ideas from theory and published research but also be substantially influenced by other sources of information, such as work carried out in other disciplines with less theoretical orientation (e.g., basic user design principles), our own collective past practitioner experiences (including findings from internally conducted studies), our collective expert judgment, and our willingness to be open to piloting new approaches and technology. This could take the form of Society for Industrial and Organizational Psychology (SIOP)-sponsored articles by leading practitioners or new article submission formats to our journals. There would likely be value in brief case studies that cover a unique applied challenge and the thought process and steps taken to address it. Such a tool could even be used in a classroom environment where students could talk through the situation and get practice thinking scientifically about real-world problems.

Empirical Versus Experience Equivalence

Many of the studies reviewed by Morelli et al. (Reference Morelli, Potosky, Arthur and Tippins2017) are concerned with empirical equivalence. The authors rightly critique this approach as Sisyphean. In our view, creation of these somewhat arbitrary or convenient device-type classifications highlights and exacerbates an existing challenge that has been evident from the beginning of UIT. Although understanding the underlying factors deserves continued study, a practical approach is to also highlight the design considerations, instructional elements, and content validity of the assessment. This approach helps ensure that our tests provide a fair opportunity for candidates and minimize or mitigate as much of the variance not attributable to job relevant differences between candidates. In the real world, we are pushed to mobile delivery to meet the applicants where they are. But even before the increased adoption of smartphones, not all applicants were completing an identical assessment on equal footing. Two people taking an online assessment built for a PC could have two very different testing environments. One might be somewhere that makes concentration difficult or interrupts the assessment, whereas another candidate may minimize or eliminate all potential distractions before starting. We can tell an applicant to block off time to focus but, practically speaking, there is nothing we can do to guarantee that the testing environments are controlled and equal. Mobile testing has brought these fears back to our minds, prompting us to imagine scenarios such as an applicant taking our assessment on a crowded bus while the Internet connection goes in and out as the bus travels.

Although we agree with limitations raised by the authors regarding past research, we propose moving forward using a more pragmatic approach reliant upon guidelines and design principles from a number of sources, including psychological theory. Past work on empirical equivalence has argued that some constructs (e.g., personality) and methods (e.g., situational judgement) demonstrate empirical equivalence but others do not (e.g., cognitive ability; Arthur, Doverspike, Munoz, Taylor, & Carr, Reference Arthur, Doverspike, Munoz, Taylor and Carr2014). We argue that this may be an issue more strongly tied to content design and is not specific to the constructs assessed in a vacuum. Furthermore, many of these studies have been done on live applicant samples and often fail to recognize significant selection effects when it comes to device choice. For example, we see differences across applicants using mobile phones, tablets, and PCs on objective biodata items related to level of education, career stability, and beliefs regarding tardiness for work. It is unlikely that increased the permissibility of devices or reductions in screen size would impact stated level of education. These differences suggest that the only true way (in any practical sense) to even look at empirical equivalence is to randomly assign devices during a concurrent validation study.

Speaking of practical concerns, the issue of empirical device equivalence is not a question many of our clients are able or willing to answer. We have performed large-scale studies of device equivalence on concurrent validations with random assignment to devices, and the logistical hurdles are immense. You need a large incumbent sample. You also need the resources to bring the devices to the applicants and the internal influence to support such an initiative. The truth is that this is a very difficult sell to most clients for what they view as an academic issue.

Given these significant practical limitations and theoretical concerns for measuring empirical equivalence, our answer to this problem is to design assessments to the lowest common denominator and scale up—the exact opposite of what many assessment providers have done over the years (Morelli, Mahan, & Illingworth, Reference Morelli, Mahan and Illingworth2014). We agree with the authors that we should not aggregate into simple categories (e.g., PC, tablet) because these categories are more transient than they first appear. A large mobile phone with a modern processor may provide a better experience than a slower tablet with a lower resolution built several years ago. Also, some tablet users have a keyboard, whereas others may not. Although most people do not draw a distinction between a large monitor on a desktop and a small monitor on an ultrabook (both being categorized as a PC), many people would differentiate between a large phone and a small tablet. We prefer to think of devices along a continuum of capability, where we imagine the lowest common denominator for a particular solution in terms of screen size, processing power, resolution, and so forth, and design our assessments to that device (e.g., specifications consistent with a mobile phone released several years ago).

For example, rather than take a large cognitive measure and shrink it to fit on a phone or force an applicant to scroll to see the entire page, we believe measures should be designed to the limitations of the phone and then the assessment presented the same way on the most capable device. By optimizing to the lowest common denominator and scaling to other device types, you effectively ensure what we would call experience equivalence. Our tests used for selection or other high-stakes purposes should be device agnostic to the extent that is practically feasible. We design each assessment so that any observed differences in the data best capture and reflect true differences in standing on the construct of interest. In practice, this means shorter situational judgment items (to fit all text onto one screen), simpler cognitive exercises, and some constructs that simply cannot be measured at all (e.g., typing speed or typing accuracy) if your lowest common denominator lacks a keyboard.