Establish Links Between New Talent Signals and Job-Related Constructs

New talent signals from people's involvement in social media refer to the content posted on social media. Examples include comments posted on Facebook or Twitter, preferences for brands mentioned on Facebook pages, comments on other people's messages, likes, and membership in specific groups. As job relatedness is a cardinal rule to ensure validity and fairness, it is pivotal to map these new signals to job-related constructs (Society for Industrial and Organizational Psychology, 2003; Uniform Guidelines on Employee Selection Procedures, 1978). To this end, job analysis methods, subject matter judgments, and theoretical models can be used. Essentially, this endeavor is similar to earlier research that mapped interviewee answers, biodata responses, or assessment center observations onto job-related constructs. Afterward, the validity of the new signals can be ascertained by correlating them with more common measures of the same constructs or with criteria the new signals should predict.

So, we recommend that organizations determine beforehand which signals are indicators of well-known individual differences such as cognitive ability, knowledge, interests, personality, or motivation (Sackett, Lievens, Van Iddekinge, & Kuncel, in press). Along these lines, Roth, Bobko, Van Iddekinge, and Thatcher (Reference Roth, Bobko, Van Iddekinge and Thatcher2016) provided a good start in describing how social media content might be mapped to existing taxonomies of individual differences (KSAOs). Clearly, our recommendation to establish evidence-based or theory-based links between these new talent signals and job-related constructs applies not only to information gathered via social media but also to candidate actions in serious games (see Bedwell, Pavlas, Heyne, Lazzara, & Salas, Reference Bedwell, Pavlas, Heyne, Lazzara and Salas2012).

Adopt the Principle of Aggregation

Once job-related signals have been gathered, aggregation is a second cardinal principle to further reduce the noise of these job-related signals. We adopt the aggregation principle (Epstein, Reference Epstein1979) for reliability purposes almost everywhere in psychology. For instance, we use multiple items in scales, rely on multiple raters (e.g., interviewers), and combine data from several observations to form an overall evaluation.

In the context of scraping social media content, it is equally important to make sure that judgments about candidates’ standing on KSAOs are made on the basis of multiple signals. The consistency among these multiple, job-related social media signals can then be determined as a way of assessing their reliability (McFarland & Ployhart, Reference McFarland and Ployhart2015). For example, the consistency among the information captured can be examined across time or across different signals thought to reflect the same job-related construct.

A specific example of such aggregation consists of combining information from different information sources. For instance, some social media content comes from self-reported information, whereas other information on social media is provided by others (e.g., endorsements and comments). Given that self- and other reports each have their respective benefits and weaknesses (Vazire, Reference Vazire2010), and that other reports have been found to incrementally predict over self-reports in the personality domain (Connelly & Ones, Reference Connelly and Ones2010; Oh, Wang, & Mount, Reference Oh, Wang and Mount2011), scraping social media for both sources of information might be beneficial. For example, data from self and others could be correlated to assess the validity of new talent signals.

Standardize the Evoking and Extracting of Information

Using talent signals derived from social media is not without risks (Roth et al., Reference Roth, Bobko, Van Iddekinge and Thatcher2016). For example, social media platforms often lack standardization and might provide abundant information for some people and no information for others. Another major challenge is that social media such as Facebook routinely provide personal information that organizations are not allowed to use for decision making, such as race, religion, political affiliation, and sexual orientation. Once decision makers are exposed to such information (also things like physical appearance and disability status, which can be apparent from pictures, etc.), it may be difficult for them to ignore. These issues complicate the task for raters and/or machines to make sense of that information, such as when comparing job candidates.

To reduce information processing and decision making biases, we suggest taking a page out of the book of structured interviews (e.g., Campion, Palmer, & Campion, Reference Campion, Palmer and Campion1997), structured references (e.g., Taylor, Pajo, Cheung, & Stringfield, Reference Taylor, Pajo, Cheung and Stringfield2004), assessment centers (e.g., Woehr & Arthur, Reference Woehr and Arthur2003), and research on judgment and decision making in general. In these literatures, two main approaches have emerged to increase standardization. One approach consists of standardizing the questions and prompts given to candidates. Clearly, this is a challenge in the context of social media because these are typically designed to facilitate social interaction rather than to evoke job-related information (Roth et al., Reference Roth, Bobko, Van Iddekinge and Thatcher2016). This is especially the case with platforms such as Facebook, Twitter, and Google+. However, a professional and career-related social network such as LinkedIn is different because it has fixed rubrics that people are asked to complete. In addition, references can be given in LinkedIn.

Whereas standardizing the assessment of social media information could be challenging, this is less the case for other new talent signals from gamified assessment. In such gamified assessment, for instance, one might elicit relevant behavior because serious games provide a variety of fixed and dynamic stimuli to evoke job-related information (Bedwell et al., Reference Bedwell, Pavlas, Heyne, Lazzara and Salas2012). Although people will not follow the same path through the game due to its adaptive and interactive nature, the general idea is to build in more generic stimuli (Lievens, Schollaert, & Keen, Reference Lievens, Schollaert and Keen2015). For example, such generic stimuli might refer to dealing with ambiguity, conflict, scarce resources, or stress. Different rounds (levels) in a game might be built around one of these challenges and provide multiple data points of how candidates deal with them.

Another well-known standardization approach relates to providing training and other aids to raters who must extract and evaluate the information available. Again, there exists a rich literature in the interview and assessment center domains of training raters and equipping them with various aids. We urge industrial–organizational (I-O) psychologists to rely on this extensive knowledge base and devise training programs and rating tools to help raters (e.g., staffing professionals, managers) identify and evaluate relevant signals when screening social media information (Kluemper, Rosen, & Mossholder, Reference Kluemper, Rosen and Mossholder2012; McFarland & Ployhart, Reference McFarland and Ployhart2015). At the same time, there have also been rapid advancements in the automated scoring of texts, interviews, videos, and work samples (see below).

Use Mechanical Integration

Our suggestion to ensure that multiple signals are gathered from social media begs the question as to how these signals should be integrated to form a final judgment regarding candidates’ KSAOs. Generally, two broad approaches can be distinguished: (a) judgmental/clinical integration via raters and (b) mechanical/statistical integration via algorithms. In the selection and educational domains, there is meta-analytical evidence that mechanical integration outperforms judgmental integration (Kuncel, Klieger, Connelly, & Ones, Reference Kuncel, Klieger, Connelly and Ones2013). This suggests relying on machine-learning algorithms for mining social media data. Along these lines, Oswald and Putka (in press) reviewed a series of innovative tools that can be used for integrating information from big data. However, most of these tools require a large sample to train the algorithm to identify and categorize the information and then apply this algorithm to score future data (e.g., from actual job applicants).

Two Examples

Two recent studies illustrate the possibilities of using social media information to facilitate personnel decisions. Van Iddekinge, Lanivich, Roth, and Junco (in press) asked college recruiters to review the Facebook profiles of students who were on the job market and then rate the students on several general job-related constructs (e.g., adaptability, leadership, work ethic) and perceived hireability. The researchers followed up with students 6 to 12 months later to collect supervisor ratings of job performance and turnover information. The recruiter ratings did not predict performance, turnover intentions, or actual turnover. In addition, there was some evidence of subgroup differences in recruiter ratings that favored White and female ratees. Although the study design incorporated aggregation and mechanical integration, it did not incorporate the other recommendations we discussed. Importantly, Facebook information was not tied to job-specific constructs, and recruiters were not trained on how to search for and evaluate the information. Overall, Van Iddekinge et al.’s design may be closer to how recruiters and other decision makers (e.g., managers) currently scrape social media information (although many organizations likely use even less structured approaches to scrape social media information) rather than the potential for what organizations could do to maximize the potential of such information.

Park et al. (Reference Park, Schwartz, Eichstaedt, Kern, Kosinski, Stillwell and Seligman2015) also studied the use of Facebook information. Specifically, the researchers examined personality data from over 66,000 Facebook users obtained from a third-party application of Facebook (myPersonality). As such, their approach exemplifies our recommendation to adopt some standardization in eliciting stimuli. Moreover, they relied on existing theory and research that has linked language to the Big Five personality dimensions (see our first recommendation). One part of their sample was used to teach the machine to categorize linguistic features (words, phrases, and topics) into the Big Five dimensions. The stability of these language-based analyses of personality was examined by splitting the data into various time periods (test–retest reliabilities between .62 and .74). Park et al. used the other part of the sample to validate the language-based results against self- and other reports obtained from personality questionnaires (overall convergent validity correlation was .45). Results also revealed modest correlations between the language-based personality information (extracted from Facebook) and external criteria such as life satisfaction.