Creating Construct Valid Measures

The first steps toward measuring a construct in a valid manner are to understand the construct of interest and to design a measure that is thought to best represent that construct. Traditionally, when researchers are designing a measure or operationalization for a construct, great care is taken to maximize overlap between the measure and the construct while simultaneously minimizing both construct contamination (i.e., irrelevant material included) and construct deficiency (i.e., relevant material excluded; McDonald, Reference McDonald1999). This process is methodologically rigorous and involves a variety of activities such as consulting subject matter experts, pilot testing, and running a plethora of statistical analyses (Shadish, Cook, & Campbell, Reference Shadish, Cook and Campbell2001). Alternatively, the volume, variety, and velocity of big data frequently accompany a lack of control over how variables are measured and what variables are included in a dataset. As a result, researchers utilizing big data are forced to establish construct validity using measures they likely did not design and that were never intended to represent the construct of interest.

As alluded to by Guzzo et al., big data in the organizational sciences likely originates from three primary sources with differing degrees of researcher control. At one extreme are naturally occurring behavioral trace data that are readily available from the Internet (e.g., Hernandez, Newman, & Jeon, Reference Hernandez, Newman, Jeon, Tonidandel, King and Cortina2015). Facebook, Twitter, Wikipedia, and others are treasure troves of data related to individuals interacting with one another over time. However, to exploit their potential, researchers must first determine how to convert and combine “likes,” friend requests, retweets, edits, and other such Internet behavior into meaningful organizational constructs. The second source of big data is organizations that already track numerous employee behaviors and ratings (e.g., Caputo & Boyce, Reference Caputo and Boyce2015). E-mails, chat logs, performance reviews, and other data are routinely collected by organizations but rarely utilized to better understand organizational processes. To do so, researchers must first properly synthesize text and code behavioral patterns into commonly studied constructs such as leadership, teamwork, and organizational citizenship behaviors. Finally, the third source of big data represents the other extreme whereby organizational researchers utilize physiological sensors or computer-based laboratory simulations to obtain large quantities of data on a predetermined set of individuals (e.g., Kennedy & McComb, Reference Kennedy and McComb2014). As a result, researchers control exactly what is measured and at what frequency. Unfortunately, these data likely still require researchers to convert button clicks, mouse movements, location, vocal amplitude, heart rate, and other raw behaviors into something substantively interesting and organizationally relevant. Therefore, overall, each of these big data sources presents differing challenges for understanding and interpreting what was measured and for combining and converting them into valid operationalizations of organizational constructs.

Statistically Establishing Construct Validity

Once a measure of a construct has been created, data are collected with that measure and statistical analyses are used in an attempt to establish construct validity. There are two primary means by which organizational researchers traditionally statistically demonstrate construct validity.Footnote ¹ Currently, the dominant approach is to assess the internal structure of a measure using confirmatory factor analysis (CFA). In this approach researchers test an a priori factor structure to determine the overall model fit as well test the hypothesized factor structure against plausible alternatives (Widaman, Reference Widaman1985). If the measure meets the common standards for goodness of fit established by Hu and Bentler (Reference Hu and Bentler1999) as well as fits the data better than the plausible alternatives according to chi-squared difference tests or likelihood ratio tests, then the measure is thought to be construct valid.Footnote ² Alternatively, historically researchers would utilize a multitrait–multimethod matrix (MTMM) to establish construct validity through a pattern of convergent and discriminant validity evidence (Campbell & Fiske, Reference Campbell and Fiske1959). Researchers gather data from multiple methods on the same construct as well as on distinct constructs and compare the relative magnitude of the resulting correlations. Typically, to accompany a new scale or measurement, the researcher will choose well-established measures with which to compare. This is of particular relevance to big data because researchers can accompany the behavioral trace data that commonly compose “big” datasets with more traditional self-report measures as suggested by Guzzo et al. Construct validity is established when measures of the same construct measured with different methods (e.g., self-report and behavioral trace) correlate more strongly with one another (i.e., convergent validity) compared with measures of different constructs measured with the same method (e.g., all self-report) and different (e.g., self-report and behavioral trace) methods (i.e., discriminant validity).

Unfortunately, the properties (i.e., volume, variety, velocity) of big data make the use of either of these statistical approaches for establishing construct validity significantly more challenging. CFA typically requires at least three indicators for each construct (Anderson & Rubin, Reference Anderson, Rubin and Newman1956). This minimum is easily achieved when developing a questionnaire with multiple items; however, in big data where behavioral traces are a more common variable type, having at least three behaviors to represent each construct is more difficult to achieve. If less than three behavioral indicators exist for a construct, the CFA model is not identified and cannot be estimated. Even when sufficient indicators exist, behavior is likely to occur at different frequencies, resulting in a large quantity of missing data for some variables that change or are recorded at slower rates. It is well known that missing or incomplete data may result in inaccurate correlational estimates (Roth, Reference Roth1994), making factor loadings unstable. One proposed advantage of CFA over means of establishing construct validity is the ability to utilize goodness-of-fit statistics (indices) to quantify the degree to which construct validity has been achieved. Unfortunately, the chi-square statistic tends to be overpowered in large samples, resulting in apparent poor model fit and biased fit indices that are based on the chi-square value (Zwick & Velicer, Reference Zwick and Velicer1986). In addition, when a small number of indicators per factor are used in large datasets, fit indices tend to be upwardly biased, resulting in an overestimate of model fit (Marsh, Hau, Balla, & Grayson Reference Marsh, Hau, Balla and Grayson1998). Because “big” datasets by definition have large numbers of records (i.e., volume) as well as rather few indicators per construct, the fit statistics estimated in CFA are likely unreliable.

Like utilizing CFA, using an MTMM to establish construct validity also becomes problematic with big data. Even in the ideal case where a researcher is able to control the behavioral trace data collected as well as include self-report survey or other established methods of measuring a construct (as recommended by Guzzo et al.), the MTMM is unlikely to produce the familiar patterns of correlations for convergent and discriminant validity. Behavioral trace data are almost always collected at a significantly higher frequency than typical survey measurement (i.e., velocity). In addition, surveys are typically administered using a Likert response scale, which produces relatively limited variability made even more restricted by common rating biases such as the halo effect (Thorndike, Reference Thorndike1920). Alternatively, behavioral measures tend to be on wildly different scales with massive variance. Correlations among variables measured at wildly different frequencies with differing amounts of variability will inevitably be rather low. Therefore, the resulting correlations among measures of the same construct with different methods (i.e., convergent validity) will likely be far lower than is typically observed. As a result, it is significantly more likely that different constructs measured with the same method (i.e., discriminant validity), especially self-report surveys, will demonstrate higher correlations than the same construct measured with different methods. This pattern of results would violate the conditions for establishing construct validity according to Campbell and Fiske (Reference Campbell and Fiske1959). Under other conditions of less researcher control such as using behavioral trace date from the Internet or organizationally collected data, it may not even be possible to utilize an MTMM to demonstrate construct validity due to lack of multiple modes of measurement or lack of an established measure to compare against.

A Path Forward

Given the challenges that big data presents with regard to establishing construct validity using traditional approaches, authors, reviewers, and editors will likely need to adapt their requirements for demonstrating validity if the organizational sciences are going to join the big data revolution. Rather than a strict reliance on statistical standards, authors will need to justify the validity of a measure utilizing a theoretical approach combined with demonstrating conceptual overlap with other, more traditional, measures of the construct of interest. For instance, an author could use theory to describe the type of behavior and behavioral pattern that would be expected for a construct and highlight how items from an established scale are designed to measure the frequency or presence of the behavior. To further provide support, authors could employ subject matter experts (SMEs) to rate the relevance of behavioral trace variables for measuring a construct of interest and then test the consistency of SME ratings against established standards for test development (e.g., Lawshe, Reference Lawshe1975). This process is a proven and accepted practice for validating psychological and education tests (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999), and it could prove invaluable for establishing construct validity when other statistical estimates using big data do not meet current standards. It is important to recognize that behavior is inherently context specific, and thus, the specific instantiation of a behavior will vary across studies and should be identified and justified as is recommended for creating behavioral anchored rating scales (BARS; Schwab, Heneman, & DeCotiis, Reference Schwab, Heneman and DeCotiis1975; Smith & Kendall, Reference Smith and Kendall1963). Also, because no single study can establish construct validity (Peter, Reference Peter1981), authors should be allowed latitude so that a preponderance of evidence across situations can be collected and evaluated to determine whether a behavior is in fact indicative of the construct of interest. Likewise, gathering statistical evidence on the relationships between behavioral trace and self-report measures will also help serve as a baseline for potentially adjusting commonly accepted standards for evaluating the size of a correlation in behavioral science (e.g., Cohen, Reference Cohen1988), making the use of CFA and MTMMs in the future more appropriate.

Another potential avenue forward for utilizing big data in spite of the challenges of statistically establishing construct validity is to focus more on process. McGrath (Reference McGrath1964) created the input–process–output model of performance that has dominated organizational thinking and research for over 50 years.Footnote ³ Historically, organizational science has been much better at identifying, measuring, and understanding the inputs and outputs rather than the process. This likely stems from the dominance of cross-sectional designs and self-report measurement. Process is inherently a complex dynamic phenomenon that is not easily studied or understood from a static perspective. Although there are excellent theoretical articles outlining critical processes for teams (e.g., Marks, Mathieu, & Zaccaro, Reference Marks, Mathieu and Zaccaro2001), leaders (e.g., Morgeson, DeRue, & Karam, Reference Morgeson, DeRue and Karam2010), and others, little empirical evidence exists to test them. Big data presents an opportunity to begin to test these theories and unpack the black box of process by examining the behaviors that individuals, teams, and organizations engage in over time.