Neubert, Mainert, Kretzschmar, and Greiff (Reference Neubert, Mainert, Kretzschmar and Greiff2015) suggest two individual differences that have been overlooked in the industrial–organizational (I-O) literature (i.e., complex and collaborative problem solving) will be critical in the 21st century workplace. Below, I will briefly summarize some history and empirical findings at odds with Neubert et al.’s (Reference Neubert, Mainert, Kretzschmar and Greiff2015) assertions.
First of all, Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) argue that I-O psychologists should pay attention to and take advantage of nascent educational research on complex and collaborative problem solving. This argument is quite odd to me given that these two skills, albeit under different names, have been widely studied in I-O psychology, particularly as major assessment center (AC) dimensions since the first industrial use of ACs in 1956 in a study conducted at AT&T (Arthur, Day, McNelly, & Edens, Reference Arthur, Day, McNelly and Edens2003). According to Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015), complex problem solving is defined as “individuals’ transversal skill of successfully handling complex and intransparent situations (i.e., those without a readily apparent solution), requiring the active acquisition and application of knowledge in various domains,” and collaborative problem solving “is directed toward problem solving in group settings, adding the necessity of social skills to the ones captured by [complex problem solving]” (p. XXX). These are essentially the same as the definitions that have been used for the two corresponding major assessment dimensions: “problem solving” and “consideration/awareness of others” (see Arthur et al., Reference Arthur, Day, McNelly and Edens2003, Table 2, for details). In particular, on the basis of Arthur et al. (Reference Arthur, Day, McNelly and Edens2003), Dilchert and Ones (Reference Dilchert and Ones2009, see Table A1, p. 267) explicitly defined problem solving as “[t]he extent to which an individual gathers information; understands relevant technical and professional information; effectively analyzes data and information; generates viable options, ideas, and solutions; selects supportable courses of action for problems and solutions; uses available resources in new ways; and generates and recognizes imaginative solutions,” and they defined consideration/awareness of others as “[t]he extent to which an individual's actions reflect a consideration for the feelings and needs of others as well as an awareness of the impact and implications of decisions relevant to other components both inside and outside the organization.” Taking research on ACs into account would lead to different conclusions than those articulated by Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015).
Second, Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) emphasize one specific method (computer simulation) used for assessing complex problem solving (note there is no mention of the assessment of collaborative problem solving in the article) as a distinctive feature of this skill. However, computer simulation is only a method. If complex problem solving is a psychometrically sound construct with a clear definition, this can (and ought to) be measured via multiple methods. To be qualified as a unique construct, complex problem solving should be assessed in combination with some similar and different constructs via multiple methods to establish the evidence of construct validity (including discriminant and convergent validity) using a multitrait–multimethod (MTMM) matrix-based approach (Campbell & Fiske, Reference Campbell and Fiske1959). Problem solving and consideration/awareness of others have been widely measured by various methods (exercises) as part of an AC (e.g., simulations, interviews, in-baskets, group discussions), and their construct validity (often via MTMM-based confirmatory factor analysis) has been well established (Arthur et al., Reference Arthur, Day, McNelly and Edens2003). In sum, no compelling evidence is offered that complex and collaborative problem solving are unique constructs distinct from problem solving and consideration/awareness of others in Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015).
Third, Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) argue that “neither classical tests of intelligence nor job-based simulation instruments can account for these characteristic elements of nonroutine and interactive tasks on the basis of both theoretically sound foundations and reliable and valid instruments” (p. XXX). They further argue that complex and collaborative problem solving are likely to be more valid than general constructs such as intelligence (general cognitive ability), personality, and job-specific instruments in predicting individual performance in the 21st century workplace. However, Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) do not provide any direct evidence for this argument. Note that several articles Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) cite to back up this claim are mostly based on students in education settings. In addition, meta-analytic evidence clearly shows that the operational validity of intelligence, the Big Five personality traits, problem solving, and consideration/awareness of others is estimated at .56, .26, .39, and .25, respectively (see Table A2 of Dilchert & Ones, Reference Dilchert and Ones2009, for details). Relatedly, on the basis of two samples (in total, over 4,500 managers), Dilchert and Ones (Reference Dilchert and Ones2009) showed the incremental validity (in the form of R) of problem solving over intelligence and the Big Five is .04, and that of consideration/awareness of others over intelligence and the Big Five is also .04. Although not reported in Dilchert and Ones (Reference Dilchert and Ones2009), the incremental validity (in the form of R) of intelligence and the Big Five over problem solving is estimated at .25, and that of intelligence and the Big Five over consideration/consideration of others is estimated at .37. Thus, it is clear that problem solving and/or consideration/awareness of others provide(s) only modest incremental validity over intelligence and the Big Five, and intelligence and the Big Five are much more important than problem solving and/or consideration/awareness of others in predicting job performance. Thus, it seems obvious that problem solving and/or consideration/awareness of others are less useful than general constructs such as intelligence and the Big Five.
Last but not least, I think there are two additional noteworthy points. First, Gottfredson (Reference Gottfredson1997) defined intelligence as “the ability to reason, plan, solve problems [emphasis added], think abstractly, and acquire knowledge” (p. 93). Based on this widely accepted definition, it follows that problem solving is one of the indicators of intelligence. That is, problem solving is part of intelligence. To postulate that problem solving increments the validity of intelligence is to postulate that one of the indicators of intelligence increments the validity of intelligence. The only way this can be true would be (a) if the intelligence measure is construct deficient (i.e., includes too few indicators) or (b) if the intelligence measure has somewhat low reliability, which allows another intelligence measure/indicator to increase its reliability and hence its validity. This suggests that the incremental validity of problem solving over intelligence, if it is not zero, is likely to be artifactual. Second, as pointed out by Schmidt and Hunter (Reference Schmidt and Hunter1996), incremental validity should be computed at the construct level, because incremental validity at the measure level can be misleading; the incremental validity observed by adding a new predictor/measure/indicator might be simply due to the effect of improving the reliability of the existing predictors/measures/indicators. Accordingly, to ensure that the obtained incremental validity is due to the unique contribution of the new construct (and not the measure per se), we need to compute incremental validity after fully correcting for measurement error (i.e., based on fully corrected correlations). It is noted that the aforementioned incremental validities were estimated using operational validity estimates (i.e., based on partially corrected correlations). Thus, it is possible that the incremental validity (in the form of R) of problem solving over intelligence and the Big Five and that of consideration/awareness of others over intelligence and the Big Five alike could be smaller than .04 and close to zero if they were computed based on fully corrected correlations.
In conclusion, many new constructs have been introduced to I-O psychology over the years, but few of them have been found to be unique and useful by providing compelling construct and criterion-related validity evidence. Before making several bold and unsubstantiated claims summarized above, Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) should have more carefully examined the current I-O literature (in particular, extensive research on ACs). Further, some of their claims are premature because direct evidence regarding the construct and (incremental) criterion-related validity of complex and collaborative problem solving in organizational settings is not yet available.
Neubert, Mainert, Kretzschmar, and Greiff (Reference Neubert, Mainert, Kretzschmar and Greiff2015) suggest two individual differences that have been overlooked in the industrial–organizational (I-O) literature (i.e., complex and collaborative problem solving) will be critical in the 21st century workplace. Below, I will briefly summarize some history and empirical findings at odds with Neubert et al.’s (Reference Neubert, Mainert, Kretzschmar and Greiff2015) assertions.
First of all, Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) argue that I-O psychologists should pay attention to and take advantage of nascent educational research on complex and collaborative problem solving. This argument is quite odd to me given that these two skills, albeit under different names, have been widely studied in I-O psychology, particularly as major assessment center (AC) dimensions since the first industrial use of ACs in 1956 in a study conducted at AT&T (Arthur, Day, McNelly, & Edens, Reference Arthur, Day, McNelly and Edens2003). According to Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015), complex problem solving is defined as “individuals’ transversal skill of successfully handling complex and intransparent situations (i.e., those without a readily apparent solution), requiring the active acquisition and application of knowledge in various domains,” and collaborative problem solving “is directed toward problem solving in group settings, adding the necessity of social skills to the ones captured by [complex problem solving]” (p. XXX). These are essentially the same as the definitions that have been used for the two corresponding major assessment dimensions: “problem solving” and “consideration/awareness of others” (see Arthur et al., Reference Arthur, Day, McNelly and Edens2003, Table 2, for details). In particular, on the basis of Arthur et al. (Reference Arthur, Day, McNelly and Edens2003), Dilchert and Ones (Reference Dilchert and Ones2009, see Table A1, p. 267) explicitly defined problem solving as “[t]he extent to which an individual gathers information; understands relevant technical and professional information; effectively analyzes data and information; generates viable options, ideas, and solutions; selects supportable courses of action for problems and solutions; uses available resources in new ways; and generates and recognizes imaginative solutions,” and they defined consideration/awareness of others as “[t]he extent to which an individual's actions reflect a consideration for the feelings and needs of others as well as an awareness of the impact and implications of decisions relevant to other components both inside and outside the organization.” Taking research on ACs into account would lead to different conclusions than those articulated by Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015).
Second, Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) emphasize one specific method (computer simulation) used for assessing complex problem solving (note there is no mention of the assessment of collaborative problem solving in the article) as a distinctive feature of this skill. However, computer simulation is only a method. If complex problem solving is a psychometrically sound construct with a clear definition, this can (and ought to) be measured via multiple methods. To be qualified as a unique construct, complex problem solving should be assessed in combination with some similar and different constructs via multiple methods to establish the evidence of construct validity (including discriminant and convergent validity) using a multitrait–multimethod (MTMM) matrix-based approach (Campbell & Fiske, Reference Campbell and Fiske1959). Problem solving and consideration/awareness of others have been widely measured by various methods (exercises) as part of an AC (e.g., simulations, interviews, in-baskets, group discussions), and their construct validity (often via MTMM-based confirmatory factor analysis) has been well established (Arthur et al., Reference Arthur, Day, McNelly and Edens2003). In sum, no compelling evidence is offered that complex and collaborative problem solving are unique constructs distinct from problem solving and consideration/awareness of others in Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015).
Third, Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) argue that “neither classical tests of intelligence nor job-based simulation instruments can account for these characteristic elements of nonroutine and interactive tasks on the basis of both theoretically sound foundations and reliable and valid instruments” (p. XXX). They further argue that complex and collaborative problem solving are likely to be more valid than general constructs such as intelligence (general cognitive ability), personality, and job-specific instruments in predicting individual performance in the 21st century workplace. However, Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) do not provide any direct evidence for this argument. Note that several articles Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) cite to back up this claim are mostly based on students in education settings. In addition, meta-analytic evidence clearly shows that the operational validity of intelligence, the Big Five personality traits, problem solving, and consideration/awareness of others is estimated at .56, .26, .39, and .25, respectively (see Table A2 of Dilchert & Ones, Reference Dilchert and Ones2009, for details). Relatedly, on the basis of two samples (in total, over 4,500 managers), Dilchert and Ones (Reference Dilchert and Ones2009) showed the incremental validity (in the form of R) of problem solving over intelligence and the Big Five is .04, and that of consideration/awareness of others over intelligence and the Big Five is also .04. Although not reported in Dilchert and Ones (Reference Dilchert and Ones2009), the incremental validity (in the form of R) of intelligence and the Big Five over problem solving is estimated at .25, and that of intelligence and the Big Five over consideration/consideration of others is estimated at .37. Thus, it is clear that problem solving and/or consideration/awareness of others provide(s) only modest incremental validity over intelligence and the Big Five, and intelligence and the Big Five are much more important than problem solving and/or consideration/awareness of others in predicting job performance. Thus, it seems obvious that problem solving and/or consideration/awareness of others are less useful than general constructs such as intelligence and the Big Five.
Last but not least, I think there are two additional noteworthy points. First, Gottfredson (Reference Gottfredson1997) defined intelligence as “the ability to reason, plan, solve problems [emphasis added], think abstractly, and acquire knowledge” (p. 93). Based on this widely accepted definition, it follows that problem solving is one of the indicators of intelligence. That is, problem solving is part of intelligence. To postulate that problem solving increments the validity of intelligence is to postulate that one of the indicators of intelligence increments the validity of intelligence. The only way this can be true would be (a) if the intelligence measure is construct deficient (i.e., includes too few indicators) or (b) if the intelligence measure has somewhat low reliability, which allows another intelligence measure/indicator to increase its reliability and hence its validity. This suggests that the incremental validity of problem solving over intelligence, if it is not zero, is likely to be artifactual. Second, as pointed out by Schmidt and Hunter (Reference Schmidt and Hunter1996), incremental validity should be computed at the construct level, because incremental validity at the measure level can be misleading; the incremental validity observed by adding a new predictor/measure/indicator might be simply due to the effect of improving the reliability of the existing predictors/measures/indicators. Accordingly, to ensure that the obtained incremental validity is due to the unique contribution of the new construct (and not the measure per se), we need to compute incremental validity after fully correcting for measurement error (i.e., based on fully corrected correlations). It is noted that the aforementioned incremental validities were estimated using operational validity estimates (i.e., based on partially corrected correlations). Thus, it is possible that the incremental validity (in the form of R) of problem solving over intelligence and the Big Five and that of consideration/awareness of others over intelligence and the Big Five alike could be smaller than .04 and close to zero if they were computed based on fully corrected correlations.
In conclusion, many new constructs have been introduced to I-O psychology over the years, but few of them have been found to be unique and useful by providing compelling construct and criterion-related validity evidence. Before making several bold and unsubstantiated claims summarized above, Neubert et al. (Reference Neubert, Mainert, Kretzschmar and Greiff2015) should have more carefully examined the current I-O literature (in particular, extensive research on ACs). Further, some of their claims are premature because direct evidence regarding the construct and (incremental) criterion-related validity of complex and collaborative problem solving in organizational settings is not yet available.