Jaccard coefficient

The Jaccard coefficient for assessing similarity using binaryFootnote ² data has its origins in the biological sciences (Jaccard, Reference Jaccard1912).Footnote ³ Jaccard attempted to categorize alpine regions based on the flora they shared. It is relatively simple to compute and results in a percentage. In set theory language, Jaccard similarity is the intersection of sets A and B (A∩B) divided by the union of sets A and B (AUB), ranging from 0 (no similarity) to 1 (complete similarity). In job analysis language, it is the percentage of the number of shared job attributes (such as tasks) to the total number of attributes present across the two jobs. Attributes that are irrelevant to both jobs are not counted.

$${\rm{Jaccard}}\ {\rm{Index}} = \left( {{\rm{the}}\ {\rm{number}}\ {\rm{in}}\ {\rm{both}}\ {\rm{jobs}}} \right){\rm{/}}\left( {{\rm{the}}\ {\rm{number}}\ {\rm{in}}\ {\rm{either}}\ {\rm{job}}} \right) \times 100.$$

The same formula in notation is

$${\rm{J}}\left( {{\rm{X,Y}}} \right) = \left| {{\rm{X}} \cap {\rm{Y}}} \right|/|{\rm{X}} \cup {\rm{Y}}|.$$

To calculate the index, simply

1. 1. Count the number of attributes that are shared between both jobs.

2. 2. Count the total number of attributes in both jobs (shared and unshared).

3. 3. Divide the number of shared attributes (1) by the total number of attributes (2).

4. 4. Multiply the number obtained in (3) by 100.

This percentage indicates how similar the two sets are.

There are drawbacks to using the Jaccard coefficient. Attributes that are not dichotomous must be dichotomized, and care must be taken in doing so, particularly for qualitative attributes, to avoid overcounting dissimilarity (Beers & Lockhart, Reference Beers and Lockhart1962). Salvatore et al. (Reference Salvatore, Rand, Grytten, Ferkingstad, Domanska, Holden, Gheorghe, Mathelier, Glad and Sandve2020) found that the tetrachoric correlation performed better than the Jaccard coefficient when the binary data of interest had an underlying continuous reality.Footnote ⁴

Gibson–Caplinger similarity index

Gibson and Caplinger developed a strategy for dichotomizing continuous job analysis data (such as frequency ratings and importance ratings) based on the rating scale definitions. For example, on an importance rating scale where 1 = unimportant, 2 = somewhat important, 3 = important, 4 = very important, and 5 = crucial, they dichotomize the distribution at the 3.0 rating such that “less than 3” is not important and “greater than 3” is important. Jobs are then compared using the following formula to determine their overlap.

$${\rm{Similarity}}{\mkern 1mu} \ {\rm{Index}}{\mkern 1mu} \left( {{\rm{SI}}} \right) = {{Nc} \over {\sqrt {Ns \times Nt} }},$$

where N _S and N _T are the number of important attributes for the source and target jobs, respectively, and N _C is the number of important attributes common to both jobs. Gibson recommends a value of .75 as indicative of “substantial similarity” (Gibson, Reference Gibson, Hoffman, McPhail, Tippins and Gibson1999; Gibson & Caplinger, Reference Gibson, Caplinger and McPhail2007).

Although referred to in I-O psychology as the Gibson–Caplinger similarity index, it has its origins in ethnology, a branch of anthropology (Driver & Kroeber, Reference Driver and Kroeber1932), as well as zoogeography, a branch of biology (Ochiai, Reference Ochiai1957)Footnote ⁵. In a review of similarity measures, Warrens (Reference Warrens2019) refers to the Driver–Kroeber and Ochiai similarity measures (and, thus, the Gibson–Caplinger index) as suitable for ordinal data. Also important, this measure of similarity does not count negative matches (i.e., agreement that the attribute is absent).

One of the advantages of using this similarity index and the Gibson–Caplinger .75 threshold is that the U.S. Department of Justice has approved its use to answer the question of what constitutes sufficient similarity, more specifically “substantial similarity,” of jobs for transporting validity evidence obtained for one job to another job.Footnote ⁶

Squared Euclidean distance

Euclidean Distance is a metric for assessing similarity of vectors of ordinal data, such as job analysis rating scale responses (e.g., time spent ratings, importance ratings). As Strah et al. (Reference Strah, Rupp and Morris2022) note, Euclidean distance accounts for differences in elevation as well as pattern. Legree et al. (2010) characterize distance measures as “overall” metrics and the correlation, absolute mean difference, and standard deviation as “component” metrics because they assess the three components of the similarity of two profiles: shape, elevation, and scatter (Cronbach & Gleser, Reference Cronbach and Gleser1953).

Euclidean distance as a measure of similarity is credited to Osgood and Suci (Reference Osgood and Suci1952). They give the formula (p. 253) as follows:

$$D = \sqrt {\sum\nolimits_i {d_{jl}^2}}$$

where d is the mathematical difference between variables j and l. Even though Euclidean distance is based on relatively simple geometry (i.e., the formula for the length of the hypotenuse of a right triangle), it is difficult to explain. Moreover, according to Osgood and Suci (Reference Osgood and Suci1952), the attributes need to be mutually exclusive, an important characteristic for all measures of similarity.

Unlike many similarity metrics, including the Jaccard coefficient, the correlation coefficient, and the Gibson–Caplinger index, higher values of Euclidean distance indicate greater dissimilarity. There are no published thresholds for degrees of job similarity based on Euclidean distance.

Correlation coefficient (profile)

The correlation coefficient is another metric for assessing similarity with ordinal data such as job descriptive information based on time spent ratings and importance ratings. As Strah et al. (Reference Strah, Rupp and Morris2022) note, the correlation coefficient captures the similarity of the pattern or profile of responses but does not capture elevation. Using Legree et al.’s (Reference Legree, Psotka, Robbins, Roberts, Putka and Mullins2014) distinction, the correlation coefficient is a component score. The formula for computing the correlation between w and y (Nunnally, Reference Nunnally1978) is

$${{\rm{r}}_{{\rm{wy}}}} = {{{\sigma _{{\rm{xy}}}}} \over {{\sigma _{\rm{w}}}{\mkern 1mu} {\sigma _{\rm{y}}}}}.$$

The correlation coefficient ranges from −1.00 to +1.00. A correlation coefficient of 0.00 means no relationship, +1.00 means they are exactly the same, and −1.00 means they are exactly the opposite.

Intuitively, an overall measure such as the squared Euclidean distance metric might seem advantageous compared with a component score like the correlation coefficient. However, a growing body of research suggests that indices of shape (e.g., the correlation coefficient) are less susceptible to unwanted variance due to response tendencies and coaching strategies that affect elevation. Research scoring situational judgement test responses against a subject-matter-expert key has shown that using shape scores (e.g., the correlation between each respondent’s vector of responses and the vector of keyed responses) reduced subgroup differences and retained test validity (Legree et al., Reference Legree, Psotka, Robbins, Roberts, Putka and Mullins2014; McDaniel et al., Reference McDaniel, Psotka, Legree, Yost and Weekley2011). Job analysis research has yielded similar findings. Garwood et al. (Reference Garwood, Anderson and Greengart1991) introduced random error in simulated datasets and compared the accuracy of clustering methods and similarity metrics (distance and correlation). They found that using the correlation coefficient as the similarity metric retained a high level of accuracy in clustering solutions despite the presence of error. Thus, if response bias is likely to be a problem, the correlation coefficient might be preferable over distance measures.