Hostname: page-component-7b7567c8c-jswmv Total loading time: 0 Render date: 2026-11-15T21:28:40.451Z Has data issue: false hasContentIssue false
  • English
  • Français

Determinants of Political Science Faculty Salaries at the University of California

Published online by Cambridge University Press:  25 September 2009

Bernard Grofman
Bernard Grofman
Affiliation:
University of California, Irvine
Rights & Permissions [Opens in a new window]

Abstract

Combining salary data for permanent non-emeritus faculty at seven departments of political science within the University of California system with lifetime citation counts and other individual-level data from the Masuoka, Grofman, and Feld (2007a) study of faculty at Ph.D.-granting political science departments in the United States, I analyze determinants of faculty salaries. For the full data set the main finding are that (1) base salaries of UC political science faculty are slightly more strongly correlated to citation rates (annualized or total lifetime citations) as a measure of research visibility than they are to seniority measured by years since receipt of the Ph.D.; and (2) that gender differences and subfield differences in salary essentially vanish once I take into account both year of Ph.D. and research visibility (as measured by annualized citation counts), while gender inequities would appear to exist if I did not control for both variables.

Information

Type
The Profession
Information
PS: Political Science & Politics , Volume 42 , Issue 4 , October 2009, pp. 719 - 727
Copyright
Copyright © The American Political Science Association 2009

INTRODUCTION

Questions about the determinants of faculty salary are of general interest. Of particular concern are issues of the relationship between seniority and compensation, gender and compensation, and research performance and compensation. Here I take advantage of data involving seven departments of political science within the University of California (UC) system where I was able to combine publicly available salary information with lifetime citation counts and other individual-level data from the Masuoka, Grofman, and Feld (Reference Masuoka, Grofman and Feld2007a) study of faculty at Ph.D.-granting political science departments in the United States.Footnote 1 While this study is limited to the determinants of faculty salaries in political science departments within the UC system, I believe that the results have broader implications.

Theory

How might we predict the relative weight of different factors in the determination of base salaries? There are competing theoretical expectations as to what we should find. The bureaucratic model suggests that, because performance is at least in part a contested concept within academia,Footnote 2 and because the UC system is a large bureaucracy, the system may cope with salary determination with fixed rules that establish a time-based ladder of salary increments that is uniformly applied across all campuses. Indeed, such rules exist in the Academic Personnel system of the university. To the extent that we have bureaucratic rigidity, we would expect a very strong correlation between salary and years since Ph.D.

At the same time, the UC system and the State of California greatly value the quality and strong national reputations of the faculty and their research programs. Thus, to the extent that there is flexibility in salary determination for base salaries, and concern about recruiting and keeping first-rate faculty, I expect the administration at the various campuses to respond to both equity and market considerations by taking performance factors into account, not merely seniority. This should show up in the data as a relatively high degree of variance in the pay levels of those with similar dates of Ph.D.

Another theoretical concern is how to best predict the nature of the differences in salary determination across the various UC campuses.Footnote 3 One possibility is that the existence of system-wide norms and a system-wide salary ladder common to all campuses will yield very high congruence across campuses in the factors affecting salaries.Footnote 4 A second possibility is that those departments with higher national visibility will, controlling for other variables, pay less because they compensate their faculty in non-monetary ways, for example, by providing them the prestige associated with faculty status at a top department, top graduate students, or other, intellectual benefits. A third possibility is that, after controlling for other variables, top departments pay more than the less visible departments. This may occur if the top departments are more willing to compete aggressively for the most professionally visible and productive faculty by paying competitive salaries and being more willing to tolerate salary differences across individuals with similar date of Ph.D. A fourth and related possibility is based on the idea that top departments would, in general, pay more attention to productivity measures (relative to seniority) than would less highly ranked departments when it came to determining faculty compensation. This differentiation can also lead to variance differences across departments in the dispersion of faculty salaries for a given number of years past Ph.D.Footnote 5

My analyses test the differing expectations of the bureaucratic and the performance-based models of salary determination, and of the various hypotheses about cross-campus differences in salaries and in salary determination methods, by using citation counts as the measure of research performance.Footnote 6 While the data base does not have teaching or service information, I was able to take into account some variables not directly related to research performance that might affect salary levels: cross-campus differences, subfield differences, and gender differences. I focus on six specific questions about faculty salaries. First, how well can we predict the salaries of UC political science faculty by simply knowing the date of Ph.D.? Second, how well does actual faculty compensation accord with compensation determined by projecting salaries based on normal progress on the UC salary ladder for someone with a given number of years since Ph.D.? Third, what is the contribution of research performance, as measured by citation counts, to the determination of faculty salaries? Fourth, is there any evidence of a gender gap? In particular, how is any gender gap in salaries affected when I control for number of citations and other variables? Fifth, is there evidence of salary differences across political science subfields? Finally, how similar are the (inferred) salary-determination processes at the different UC campuses? That is, are there significant intercampus differences in salaries even after controlling for the seniority and research visibility of the campus's faculty? And, if so what is the nature of these differences?

Data

This project builds on a prior study of citation counts of all faculty at Ph.D.-granting political science departments in the United States done by Masuoka, Grofman, and Feld (Reference Masuoka, Grofman and Feld2007a). Using the list of Ph.D.-granting political science department faculty as of 2002 as a basis, Masuoka, Grofman, and Feld compiled the total citation counts for each faculty member using the Web of Science/ISI online database. The principal investigators also coded a number of other variables I used in these analyses: the year of receipt for the Ph.D., the institution of each faculty member, field of specialization within political science, and gender. I primarily utilized career citations here, spanning the 1960–2005 period. I also calculated a measure of average citations per year since receipt of the Ph.D. Salary data are from “University of California, Fiscal Year 2005–06 Payroll Data” reports as available at the Langston Library at UC Irvine.Footnote 7 I used base salary as listed in these reports, with some modification.Footnote 8

My comparisons of citations are slightly more precise than those in Masuoka, Grofman, and Feld (Reference Masuoka, Grofman and Feld2007a) because I restricted myself to active tenure-track faculty, which excludes emeriti and non-tenure-track appointments that are sometimes included in studies that do departmental citation counts. I also excluded UC faculty members who held a full-time administrative position and whose salary is based on this position, such as a vice chancellor or dean. I excluded faculty who are paid on the UC salary scale for law schools in order to study faculty whose work is primarily in political science and who are evaluated within political science departments.Footnote 9 This yields a total database of 153 faculty who were employed in seven UC political science departments in 2002 and are still employed full time in 2005 (UC Santa Cruz and UC Merced are not included because their Ph.D. programs were not yet established at the time the data were collected). In short, I compared how faculty salaries vary among a group who work in the same discipline of political science, and who are evaluated by a common standard of academic performance by UC political science departments and the UC Academic Personnel system.

SENIORITY AND UC BASE SALARY

How strongly is seniority related to salary? I first looked at the bivariate relationships between base salary and years since Ph.D. I expected that faculty salaries generally rise with experience as people become more expert in their profession as teachers and stronger contributors to the university and the community, in ways separate from research impact. I used years since receipt of the Ph.D. as a predictor of accumulated experience in the profession There is a strong positive and statistically significant relationship between years since Ph.D. and the faculty member's 2005–06 base salary (Pearson r = .52). However, there is clear evidence of heteroscedasticity (see Figure 1).

Figure 1.

Base Salary (2005–06) by Years since Ph.D.

Source: Salary data from “University of California, Fiscal Year 2005–06 Payroll Data”; years since receipt of Ph.D. from Grofman, Masuoka, and Feld database. N = 153.

My second question is the degree of fit between base salary and the salary predicted by the UC salary ladder based on normal time in step and normal time in grade (using the 2005–06 salary schedule and accompanying documentation). I used time since Ph.D. to predict the step (and thus salary) faculty members if they had merit evaluations on the normal time basis and had positive evaluations at each merit evaluation (using the 2005–06 salary schedule and accompanying documentation).Footnote 10 The correlation between base salary and expected normal salary is .51, virtually the same as the correlation between base salary and years since Ph.D. Figure 2 groups time since Ph.D. into six-year intervals to show the average patterns in the data. Average salaries are considerably higher than the salary-scale norm for political science faculty early in their careers. For instance, there is roughly $25,000 per year in additional salary compared for those from one to 18 years past Ph.D., and an average of nearly $50,000 higher for faculty who are in their 10 to 24 year since Ph.D. (i.e., the cohort that is now about 20 years past Ph.D. seems to be especially well paid relative to Academic Personnel norms). However, the gap between the UC-ladder norm and actual salary then narrows considerably until, for most senior faculty, there is essentially no difference between what the ladder system would expect them to be paid based on normal progress and their actual baseline salaries.Footnote 11

Figure 2.

Salaries and Normal-Progress Predicted Salaries by Years since PhD

Source: Salary data from “University of California, Fiscal Year 2005–06 Payroll Data”; years since receipt of Ph.D. from Grofman, Masuoka, and Feld database. N = 153. Salary-ladder data for tenure-track faculty as reported by the UC Academic Personnel system for 2005–06.

THE BASIC TWO-VARIABLE MODEL: TIME SINCE PH.D. AND CITATIONS

I begin this part of the analyses with a simple two-variable model to explain the base salary of UC political science faculty.Footnote 12 The first variable is years since Ph.D., which we I have seen is already quite powerful in explaining salary variation. The second variable is a measure of research impact. I recognize that the research contributions of faculty can take many forms, and the recognition of these contributions can take many forms. To measure research impact I relied on citation counts for the 1960–2005 period as reported by the Web of Science/ISI database. This measure of research productivity seems preferable to other indicators, such as number of publications or external grant funding, because citations measure how much the larger scholarly community utilizes a faculty member's research.Footnote 13 Fitting my expectations, there is a strong positive relationship between the total citations and the 2005–06 base salary (r = .59). There is also an equally strong relationship between the yearly rate of a scholar's citations (citations/year) and base salary (r = .58).

As I expected, years since Ph.D. and citation counts are positively related (r = .45), but they reflect distinctly different sources of advancement. Thus, the next step in the analysis combines both predictors in an ordinary least squares (OLS) regression to predict salary. The results are largely consistent with the above analyses for each variable separately (see Table 1). For instance, the left columns in Table 1 measure lifetime citations; a year of experience adds about $1,040 to the base salary, holding constant research citations. Each individual citation adds $36.50 on average, holding constant years since Ph.D. Similarly, the right-most columns in Table 1 use a faculty member's annualized citation count. An increase in the average of one citation per year predicts a salary increase of $1,234, almost as much as the estimate for a year of additional seniority. The standard errors indicate that all the relationships are well above the usual levels required for statistical significance. Also, the standardized coefficients in Table 1 show that years since Ph.D. and citation counts are both important as salary determinants, but with the latter somewhat more important.

Table 1.

Predicting UC Political Scientists Base Salary from Years since Ph.D. and Citations (Lifetime Citations and Citations per Year)

Source: Salary data from “University of California, Fiscal Year 2005–06 Payroll Data”; citations and year of Ph.D. from Grofman, Masuoka, and Feld database. N = 153.

Note: The first data column presents un-standardized regression coefficients; standard errors are in parentheses; the second column presents standardized coefficients.

To help readers understand the practical implications of these findings, Table 2 presents some descriptive information about total citation counts and annualized citation counts for UC faculty at different stages of their careers. The average number of citations received annually by faculty is relatively low—averaging 14 per year for all faculty in the database. Moreover, as noted earlier, the average is not highly correlated with years since Ph.D.Footnote 14

Table 2.

Citation Counts and Annualized Citation Counts by Years since Ph.D.

Source: Citations and year of Ph.D. from Grofman, Masuoka, and Feld database.N = 153.

GENDER EQUITY

The University of California is strongly committed to an equitable merit and promotion system in which faculty are evaluated on the basis of merit. There is a continuing UC concern about whether gender or other variables are systematically related to merit evaluations, but external non-subjective evidence on research visibility is seldom available as part of these considerations. For example, at my home campus, the provost conducts an annual gender equity study that compares men's and women's salaries within given professorial ranks for each of the administrative units. That study controls for years since Ph.D. but not for other variables that might be regarded as indicative of research (or teaching) performance (see http://www.ap.uci.edu/Equity/studies). The methodology used for this study is based on recommendations developed for the entire UC system. The availability of citation counts in the research I report here provides a new perspective on gender-equity questions.

There is a gender gap in the simple comparison of base salary by gender. The left bar in Figure 3 shows that women on average receive $13,442 less than men among all UC political science faculty in the study.

Figure 3.

Gender Differences in 2005–06 Salary (UC Political Science Faculty)

Source: Salary data from “University of California, Fiscal Year 2005–06 Payroll Data”; citations and year of Ph.D. from Grofman, Masuoka, and Feld database. N = 114 men and 39 women.

Note: The first data column presents gender gap in base salary; the second column presents the gender gap controlling for years since Ph.D.; the third column controls for years since Ph.D. and total citations.

A portion of this difference may be because women tend to be younger and thus hold less-senior ranks (and salary). Thus, the second bar in Figure 4 displays the gender gap controlling for years since Ph.D.: the gap reduced dramatically, to a $4,920 difference.

Figure 4.

Salary Gap by Subfield after Controlling for Seniority and Citations

Source: Salary data from “University of California, Fiscal Year 2005–06 Payroll Data”; citations and year of Ph.D. from Grofman, Masuoka, and Feld database. N = 153.

Note: The figure presents the subfield-level residuals in salary predicted from the basic two-variable model. Positive values indicate an average higher salary for a subfield after controlling for seniority and citations.

As noted above, the UC system and individual campuses have conducted such analyses of gender differences in the past, but these studies have not had access to systematic measures of research impact, such as citations, that may help explain salary variation. When I added the citation variable to the equation, the gender gap reduced to $1,844.Footnote 15 Furthermore, using conventional measures of statistical significance, the highly significant differences in the base salary and first control model are no longer significant in the two-variable model (F test = .138, sig. = .71). In summary, my overall findings provide evidence for gender equity of salary determination in the UC system as a whole. System wide, there is no statistically significant gender gap in salaries after controlling for seniority and citations.

SUBFIELD DIFFERENCES

I also compared salary residuals for the six major subfields used in Masuoka, Grofman, and Feld (Reference Masuoka, Grofman and Feld2007a): American politics, comparative, IR, methodology, political theory, and public policy (including law and public administration).Footnote 16 I had thought there might be subfield differences in salaries, but I did not find statistically significant differences. The only substantial deviation, a lower salary, once I took citations into account, is for the methodology subfield. Of course, there is always the possibility that a larger sample might yield statistically significant results for other subfields, but it is unlikely to change the estimates of the magnitudes of the effects, which are (even methodology) quite small. Moreover, as we will see later, even the difference for the methodology subfield vanishes once I estimated a more complete model.

CROSS-CAMPUS DIFFERENCES

A distinctive feature of the University of California is the unified nature of the system. All 10 campuses are part of a single unit, with common academic and educational goals. In addition, the campuses have a common merit-evaluation and step system—and in theory should be applying this merit-evaluation system equivalently across campuses. One does not expect that all UC departments will be equally highly ranked nationally because, if nothing else, the size of the departments vary widely across campuses from 45 at UCLA in the full database to five at UC Riverside, but the UC system expects that faculty with comparable records are rewarded comparably across campuses. This is the logic of the UC Academic Personnel system.

There are clear differences in department rankings within the University of California. Masuoka, Grofman, and Feld (Reference Masuoka, Grofman and Feld2007b) find that UCLA, UC Berkeley, and UC San Diego ranked sixth, seventh, and eighth, respectively, in the cumulative citations of all faculty over the 1960–2005 period (or eleventh, fifteenth, and sixth in per capita terms). UC Irvine was ranked seventeenth (thirteenth per capita), and neither UC Davis nor UC Santa Barbara were ranked in the top 30.Footnote 17

Figure 5 reports differences across departments in their average citations and median citations using the database, which is limited to full-time current faculty at these departments. UC San Diego and UC Berkeley are either ranked first or second on both statistics; UCLA and UC Irvine are either ranked third or fourth on both statistics; and the other three campuses are ranked lower on both measures. However, the UC campuses differ in the age distribution of their faculty. On average, UC Berkeley and UC San Diego faculty are 28 to 29 years past their Ph.D.s; UC Davis, UC Irvine, UCLA, and UC Santa Barbara faculty average 22 to 23 years beyond their Ph.D.s; and for UC Riverside the average is 18 years.

Figure 5.

Average Lifetime Citations per Faculty Member

Source: Citations from Grofman, Masuoka, and Feld database. N = 160; these analyses include faculty on administrative salaries who are excluded from my salary comparisons.

Because simple citation counts may be rather misleading due to age-composition differences across the campuses, Table 3 shows another perspective on these analyses by comparing citation counts across the campuses broken down by years since Ph.D. With only 153 cases, I divided the sample into faculty from the three highest-ranked political science departments (UC Berkeley, UCLA, and UC San Diego), and the other four Ph.D. programs (UC Davis, UC Irvine, UC Riverside, and UC San Barbara). The table shows whether citation-count differences across the two sets of campuses are due merely to the fact that faculty at UC Berkeley, UCLA, and UC San Diego are generally older than those at the other campuses. On average, the three top-ranked campuses are apparently now doing a better job in recruiting and retaining highly cited faculty than their four sister campuses, but the gap narrows and then even reverses among faculty at the very highest level of seniority, but the sample sizes in that group are very low and I do not regard these differences as important.Footnote 18

Table 3.

Comparisons of Citations between Top-Ranked and Other Departments

Citation counts—as a measure of research visibility and merit—have more effect on salary (as shown by the standardized coefficients) than simple seniority. This led me to inquire whether the relative weight of these two predictors varies significantly across UC campuses. With 153 cases, I again divided the sample into faculty from the three highest-ranked political science departments (UC Berkeley, UCLA, and UC San Diego), and the other four Ph.D. programs (UC Davis, UC Irvine, UC Riverside, and UC Santa Barbara). (See Table 4.)

Table 4.

Predicting 2005–06 Base Salary across Campuses

Source: Salary data from “University of California, Fiscal Year 2005–06 Payroll Data”; citations and year of Ph.D. from the Grofman, Masuoka, and Feld database. N = 153. The significant predictors are denoted by an asterisk (p < .05).

Note: The first data column presents un-standardized regression coefficients, standard errors are in parentheses; the second column presents standardized coefficients. The omitted campus is UC Berkeley; the omitted subfield is American politics.

There is a striking contrast across these two groups of UC campuses. While the coefficient on citations is essentially the same in both groups of campuses, the relative importance of the citations variable is greater at top-ranked departments because these departments display only modest salary increases as purely a function of years since Ph.D. (b = 458), while the seniority variable is much stronger in determining salaries at the other four UC campuses (b = 1,614). In other words, ceteris paribus, it pays three times more to grow older at the latter group of four campuses. On the other hand, and cutting in the opposite direction, the constant is higher at UC Berkeley, UC San Diego, and UCLA, suggesting that these campuses may have higher baseline salary levels.Footnote 19 Because these effects do not all go in the same direction it is helpful to look at the expected salary levels across the two different groups of campuses as a function of years past Ph.D., holding constant mean citations (per cohort), as shown in Figure 6.Footnote 20

Figure 6.

Comparison of Salary Levels for a Typical UC Campus-Wide Political Science Faculty of a Given Number of Years Past Ph.D. and with Citations Equal to the System-Wide Average for that Cohort

In summary, despite a wage scale that is, in principle, identical across the UC system, my overall findings provide evidence for considerable differences in the processes of salary determination in the UC system. The more highly ranked campuses, on balance, pay their faculty more, even after I controlled for citations.

AN ALTERNATIVE ESTIMATION MODEL

My analyses focused on a two-variable model to predict faculty base salary, and then used residual analyses to compare differences by other characteristics. In part, this methodology follows the residual analysis models that are often used in UC salary-equity studies. In addition, there are theoretical arguments that seniority and citations should be considered prior to comparing gender, subfield, or other group differences.

However, despite the modest N for the total sample, I recognize that other analysts might wish to include all the variables I discussed into a single multivariate regression rather than performing residual analyses on combinations of variables. Therefore, I also present such a full model in Table 5. I began with seniority and citations from the basic model; I then added dummy variables for each UC campus (treating UC Berkeley as the excluded category), and dummies for the subfields of political science (treating American politics as the excluded category), and for gender.

Table 5.

Predicting 2005–06 Base Salary (UC Political Science Faculty)

Source: Salary data from “University of California, Fiscal Year 2005–06 Payroll Data”; citations and year of Ph.D. from the Grofman, Masuoka, and Feld database. N = 89 for first group and 64 for second group.

Note: The first data column presents un-standardized regression coefficients; standard errors are in parentheses; the second column presents standardized coefficients.

The results largely confirm my earlier conclusions but some variables earlier found to be statistically significant in reduced equations are no longer so in the fuller model. Research citations continue to exert the strongest influence on base salaries, with a coefficient comparable to that found in the two-variable basic model. Seniority again has a secondary effect. There remain some significant inter-campus differences in salary. In this equation, I allowed for campus-specific differences. UCLA pays significantly more than other campuses (+$22,408) and UC Irvine significantly less (−$14,307), even after controlling for the other variables in the model. None of the subfield differences emerged as statistically significant.

There is one other slight but still potentially important difference in outcomes between these two methodologies. There was a small albeit statistically insignificant gender gap in salaries in the residual analyses from the two-variable basic model, with women earning $1,844 less than men. In the full multivariate model that accounts for the different subfield specializations of women and their different distribution across campuses, there is now a reversed gender gap of $631 to the benefit of women. Again, however, this coefficient is not statistically significant. Given the lack of statistical significance and the difference in sign, I interpret these results as again showing no evidence of a gender gap in political science salaries within the UC system once appropriate controls are introduced.

DISCUSSION AND SUMMARY

The salaries of political science faculty within the UC system appear to be determined to a large degree by a combination of research performance, as judged by lifetime or annualized citations, and a norm of an upward path for salary over time. Gender effects or subfield effects appear to be essentially nonexistent once controlled for seniority and research visibility. Thus, at least taken as a whole, UC political science departments have done a very good job in addressing gender-equity issues once the two key salary determinants of seniority and research visibility are taken into account.

Of course, there are performance measures other than citation counts that could be used for research performance, and there are important areas of performance, for example, teaching and service, for which I have no information, and whose inclusion might affect my conclusions about the relative importance of seniority and citation variables.Footnote 21 Moreover, there are issues of multi-collinearity that make assigning the relative importance of research performance and time since Ph.D. difficult to disentangle—albeit these seem relatively minor when I use citations per year as the citation measure. But, caveats notwithstanding, my system-wide comparisons generally find that research performance, measured by lifetime or annualized citations, explains more of the variation in salaries than mere time since Ph.D. And it also appears that the three most highly ranked University of California political science departments generally assign less relative weight to seniority in making salary judgments than do the four other departments, and have a greater variance in salaries among those roughly the same number of years past Ph.D.—especially for younger and middle-aged cohorts—as well as having higher salaries for all but the most senior faculty once I took into account citation differences.Footnote 22

APPENDIX: Methodological Comments

This report focused on basic comparisons of salary across UC political science faculty. In the course of these analyses, I encountered a number of methodological or interpretative points that should be noted.

Base Salary

I used published statistics on base salary because this is the only measure at our disposal. In the vast majority of cases, this is the most appropriate measure of the university's compensation to faculty. If an individual is a department chair or has other part-time administrative responsibilities, or has external grant income, this should be treated as separate from their base salary. I did not include compensation beyond base salary in these analyses because the reasons for that extra compensation can vary and is not identified in the UC budget summaries. In some instances, faculty might receive additional compensation for additional administrative responsibilities (such as a department chair or center director) or summer support on research grants, but in other cases compensation might be summer salary that theoretically should be added to the base. Because I could not differentiate, I relied solely on the base salary. My strong impression is that the willingness to make use of summer salary or other salary supplements varies across the seven campuses, because in viewing the raw data it became apparent that there are substantial campus-by-campus differences in other compensation that affects total compensation. (This appears to be most common at UCLA, where almost half the faculty receive additional compensation and half of these receive at least two-ninths in addition to the base.) I suspect that looking at total compensation rather than baseline compensation would further increase the compensation differences between campuses, even holding years since Ph.D. and citations constant, but limitations of the published data make it impossible to conduct such analyses.

Predicting Salaries

Quite obviously, research is only one aspect of salary determination, and I do not have data on other aspects of professional careers that may impact salary levels such as teaching and service. I recognize that there are many reasons for salaries being what they are, including extraordinary teaching, unusual service or positions held (either administrative or in professional societies), the need to compensate for outside offers, and so on. This report is only the beginning of the kinds of empirically grounded analyses that might be done. In addition, citations are merely one measure of academic visibility and merit. The more such measures we have, the more likely are any true inequities to be accurately recognized and thus corrected. However, the University of California prides itself on being, collectively, the preeminent public research university in America, and thus the impact of research is expected to be a key factor in UC merit and promotion.

Citations as a Measure of Research Impact

There are various issues in using citations as the sole measure of academic visibility and impact. First, while Masuoka, Grofman, and Feld (Reference Masuoka, Grofman and Feld2007a) sought to be extremely careful in generating their citation-count data, numerous problems exist having to do, for example, with scholars with similar names and scholars with complex or hyphenated names that may appear in multiple forms within the database, as well as with issues in the American Political Science Association (APSA) data set such as how emeritus faculty were listed and when members came or left the faculty. There are some differences in the data set I use here from that found in Masuoka, Grofman, and Feld (Reference Masuoka, Grofman and Feld2007a) to which I need to call attention. Masuoka, Feld, and Grofman's list of faculty came from information submitted by department chairs to the APSA and they did not have the resources to verify the faculty listings for all 132 campuses. Because I examine only seven UC campuses and I have salary data to identify the full-time current faculty at each of the campuses, I only report data for actual full-time faculty who are also in the Masuoka, Grofman, and Feld database (ca. 2005, the most recent year for which the salary data are publicly available). Relatedly, I must acknowledge one important limitation of using the Masuoka, Grofman, and Feld (Reference Masuoka, Grofman and Feld2007a) study, or similar data from the Web of Science database, namely the fact that it omits a substantial number of citations to co-authored work when the faculty member was not first author, especially books. However, I believe that this problem does not introduce systematic bias into our estimates for departments and broad groups of faculty. Still I would note that a few individual faculty would see their citation count change significantly if these citations were added. Second and relatedly, I have not attempted to weight citations for number of authors and place of authorship. Third, while most of my analyses are based on total lifetime citations, which can create bias in terms of career stage, I also replicated these analyses using annual citation averages. Because I obtained broadly similar results I have not bothered to present these duplicate analyses. Fourth, there are other ways than citations to measure research visibility, such as counting number of books and other publications, perhaps weighted by some prestige or impact measure. However, citation data can be preferable to publication data in that many publications, even ones in prestigious journals, or published by prestigious presses, have little or no impact or visibility in the field, while some work published in less prestigious journals may nonetheless come to be highly visible and influential. While it is true that the Social Sciences Citations Index (SSCI) only lists citations that are found in articles, all citations in those articles, whether to books, or to other articles, or even to unpublished materials, are tallied in the SSCI counts. Relatedly, while not all journals are in the SSCI citation database, this is a very large set and includes virtually all well-established disciplinary journals as well as journals in other disciplines that some political scientists may publish in. Fifth, both citation counts and publication counts are only imperfect measures of research visibility. Some works not directly cited because of their lack of direct substantive relevance can nonetheless be highly important in shaping perspectives in the field. Indeed, sometimes older publications may largely drop from citation only because their main message has been so thoroughly absorbed. And, perhaps even more importantly, there are other measures of research visibility and impact such as office holding in professional societies, major grants, and other awards and honors of various types that may only be imperfectly correlated with citation counts.

Alternative Models

To estimate the impact of a year of service or a citation on salaries I relied on linear OLS models as the simplest and most intuitive method of estimating relationships. I have observed that most relationships appear basically linear and additive. However, the relationship between citation counts and salary is somewhat non-linear, reflecting a common pattern of diminishing returns. That is, the salary consequence of citations decreases as the total number of citations increases. I also computed the basic two-variable model using a log transformation of citations to represent this non-linear pattern. This increases slightly the standardized coefficient for citations (.55) and decreases slightly the coefficient for seniority (.21), but the overall explained variance does not change. Therefore, I relied on the basic linear model in my analyses.

Footnotes

1 Their database with the citation counts for all faculty at Ph.D.-granting political science departments in the United States is available at: http://www.socsci.uci.edu/~bgrofman/Masuoka_Grofman_Feld_PS400%20FULL%20listing4.xls.

2 How should teaching, service, and research be weighted in judging faculty performance? And how best should each be measured? Answers to these difficult questions are being developed internationally, such as the nationwide evaluation of departments done in Britain on a five-year cycle, with similar models under development in Australia and other nations. See the 2001 Research Assessment Exercise (RAE) in Britain: http://www.hero.ac.uk/rae/.

3 Both in the U.S. and abroad, the University of California is often seen as a model of how to create a decentralized university system that still shares a common administrative structure.

4 In principle, the University of California system is unified in that there is an identical salary ladder at all campuses, and it has the expectation of “comparable salaries for comparable merit,” across all political science departments in the system.

5 Another factor that might affect salary differences across campuses is that cost of living (and especially cost of housing) varies in different parts of California. Also, some campuses provide funds for mortgage support, or offer partly subsidized on-campus housing. These forms of compensation are not reflected in base salaries. Investigation of these differences must be left to future research.

6 There are various important issues in using citation counts as the sole measure of research visibility. See the discussion in the methodological appendix to this article and further discussion of both the limits and the advantages of citation counts as a measure of research impact in Masuoka, Grofman, and Feld (Reference Masuoka, Grofman and Feld2007a) and references cited therein.

7 Salaries of all state employees are, according to California law, public information. Although the UC Board of Regents possesses considerable autonomy, and there is a separate pension system for the university, faculty and staff in the University of California system are still considered state employees. The Regents has now made it possible to ascertain individual staff and faculty salaries within the UC system. Previously, in practice, the only UC faculty salary data that were publicly available were data on average salaries at the department level or even higher levels of aggregation. Even now, using the salaries of individual faculty or staff for the entire 10-campus UC system as data for research purposes is not that easy because the files are not online but are available only as printed budgetary reports.

8 See methodological appendix.

9 In three instances, it seemed that the base salary reflected part-year appointments. For the UC Irvine case I was able to determine the fraction of the appointment, and adjust the base salary. For the other two cases, I was not able to determine the fractional appointment, and thus I have deleted these three cases.

10 I calculated a two-year merit cycle for assistant and associate professors, and a three-year cycle for full professors. I then assigned them a predicted salary based on the step they would hold if they had followed this normal-progress course.

11 I remind the reader that I am limiting myself to base salaries and there may be other ways in which senor faculty are reimbursed that do not show up in that base.

12 I have also analyzed the difference between actual salary and the salary expected from Academic Personnel norms based on the number of years since Ph.D. Because the results are very similar, but are harder to interpret, I have not reported those results here.

13 It has been suggested that subfield within political science might explain some variations in citation rates, since some subfields are larger than others. However, an informal examination of the Masuoka, Grofman, and Feld data suggests that at least the best-known scholars in the various subfields have roughly comparable citation counts.

14 Why annualized citation rates are so weakly correlated with years since Ph.D. is an intriguing research questions, but not one I pursue here. Multiple factors are likely at work. For instance, high visibility perhaps comes earlier in one's career, and then early work is replaced by later work and thus drops off in citations over time. Differences in cohort size, that is, the growth in the number of political scientists and publications per capita over time, ceteris paribus, might suggest that the citation counts reflect this population inflation. The publication norms in political science have also changed over time, with graduate students more involved in the publication process even before completing their Ph.D.'s. We might also expect that younger scholars are more likely to read and cite more recent work because it appears in the journals they are currently reading.

15 This is the residual from the two-variable model with total citations as the measure of citation visibility.

16 I report residuals from the two-variable model with total citations as the measure of citation visibility.

17 US News and studies of Ph.D. placement and published articles can also be used to provide national ranks. Here the latter four campuses are generally lower down, but even so, a similar general pattern of relative placement holds for most measures.

18 Thus, the data suggest that, as the present older cohorts retire, the average citation gap between the political science faculty hired by UC Berkeley, UCLA, and UC San Diego and those of the other four campuses may widen.

19 One might debate the causal flow underlying these relationships, but they indicate that the top UC political science departments relate salary more directly to the research visibility of the faculty, as well as generally offering higher salaries for a given level of seniority. On the other hand we must be careful in interpreting these findings vis-à-vis expected salary levels, because the functions shown above are simple linear ones and prediction error is heteroscedastic over years since Ph.D.

20 We need to be careful, however, in interpreting Figure 6, because it assumes a counterfactual, namely that the average citation counts per faculty are the same, on average, in each cohort at the first three campuses as at the latter four, and we know from Table 3 that this is not true.

21 In particular, if teaching success and/or service contributions are highly positively correlated with research performance as measured by citations, then these estimates of citation-count effects on salary may be overstated. Of course, if teaching success and/or service contributions were highly negatively correlated with research performance as measured by citations, then the estimates I have made about the contributions of citation counts to salary determination would be understated.

22 While correlation is not causation, these findings strongly suggest that, at least within the UC system, developing research excellence by hiring and retaining first-rate political science faculty, especially junior and mid-level faculty, is likely to be associated with salaries that are more clearly tied to research visibility and less to simple seniority, as well as to a higher overall salary. Such choices seem strongly associated with the policies pursued by the most highly nationally ranked political science departments in the system.

References

Masuoka, Natalie, Grofman, Bernard, and Feld, Scott L.. 2007a. “The Political Science 400: A 20-Year Update.” PS: Political Science (January): 133–45.Google Scholar
Masuoka, Natalie, Grofman, Bernard, and Feld, Scott L.. 2007b. “Ranking Departments: A Comparison of Alternative Approaches.” PS: Political Science (July): 531–37.Google Scholar
Figure 0

Figure 1.Base Salary (2005–06) by Years since Ph.D.Source: Salary data from “University of California, Fiscal Year 2005–06 Payroll Data”; years since receipt of Ph.D. from Grofman, Masuoka, and Feld database. N = 153.

Figure 1

Figure 2.Salaries and Normal-Progress Predicted Salaries by Years since PhDSource: Salary data from “University of California, Fiscal Year 2005–06 Payroll Data”; years since receipt of Ph.D. from Grofman, Masuoka, and Feld database. N = 153. Salary-ladder data for tenure-track faculty as reported by the UC Academic Personnel system for 2005–06.

Figure 2

Table 1.Predicting UC Political Scientists Base Salary from Years since Ph.D. and Citations (Lifetime Citations and Citations per Year)

Figure 3

Table 2.Citation Counts and Annualized Citation Counts by Years since Ph.D.

Figure 4

Figure 3.Gender Differences in 2005–06 Salary (UC Political Science Faculty)Source: Salary data from “University of California, Fiscal Year 2005–06 Payroll Data”; citations and year of Ph.D. from Grofman, Masuoka, and Feld database. N = 114 men and 39 women.Note: The first data column presents gender gap in base salary; the second column presents the gender gap controlling for years since Ph.D.; the third column controls for years since Ph.D. and total citations.

Figure 5

Figure 4.Salary Gap by Subfield after Controlling for Seniority and CitationsSource: Salary data from “University of California, Fiscal Year 2005–06 Payroll Data”; citations and year of Ph.D. from Grofman, Masuoka, and Feld database. N = 153.Note: The figure presents the subfield-level residuals in salary predicted from the basic two-variable model. Positive values indicate an average higher salary for a subfield after controlling for seniority and citations.

Figure 6

Figure 5.Average Lifetime Citations per Faculty MemberSource: Citations from Grofman, Masuoka, and Feld database. N = 160; these analyses include faculty on administrative salaries who are excluded from my salary comparisons.

Figure 7

Table 3.Comparisons of Citations between Top-Ranked and Other Departments

Figure 8

Table 4.Predicting 2005–06 Base Salary across Campuses

Figure 9

Figure 6.Comparison of Salary Levels for a Typical UC Campus-Wide Political Science Faculty of a Given Number of Years Past Ph.D. and with Citations Equal to the System-Wide Average for that Cohort

Figure 10

Table 5.Predicting 2005–06 Base Salary (UC Political Science Faculty)