Kurt Weyland brings welcome attention to an issue of clear importance to political scientists. To learn more about the external-review process, I reviewed the record of my department’s solicitation for promotion letters from 2005 through 2018 (with candidates’ names removed). Of the 435 total promotion requests for 47 candidates (21 to associate professor, 26 to full professor), 292 of those solicited (67%) agreed to write and 106 (24%) declined; 37 (8.5%) did not respond.

The department requested an average of 9.3 letters per candidate. It received 6.2 letters per candidate, and another 2.3 potential reviewers declined to write. Fewer than one solicited reviewer per candidate (0.79) did not respond. Of the 47 promotion candidates, 18 had 0 or 1 declination; 16 had 2 declinations; 10 had 3 declinations; 10 had 4 declinations; 9 had 5 declinations; and 5 had either 6 or 7 declinations.^{Footnote 1} Among the solicited reviewers’ reasons for declining, 36 indicated they were too busy; 19 wrote they were committed to other promotion letters; 11 explained they were on sabbatical or in the field; 7 replied their administrative duties precluded them from writing; and 15 answered they were unfamiliar with the candidate’s work. Only one external reviewer declined because of the lack of confidentiality (with the state of Texas’s open-records laws).

Of the 292 letters received, four fifths were “helpful” external reviews (i.e., “good signs,” in Lieberman’s words) in my assessment, based on being a member of the department’s executive committee for almost all of those 12 years. These were thorough, forthright, and fair letters that evaluated the quality, originality, and impact of the candidate’s contributions to the field or subdiscipline. They were straightforward in their judgement of the candidate’s merits and weaknesses. They contextualized the candidate’s scholarship in a disciplinary genealogy. And they placed the candidate relative to others in her or his cohort (as they were requested to do).

About a fifth of the letters were “unhelpful” (i.e., Lieberman’s “bad signs”), insofar as they did not closely examine or analyze the candidate’s scholarship, but, instead, were overly general and uncritical. They sometimes omitted important aspects of the candidate’s scholarship, frequently did not answer the posed questions (per Johnson’s and Junn’s comments), and often merely stated the number and name of the journal or press where the candidate published, in lieu of engaging in the quality of the ideas, substance, and methods of the candiate’s research. Consistent with Weyland’s observations, the less useful letters were typically written on behalf of the less compelling candidates.

Even so, the promotion candidates did not neatly divide into two exclusive camps. Relatively few candidates had unimpeachable records and even fewer were clearly unworthy of promotion. The number of “problematic” or “weak” candidates—those whose scholarship was “usually not very good” and “usually not very enlightening,” in Weyland’s words—was small, perhaps because of the department’s more exacting third-year and annual reviews. Neither was it always possible to know what to conclude from the declinations. Declinations could be a reflection of the candidate’s subfield or the degree of her or his specialization. Or, they could be simply bad luck, an artifact of the small number of reviews being solicited per candidate and the fact that any set of invited external reviewers could have conflicting administrative responsibilities, previous external-review commitments, be on leave or on sabbatical, have injuries, or experience other issues.

The department’s data nonetheless confirmed the deficiency of the external-review process. When the number of declinations and non-responses was added to the number of unhelpful letters, I found that when asked to write an external-review letter for a promotion case, about one-half of the faculty dropped the ball. They declined to write; they wrote unhelpful letters; or they did not respond to the request for an external letter (or, as if they agreed to write a letter, it was never submitted).^{Footnote 2} For a profession that seeks to govern itself and one whose members—or some of whose members—are concerned about administrative overreach,^{Footnote 3} this is a troubling deficit of professionalism. This record is particularly disturbing in view of the many talented scholars who are underplaced or have yet to land tenure-track jobs.

Offering honoraria to external reviewers would likely promote a higher yield among those solicited. It makes sense to compensate external reviewers so as to tangibly acknowledge the effort it takes to write comprehensive and candid letters. “It takes time to be critical,” as Opheim points out. The granting of more than modest payments raises real questions, however, those of the erosion of professional norms (i.e., writing promotion letters no longer being viewed an academic obligation), of how the honoraria would be funded—especially with the financial inequalities among institutions—and of the impact of the payment of honoraria on the thoroughness and sincerity of external letters.

This may be more of a numbers issue, however. As colleges and universities request more letters per candidate (e.g., one institution I know recently increased its minimum from four to five letters) and as more institutions seek to improve their research credentials, there are more requests for external reviews (per Deardorff’s observation). With the higher demand for letters and the relatively smaller supply of prominent scholars able to write, it is little wonder that there are more declinations and, when external faculty agree to write, more unhelpful letters.

An obvious solution is to expand the pool of letter writers. One way to do so is by relaxing the “peer-institution” restriction that universities have adopted, whether formally or informally, given how widely dispersed expert faculty are across the United States and around the world. The University of Texas, for instance, requests that external reviews be only from Association of American Universities and R1 “peer institutions” or from the few foreign universities that rank in the global top 50. However, the individual credentials of a scholar for the purpose of evaluating a promotion candidate’s record is far more significant than is the prestige or ranking of that scholar’s home institution. Yet university administrators assume institutional ranking to be a proxy for the quality of any one faculty member and her or his department.

Given that many institutions ask that only full professors write in tenure cases, another way to expand the pool is to allow associate professors to write letters for promotion to tenure. Outstanding associate professors may have more recent, more extensive, and more specialized knowledge of a candidate’s disciplinary contributions; they may also have fewer administrative commitments and other responsibilities than their full-professor colleagues and be more willing to write external letters. Again, the CV and scholarly reputation of an external reviewer is of greater significance than whether she or he has (yet) been promoted to full professor.

Both measures would mitigate the numbers problem and likely increase the yield among solicited reviewers—more, I suspect, than by paying honoraria. Whatever the answers, the external-review process merits our sustained thought—as Professor Weyland and the above respondents to his essay have begun to do.