%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
A paper arrives in the editorial office of an academic journal. A number of years ago it would have been sent by post in an envelope – occasionally sent ‘express’ by an American author anxious about the postal system. Later, it would have been sent as an attachment to an email. Nowadays, it is likely to be uploaded to a website. By whatever means it arrives, an editor will have to read the submission to assess its prima facie suitability for publication. Assuming that the editor thinks it suitable for consideration, it will then be sent to reviewers or referees for a report. In due time, and with some luck, the reports from the referees will come in. Each referee will have come to a judgement about its suitability for publication. They may agree in their judgements; they may disagree. If they disagree, the editor will have to decide whether to call for more reports, involving more delay, or to make a decision on the reports at hand. Sometimes, these reports and the editor’s judgement will lead to a common conclusion that the paper should be published as it is or with only minor revisions. Sometimes, the consensus will be that the paper should be rejected. Most commonly, even a positive inclination to publish will require significant revisions, possibly involving more data collection, further analysis, attention to a missing theoretical framework, a new conceptual argument, the reworking of a proof, an argument of principle or simply better linguistic expression.
When revisions are called for, the reports and editor’s letter are sent to the author. Authors then have to read and make sense of the reports and the covering letter. They will have to consider whether to make the revisions or to send the paper as it is to another journal. Assuming that they are prepared to make the revisions, they will have to undertake the work required, possibly putting off other work to which they are committed. Having undertaken the work, the paper will be re-submitted to the journal. If the original decision was conditional upon a further report from reviewers, then the original cycle will be repeated, but with the referees informed of their original views, and possibly also the views of other reviewers, together with a request to offer a further view, which will eventually inform the judgement of the editor. If the original decision only required the editor to make a decision about the adequacy of the revisions, then the editor will have to reconsider the paper in the light of the scale and quality of the revisions.
It is not unknown for the review process to go through a further iteration at this point, but more normally it will end in a decision to reject or to accept. If it is the latter, the article, with more or less tidying up by the authors and some hands-on work by the editor, will go into production, first to a copy-editor, who may request more details or clarification of details, and then to the typesetters. The author will receive proofs in due course, proofs that will require checking, and perhaps the provision of further information, before being returned to the typesetter for revision and later to the printer. The editor will be involved in scheduling the article, fitting it into an issue, and if he or she is doing their job, they will remember that – whatever the absence of page constraints on the web – hard-copy issues should be made up in such a way that the number of pages is a multiple of eight, or even of sixty-four. Eventually, the article will appear, nowadays either in electronic and paper form or most often in both. In the case of this Journal, the author receives a free pdf of their paper as soon as it is published online, a copy of the issue of the Journal in which the article appears, and the option to purchase twenty-five offprint copies of their article. The paper then inhabits Karl Popper’s ‘first world’ of knowledge encoded in books and machines, where it waits to be read and – with luck – cited.Footnote 1 Another piece in the vast mosaic of human understanding will have been put in place.
This process can be described and in some ways quite adequately analysed in cybernetic terms. We can think of the journal publication process as a system with inputs, throughputs and outputs. At various points after information – or at least digital data – has been fed into the system, a series of switches come into play that determine whether the information, suitably modified by transformers, will emerge as an output at the other end, or will simply be some form of friction in the system. In high-demand journals, the switches will be set such that their default mode is to reject the input from the system, unless there is a strong countervailing signal. For example, if each reviewer is one switch, then it may take three switches in the ‘on’ position for a paper to make it past review to the next stage of publication. Some switches may be very good signal detectors, but others may exhibit a large stochastic element in their response – a particularly interesting phenomenon if one of these random switches is an editor. There are also feedback loops from output to input: a high output of one type of signal increasing the probability that more such signals will be sent as input, perhaps leading to the amplification of certain types of signal in the journal output.
I say that the process can be described in cybernetic terms, and there is some merit in this way of looking at the matter, but this is far from saying that it has to be described in those terms. Indeed, we miss much about what is important in the decision process if we regard it in such an abstract way, just as systems theorists of politics missed much about their subject by their adoption of an overly abstract way of thinking. Here is a paradigmatic statement of the systems perspective when applied to the study of politics:
… instead of the concept of the ‘state,’ limited as it is by legal and institutional meanings, we prefer ‘political system’; instead of ‘powers,’ which again is a legal concept in connotation, we are beginning to prefer ‘functions’; instead of ‘offices’ (legal again) we prefer ‘roles’; instead of ‘institutions,’ which again directs us toward formal norms, ‘structures’…Footnote 2
As this quotation makes clear, what is lost from the systems description of a decision process are the ‘meanings’, ‘powers’ and ‘norms’ that actors in the system use to describe to themselves and to others what it is they are doing when they are acting in the system. Yet any institution or practice, which is what a decision system is, is also a normative system in which the various actors have entitlements, claims, powers, duties, liabilities and Hohfeldian freedoms that make up any normative order above minimum complexity. It is this normative order that constitutes the scholarly authority of journals. In the rest of this article, therefore, I shall analyse the scholarly journal as a system of norms.
The primary method upon which I base my analysis is that of participant observer, as author, reviewer and editor. I have been an author for thirty-five years, a reviewer for nearly as long and a journal editor for half that time. Unfortunately, although I have given talks and presentations over the years on my experience, I have not kept field notes, as all good participant observers should. I shall, therefore, be in breach of one of the norms of contemporary reliable scholarship in not having sources of data that can be made publicly available to others for replication. Whether someone in breach of this norm is entitled to write about the journal as a system of norms is a nice question that I leave for readers, who should in any case triangulate what I say with their own experience as well as the reports of others.
I begin by describing in outline the norms that are applicable to the principal actors in the journal system. Some of these norms can be stated clearly, but others, some of which are the most important, are subject to variability of interpretation. This fact alone warrants a discussion of the contrast between norms as implicit as distinct from norms as explicit features of the system. I stress the ‘tacit knowledge’ that is an integral part of the norms of a scholarly journal, which in turn leads on to a discussion of norms as developing and the problem of monitoring. In discussing these points I assume that the overarching purpose of journal publishing in political science is the production of reliable inter-subjective understanding of politics and the political.
The System Described
Authors
Journals state the norms that authors are expected to follow. A typical selection of norms will look like this: the paper should be an original piece of work by the author; it should not be under consideration for publication elsewhere; it should be within the field of study indicated by the journal’s statement of purpose; it should conform to a certain number of words; it should be properly referenced.
Let us begin with the last of these norms. In my experience, many authors are too bothered about meeting the letter of this norm, but insufficiently bothered about meeting its spirit. For example, the British Journal of Political Science has always used a footnote system, whereas many journals these days use the Harvard system. Authors who have written a paper using the Harvard system sometimes wonder whether editors will reject the submission initially merely on the technicality. Sometimes, they would ask; more often, they would send in the paper with a covering note saying that, if the paper were to be accepted, then they would alter the referencing system. From this point of view, the norm of proper referencing is easy enough to follow.
What seems more difficult for authors to follow is proper referencing in the scholarly sense. What does one do with a reference of the form: ‘John Rawls, A Theory of Justice’ with no pages indicated? A Theory of Justice makes many arguments and offers a great many concepts.Footnote 3 Even something as generic to the book as ‘justice as fairness’ ought to be referenced to particular pages or at least particular sections. Too often this elementary maxim of scholarly accuracy is ignored. In such a case, referencing is not being used to help the reader follow and expand the evidence and argument but becomes a form of ritualistic mantra seeking to establish spurious authority. The problem becomes even worse when authors persistently not only cite a book but ascribe to it a view that it does not contain. Time after time you will read that Anthony Downs held that it was not rational to vote and only the book will be cited without benefit of pages.Footnote 4 Should anyone bother to read pages 266–71 of that fine work, they will find that Downs thinks that part of the return to voting is the long-run participation value that people derive from the act, and that this is an element in the calculus. He certainly does not hold that it is irrational to vote. Nor incidentally does he hold that in an election individual voters are trapped in a prisoner’s dilemma; rather, they are in ‘a maze of conjectural variation’ taking the form of a chicken game, in which it becomes more rational to vote the more likely others are to abstain, and less rational the more that others are to vote.Footnote 5
Failure to cite – the fault of negligence – is as serious as committing the sin of inaccurate or vague citation. My guess is that in most cases failure to comply with the norm of fair dealing that is implicit in the requirement of accurate referencing is due to laziness and poor record keeping rather than a deliberate attempt to scorn or downplay a rival. Failure to cite can be imprudent, however. Many scholarly sub-disciplines are quite small, and the number of reliable people available and willing to review in a sub-discipline may be very small. So authors should realize that their failure to cite may redound to their disadvantage if they do not take full account of the work of those who are likely to act as reviewers.
Self-citation is less of a problem for a journal than it is for evaluators of research performance.Footnote 6 What is problematic, however, is adherence to the norm that the paper submitted be an original piece of work. There is, I think, a fine line between a genuinely new paper that grows out of a continuing stream of work and an attempt at salami-slicing one paper into two or three. In following the norm of submitting an original piece of work, authors too often fall prey to self-love, which as John Locke pointed out makes ‘Men partial to themselves’,Footnote 7 and so they think that salami-sliced work is really a new paper. Of course, the incentives to salami-slice have become greater over the years as more explicit research assessment at the individual and institutional level takes place. There is, however, a certain irony in the situation that salami-slicing is only consistent with a high reputation provided that no one actually reads your corpus of work.
Reviewers
Reviewers or referees (I use the terms interchangeably) are central to the workings of a journal. Indeed, although authors are two a penny, good reviewers are worth their weight in gold. As an editor, I came to the view that the quality of reports provided by a reviewer was as reliable an unobtrusive measure of someone’s professional commitment as one could get. Refereeing is an anonymous activity. To be sure, some journals, including the British Journal of Political Science, allow reviewers to reveal their names to authors, which in the case of papers that are eventually published can lead to explicit, and sometimes warm, acknowledgement. However, in cost–benefit terms, this must be a pretty thin incentive for most people, as is the reward of seeing one’s name in a list of dozens who have similarly helped the journal in the task of sifting through submitted papers. Of course, committed scholars will often learn something to their own advantage in undertaking refereeing, even if it is only a few more references. Yet, the sense of personal reward, understood in purely self-regarding terms, must be quite small, and lower than the time and effort taken in reviewing. Here, as elsewhere in life, the old feminist joke can be suitably generalized: you have to kiss an awful lot of frogs before you meet the prince.
What possible motive, then, can there be to undertake the task of reviewing? For some people it will be a purely disinterested concern for the common good. If such people cannot properly be described along the lines of Urmson’s ‘saints and heroes’, they certainly come close in scholarly terms and are a key element of assessment upon which editors rely.Footnote 8 For others, the motive to review will be less a matter of purely disinterested concern and more a matter of fair play: if one expects others to review one’s submitted papers, one ought to play one’s own part in maintaining the practice of peer review. The late Lee Sigelman, who was a fine editor, used to advance a formidable principle in this regard. To the person who cavilled at being asked a third time to review something, he would ask how many papers the reviewer expected himself to submit. If the person said, for example, four, then Sigelman would point out that four papers implied twelve referees, so that even after three reviews, the person being requested to review would still owe the system nine more. If all authors acted on the maxim that for each paper they submitted a quota of three reviews delivered to time was implied, the journal enterprise would function more efficiently and effectively.
A trickier question of reviewer norms, however, is not their motive but defining what counts as good practice. The basic principle of reviewing is simple to state. Reviewers have a responsibility to judge honestly, fairly and in an informed way the scholarly merit of the article they are confronted with. In my view, this duty is owed to the editor and the journal that is making the request, rather than to the author. There are duties to authors, in particular that of refraining from gratuitously rude comments on the merits of the paper and the performance of the author. However, the chief positive duty is one that is owed to the journal and the editor. If there is a fault into which even some of the best reviewers fall, it is that they attempt to be too helpful to the author, turning their report into a catalogue of advice as to how it might be improved and failing to be clear for the editor what are its merits as it has been submitted. The fault is compounded when referees take it upon themselves to internalize the interests of their sub-discipline and qualify their judgement of merits by a judgement about the desirability of the journal publishing more work in field X or field Y. Once the desirability of promoting the collective interests of a sub-field becomes confused with the merits of a particular paper, the norm of a fair and honest judgement has been compromised.
A question that often arises is the propriety of offering a view on a paper for which one has already been a referee for another journal. Some referees think it unfair on the author if they make the same negative judgement for one journal as they made for another. However, this view falsely presupposes that a ‘no double jeopardy’ rule ought to apply to journal submission, such that if an author has been turned down by one journal as the result of a set of reviews, then the author ought not to be exposed to the same risk with another journal. In my view, there are a number of fallacies involved in this line of reasoning (although simple prudence, let alone courtesy, ought to lead editors to respect the judgement of a reluctant referee).
The first fallacy stems from the point about to whom the duty of judgement is owed. If it is owed to the journal, then those running the journal have a right to know that a paper has been judged deficient by an expert, particularly if there have been no changes at all. Spotting problems is not always easy, but once problems have been spotted they cannot be wished away. Secondly, avoiding double jeopardy is not a principle that should be applied to authors. If their paper falls into the hand of a referee who has previously rendered a negative judgement, they are not risking punishment again for a crime they may have committed, but are simply being exposed another time to the judgement of a qualified peer. In the editorial equivalent of a Habermasian ideal speech situation, editors would have to hand the judgement of all relevant peers.Footnote 9 Perforce they only have a sample of the community, but if there is a negative voice in that sample, it should not be suppressed.
Editors
And so to editors: their job is to publish the best set of papers that they can in every issue in such a way as to advance understanding. There are obvious differences between running a general journal that aspires to cover the full range of sub-fields in a discipline and editing a specialist journal, but in many ways the similarities are as striking as the differences.
In relation to authors, the editors’ task is to make a decision in as timely a fashion as possible given the judgements that they have received from referees. It makes a difference to the way in which they discharge their obligations as to whether they conceive of the referees as advisers or judges. If it is the latter, then there appears to be a simple rule of management that some find it attractive to follow, namely do not publish a paper that does not get the support of all the referees: one strike and the paper is out. This is attractive for journals with a large number of submissions, where the need is to filter out the submitted papers that are unsuitable until a manageable number remains. Moreover, it is a rule that is particularly attractive to those who edit association journals, especially when there is controversy in the association about the style and coverage of the journal. A rule that says one negative judgement is sufficient to render a paper unpublishable appears to have the characteristics of due process and equity, while being speedy and simple to apply.
Whatever the merits of such a mechanistic approach from the point of view of protecting editors against accusations of bias and partiality, they are outweighed in my view by the likelihood that the result will be a journal full of papers acceptable to those in a range of sub-disciplines but lacking any reach across sub-disciplines. Moreover, as the late Brian Barry was keen to point out, if such a rule applied, then very few normative theory papers would be published, normative theorists being such a cantankerous lot. Particularly in a general journal that aspires to publish innovative work that has resonance across the discipline, to make referees judges and to require unanimity of opinion among judges is to undermine the goals that the journal is seeking to achieve. This is the heart of the case for editors to live by the norm of editorial responsibility and to take upon themselves that task of making an assessment of just how seriously to take the opinion of referees. Sidney and Beatrice Webb used to have an agreement by which Sidney would take the important decisions, Beatrice would take the unimportant decisions, but Beatrice decided what was important and what was unimportant. Someone has to play the Beatrice role with respect to referees’ reports, and the only person who can perform that function is the editor – or the editorial team when they act as a collective.
This means that an editor has to assume the responsibility for the decision on a paper that is ultimately made. During the seventeen years for which I was one of the co-editors of the British Journal of Political Science, I wrote hundreds of rejection letters. In virtually every case, I somewhere used the phrase that ‘saying “no” is the worst part of an editor’s job’ and I meant it every time. Of course, those who live by the sword will die by the sword, and just as I have sent rejection letters, so I have received them. But for all concerned, no matter how battle hardened, it is a bitter moment.
I shall here take the opportunity to return to authors, and ask what the correct norms are for authors who undergo rejection. In my experience, the modal response by far is simply to take the decision in silence. Some rejected authors seem to be deterred from ever submitting to the journal again; for others, it seems to strengthen their resolve to continue to submit, in the hope of eventually getting something in. From an editor’s point of view, there is no reason to expect a response to a rejection. After all, the author needs to be thinking of pastures new. Sometimes, however, an author will write in, thanking the editor and reviewers for their work, even if rather ruefully, and noting that rejection is part of the game. Occasionally, and in seventeen years I can count the instances on the fingers of one hand, a rejected author will launch a broadside. I remember one author who seemed to think that the correct response to a letter of rejection was to accuse me of being anti-Semitic and threatening to make this fact known among his acquaintances. It was never explained by the person concerned why this particular soubriquet was thought appropriate.
Any editor involved in a journal is likely to be less worried by false negatives than false positives. Nowadays, the degree of technical sophistication in statistical analysis and formal modelling is so great that any editor has to rely upon the technical abilities of reviewers. I was always fortunate in reviewers in that they seemed to be able to spot problems and explain the details in a non-technical way so that I could see what the problems were, but I still had more than the occasional anxiety about letting something through that would turn out later to be found to be literally nonsense. Yet, alongside this understandable concern about false positives, there are good reasons why editors also ought to worry about false negatives. If you reject a paper, the temptation is to comfort yourself with the thought that, if it has merits, it will find a home elsewhere. Yet, this can be a false comfort. Perhaps, in rejecting an interesting paper by mistake, you have deprived that paper of just the readership that it needs.
Some General Themes
Norms are intrinsically open-textured. The principal reason for this is that they cannot be stated in such a way that all the possible contingencies in which they might be applied can be foreseen and in which they might have to be qualified or modified. This applies as much to norms that appear precise as it does to norms that wear their imprecision on their faces. A norm stating a word limit of ‘no more than 12,000 words’ might seem more precise than a norm requiring ‘adequate referencing’, and in most circumstances it will be. However, every so often a paper will come along that breaches the norm of word length, but it will still be considered. The more precise the norm, the more it is likely to have the adverb ‘normally’ attached to it, and it then becomes as open-textured as any other norm. Another way of putting the point is to say that though many important journal norms can be stated explicitly, their real significance is implicit, denying formal statement.
The open texture of norms requires tacit knowledge – ‘know how’ rather than ‘know that’ – in the application of judgement to particular cases. There are some papers that wear their publishability or otherwise on their sleeve, but most do not. When there are discrepant reports, the editor will have to weigh up the relative merits of the reasons for publication and the reasons against. There is no algorithm for this process of judgement and, in my view, it is impossible to routinize it or characterize it in a formula. This is unsatisfactory for authors who want a clear statement not only of the grounds on which the decision has been taken but why the considerations that have shaped the decision have been weighed in the way that they have.
As well as being open-textured, norms also develop. One important norm that has developed in the last twenty years has been the move towards insisting that when researchers publish a paper, they should make the data on which that paper is based available to other researchers, so that the results can be replicated. In a world in which first publication rights are important, this is not a trivial norm to follow, but as ‘climategate’ at the University of East Anglia has shown, researchers can run into trouble if they appear not to follow it.Footnote 10
Another norm that has developed in the last twenty years is that of double-blind reviewing. Here, I confess to being a heretic, for a number of reasons. One reason is pragmatic. Salami-slicing and plagiarism are more difficult to detect with double-blind reviewing. Moreover, in a world in which authors make working papers available on websites, it can be a relatively simple matter for someone to discover the name of the author, even if they did not know it beforehand. This means that we are not comparing full anonymity with full disclosure; instead, we are comparing partial anonymity with full disclosure. And whilst one can make an argument that in the former case full anonymity is to be preferred, it is hard to see what the argument is for favouring partial anonymity. The case for double-blind reviewing is really an attempt to substitute a standard procedure for reliance on the internalized norms of professional responsibility that the disclosure of authors’ names used to require. Of course, revealing author names is quite consistent with not revealing reviewer names. Some have urged full disclosure on both sides, but the benefits of candid assessment that reviewer anonymity provides outweigh, in my view, any supposed gains in transparency that there might be by such a practice.
The efficient production of public goods within a social institution is facilitated by monitoring.Footnote 11 If reliable understanding of politics and the political is the purpose of the journal system, in the light of which the suitability of its particular norms is to be judged, then it would seem that compliance with norms poses a significant problem, since there are many features of the way in which journals work that make monitoring hard or virtually impossible. I have already mentioned the anonymity that goes along with double-blind reviewing. In addition, there is the sheer number of journals dispersed in time and place with the multiplicity of decisions taken every day. Research assessment at individual and institutional levels may have created strong incentives to manipulate and cheat.
Some conspicuous examples of cheating have taken place in the natural sciences, but even political science journals have had to withdraw articles in which plagiarism has been discovered. In this context, there are some bureaucratic pressures to tighten up on administrative procedures in universities so that suitable ethical standards are maintained. My own view is that it is still the case that the strongest pressure to produce valid understanding comes with the internalized norms that are associated with good professional training and apprenticeship. The authority of knowledge is not easily protected by the authority of rule-making.
Conclusion
In summary, the academic journal is an institution, and as an institution it is a system of norms that constrain and enable certain types of behaviour. We should analyse those norms in the light of the overall purpose of the institution, which is the production of reliable inter-subjective understanding of, in our case, politics and the political. Once we do this, we can see that controversies and ambiguities about the appropriate norms can be resolved in particular ways. However, clarity about norms is never sufficient to render the need for judgement obsolete. The journal is a system of norms; but it is not only a system of norms: it is also responsible behaviour within those norms.
