2. Bottom line

For larger conferences, the lack of effort (and respect/collegiality) is shocking (and widespread). As a result, I announced on social media that I am done with large conferences that use ARR and OpenReview, though I will continue to support journals like this journal and TACL, as well as smaller conferences such as SIGIR. If that means that I can no longer participate in certain events, so be it. I can no longer support the toxicity. Just say no.

Many of you will continue to do the minimum because you may not have a choice, but large conferences need volunteers that do more than the minimum. They also need help from people like myself that can afford to just say no.

Enough negativity. What can we do to be part of the solution? The discussion below includes a number of constructive suggestions, many of which have been published elsewhere (Church Reference Church2005; Church Reference Church2020; Church and Kordoni Reference Church and Kordoni2022; Church et al. Reference Church, Chandrasekar, Ortega and Ahmad2025). While these suggestions may not be new, they need to be taken seriously in the near future, or else large conferences may not have a future.

3. Some venues are better than others

Conferences are often at risk when they appear on lists of top venues such as this.Footnote ^h These lists are like lists of popular tourist destinations. These destinations may have been great once, but less so after they have been “discovered” (and struggle to manage the crowds).

IMHO, some venues are better than others. I was pleasantly surprised by SIGIR. SIGIR has a number of advantages. Unlike more popular destinations, it remains relatively undiscovered. It is smaller, both in terms of raw submissions, as well as growth rates over time. The community is unique (and dedicated to the cause).

As a co-chair of the perspectives track, it became clear that SIGIR reviewers are more qualified (in terms of h-indexes). SIGIR reviewers believe in the cause. There is a clear difference in effort.

In addition, SIGIR is more willing to respond to criticism. In particular, it has been suggested that SIGIR has been too resistant to changing times, creating opportunities for newer venues such WWW, WSDM, and RecSys. The perspectives track is an attempt to address that criticism. While the attempt may or may not succeed, I respect the effort.

4. Incentives

For larger conferences, the process has become too transactional. In exchange for each submission, authors are drafted as reviewers. Now that reviewers are treated as lowly conscripts, many of them find ways to return the insult by doing as little as they can get away with.

In the failed Soviet Union, there was a joke:Footnote ⁱ

Incentives matter. We used to review because we believed it was the right thing to do, but if we are now reviewing because we have been drafted, we will do less than we did when we believed in the cause.

A student recently suggested that I be a co-author on his paper, but I turned him down because I do not want to review for ARR. That said, I would be happy to nominate his paper for review under the proposed process below. That would also require me to review, of course. So why do I feel better about reviewing under the proposed process than under the current process? What is the difference?

Answer: I am happy to support students, but not toxicity. As a nominator, I am contributing my reviews in order to help a promising student that I believe in. Moreover, under the proposed nomination process, many of the other reviewers would also be contributing reviews to support students they believe in. That is a community I can support (as opposed to the current toxic swamp).

5. Recent experience as area chair (AC)

As for my recent experience as an area chair, about 1/3 of the reviews in my batch were late. The senior area chair kept pestering me to pester the late reviewers, but I was concerned about the lack of concern for the quality of the reviews that arrived on time.

What goes around, comes around. The lack of effort is a consequence of the lack of respect for one another’s time. Bots were sending me way too many nasty emails, many demanding immediate attention to some silly thankless chore that added little value. Only a bot could be so concerned with busy work, and so disrespectful of everyone’s time.

As an area chair, I need to decide which 20% to accept and justify those decisions. In my batch, almost all grades were between 2 and 3 (on a scale of 1 to 5). It was hard to distinguish one submission from another based on these reviews. The reviews were uniformly negative. Few added value. It was hard to find positive/constructive statements to justify accepting any of the papers in my batch. It has been suggested that reviews are often written by bots. I do not know if that is true, but it might explain why so many of the reviews were so unhelpful. Most reviews picked on some tiny irrelevant detail that was easy to pick on but not helpful for my task.

Given so little signal from the reviewers, I was forced to read the papers myself and make my own judgment about which papers to accept and which to reject and why. I tried to justify those decisions by quoting bits and pieces of the reviews in my meta reviews, but honestly, it would have been easier to review the papers myself than to cherry pick what little I could from the reviews I had to work with.

Worse, as an author, bots were sending me nasty emails threatening to hold reviews hostage unless I convinced my co-authors to do such-and-such. These emails tended to come from no-reply@xyz, so it was impossible to negotiate a reasonable compromise. If the committee does not respect my time enough to address legitimate concerns with emails from a real person who is open to discussion, then they can hardly expect me to address their concerns with much enthusiasm.

At some point, I received email requesting (demanding) emergency reviews. I tried to decline the request (since my reviews were held hostage), but these bots make it very hard to decline “invitations” to review. Seriously? It feels like our venues are no better than online scams that make it way too easy to subscribe and way too hard to unsubscribe.

Reviewing for a conference should be easier than reviewing for a journal. Most journals make it easy to decline an invitation to review. Journals are less toxic and more collegial. Editors are responsive.

Based on this recent experience, I decided to continue supporting journals and smaller venues that have yet to be “discovered,” but not top tourist destinations until more effective processes are put in place to manage the crowds.

6. Scale and bloat

Given the size of these large venues, the primary requirement is scale. Other requirements are nice to have, but secondary.

There have been many attempts to improve the process. Most such attempts fail. Committees do what committees do. Committees often make a bad situation worse by adding more and more steps (bloat) that may be well intentioned but fail to manage the crowds.

ARR was one such attempt that made a bad situation worse. We used to have about four conferences a year. ARR replaced that with a dozen cycles but now we are back to four. The point of the dozen cycles was to speed up the process, but that was never the case, as discussed in Table 2 of Church and Kordoni (Reference Church and Kordoni2022). Where are we after all this process re-engineering? We used to have about four program committees for four conferences. We now have that, plus rotating ARR committees with overlapping authority. Too confusing to scale. More bloat. Little progress to show for all this work. Much of this work had little to do with managing crowds.

In politics, there is a cliche: it is the economy stupid; in our case, it is scale stupid! Big venues will die if we cannot figure out how to deal with scale.

The process is too complicated. To fix the process, we need to eliminate bloat.

Consider rebuttals. Are they worth the effort? Rebuttals are expensive in terms of time and effort. They rarely change the outcome. What is the return on investment?

Similar comments apply to discussion. Meaningful discussion rarely happens in Open Review and rarely changes the outcome. If a step in the process fails to produce significant returns on investment, the step needs to be eliminated.

Reviewers for large venues should not have to look at a submission more than once. Discussion and rebuttals are nice to have, but secondary. Scale is the primary requirement.

Similar comments hold for checklists. If checklists are really necessary, put your money where your mouth is; pay some students market rates to enforce those rules. That will shed light on the real cost of these rules and make it easier to decide if the return on investment is sufficient to justify the rules.

One of the original motivations for ARR involved resubmissions. It was believed that exponential growth was caused by the same paper being submitted again and again. As an Area Chair for ARR, I see little evidence to support this belief. Few papers are tagged as resubmissions, perhaps because resubmissions are rare, or because authors believe that it is in their interest to avoid the resubmission tag. In any case, the resubmission process appears to be yet another well-intentioned step that failed to address the primary requirement: scale. Like rebuttals and discussions, if the return on investment is disappointing, we should discontinue the resubmission process.

More generally, in order to decide which steps to keep investing in and which to discontinue, we need a tracking mechanism to collect metrics on investments and returns by step. The point of this accounting is to make progress on the challenge: too many submissions and too few qualified reviewers. Scale has become a problem because submissions have been growing faster over time than qualified reviewers.

It is often suggested that it is not possible to do a good job because of scale. As discussed in Section 3.2 of Church and Kordoni (Reference Church and Kordoni2022), scale should be a blessing as opposed to a curse. With a larger community, it should be possible to do a better job of matching submissions to reviewers. Ideally, most submissions should be reviewed by someone extremely qualified, familiar with the background material and motivated, typically someone cited in the bibliography or someone nominated under the nomination process proposed below.

7. Reviewing pool

Why do we ask authors of submissions to review? Most of their papers will be rejected. We can find more qualified reviewers than that.

The pool of cited authors is larger (and better informed) than our current pool of reviewers. I would be much happier reviewing submissions that cite my work than what I currently get to review.

We can guess who is likely to be cited well in advance (by sorting authors in the anthology by h-index). If they are not already in the reviewing pool, we could get many of them to opt in by using social media methods. Encourage a friend of theirs (e.g., co-author) to send them an invitation to connect.

When connecting to highly cited experts, the invitation to review should demonstrate appropriate respect for their time and promise the world:

1. 1. Relevant submissions that (almost always) cite the reviewer’s work

2. 2. Task simplification: The task is to distinguish the top 20% from the rest. Abstentions and negative votes do not count. Constructive suggestions that make the conference stronger are welcome, but there is no need for non-constructive negativity. If a paper is not in the top 20%, there is no need to beat a dead horse and pick on silly irrelevant points.

3. 3. A respectful work environment with zero tolerance for toxicity and busy work:

   1. (a) Respect for one anther’s time: Modest loads and reasonable deadlines

   2. (b) One and done: Reviewers will be asked to look at each submission once (and only once)

   3. (c) No nonsense/bloat: No checklists, spam emails, multiple back-and-forth pointless/ineffective discussions with too many additional deadlines.

In short, reviewing for a conference should be easier than reviewing for a journal.

8. Exponential growth

Obviously, scale is a major challenge that needs to be taken seriously as the primary requirement. Many venues are reporting huge numbers of submissions, with alarming increases over time:

• • Like most AI conferences, NeurIPS has seen a substantial increase in the number of submissions in the past few years, growing from 9,467 submissions in 2020 to 21,575 in 2025. Footnote ^k

• • The acceptance results of AAAI 2026 are officially announced! This year, the number of submissions soared to 23,680, and the acceptance rate was only 17.6%. Footnote ^l

Tables 1 and 2 are based on data from openaccept.org.Footnote ^m Table 1 shows the pattern mentioned above holds over many conferences. These conferences are seeing at least twice as many submissions as they used to receive. The bottom row shows even larger increases for ICLR and ICML. The increase for each conference is estimated by dividing the largest value over the smallest value (ignoring missing values). Increases tend to be larger for conferences in Machine Learning and smaller for conferences in Information Retrieval (SIGIR, CIKM).

Table 1.

Submissions have increased considerably over the last decade

Table 1. Long description

A table comparing the number of submissions to various conferences from 2012 to 2026. The table has 15 rows and 17 columns, including column headers and row labels. Column headers include conference names such as ICLR, ICML, ACL, AAAI, NeurIPS, EMNLP, CVPR, NAACL, ICCV, COLING, KDD, IJCAI, SIGIR, and CIKM. Row labels include years from 2012 to 2026 and an ’Increase’ row. Row 1: 2012, NA, NA, NA, NA, NA, NA, NA, NA, NA, 1152, NA, NA, NA, NA. Row 2: 2013, 67, NA, 664, NA, 1420, 772, 1798, 460, 1629, NA, 726, 1473, 366, NA. Row 3: 2014, 87, 1238, 572, 1406, 1678, 790, 1778, NA, NA, 705, 1036, NA, 387, NA. Row 4: 2015, 143, 1037, 692, 1991, 1838, 1300, 1815, 714, 1698, NA, 819, 1996, 351, 646. Row 5: 2016, 265, 1320, 825, 2132, 2406, 1087, 1865, 680, NA, 1039, 784, 2294, 341, 701. Row 6: 2017, 507, 1676, 751, 2590, 3240, 1418, 2620, NA, 2143, NA, 748, 2540, 372, 855. Row 7: 2018, 981, 2473, 1045, 3800, 4856, 2231, 3303, 1089, NA, 888, 983, 3470, 409, 826. Row 8: 2019, 1591, 3424, 1740, 7095, 6743, 2876, 5165, 1733, 4303, NA, 1179, 4752, 426, 1031. Row 9: 2020, 2594, 4990, 2248, 7737, 9467, 3359, 6656, NA, NA, 1862, 1279, 4717, 555, 1175. Row 10: 2021, 2997, 5513, 3350, 7911, 9122, 3600, 7015, 1797, 6236, NA, 1541, 4204, 720, 1251. Row 11: 2022, 3391, 5630, 3378, 9020, 10411, 4190, 8161, 2103, NA, 1563, 1659, 4537, 793, 1175. Row 12: 2023, 4938, 6538, 4864, 8777, 12343, 4909, 9155, NA, 8088, NA, 1416, 4566, 822, 1472. Row 13: 2024, 7304, 9473, 4407, 9862, 15671, 6105, 11532, 2434, NA, 3471, 2046, 5651, 791, 1496. Row 14: 2025, 11565, 12107, 8360, 12957, 21575, 8174, 13008, 3185, 11239, 2089, 2955, 5404, 1071, 1627. Row 15: 2026, 18949, 23918, 12148, 23680, NA, NA, 16092, NA, NA, NA, NA, NA, 1271, NA. Row 16: Increase, 283, 23.1, 21.2, 16.8, 15.2, 10.6, 9.1, 6.9, 6.9, 4.9, 4.1, 3.8, 3.7, 2.5.

Why are there so many submissions? And when did the increase begin? Many potential causes have been suggested: Recent advances in technology? Bots? AI slop? In fact, the exponential growth predates bots by at least a decade, as shown in Figure 1. Bots may be an issue, but they did not cause the exponential growth. Thus, investments in counter-measures (such as bot-catching bots) may be more of a distraction than a solution. The problem is scale, stupid.

Table 2.

Unlike submissions, acceptance rates have not changed much over time

Table 2. Long description

A table comparing acceptance rates for various conferences from 2012 to 2026. The table has 15 rows and 18 columns. The columns are labeled with conference abbreviations: ICLR, ICML, ACL, AAAI, NeurIPS, EMNLP, CVPR, NAACL, ICCV, COLING, KDD, IJCAI, SIGIR, CIKM, and Increase. The rows are labeled with years from 2012 to 2026 and an Increase row. Each cell contains the acceptance rate for the corresponding conference and year. Notable trends include varying acceptance rates across different conferences and years, with some conferences showing increases over time.

I do not know why program committees continue to be surprised by the load. Submissions are increasing exponentially because they always have. It is well known that the academic literature doubles every decade or two (Bornmann and Mutz Reference Bornmann and Mutz2015; Church et al. Reference Church, Chandrasekar, Ortega and Ahmad2025; Larsen and Von Ins Reference Larsen and Von Ins2010; Redner Reference Redner2005). It should be easy to extrapolate this exponential growth over the next decade. Our large conferences may be large today, but historical trends suggest they will become far larger (unless they collapse under their own weight).

Note that both submissions and publications have been growing exponentially with similar time constants. If the time constants were different, then acceptance rates would diverge over time from the current rate of roughly 20% to an extreme value near 0% or near 100%. Since acceptance rates are relatively flat over time, Figure 2 uses box plots to show the data in Table 2, unlike the plot in Figure 1, which calls out the exponential growth over time, with little or no evidence of systematic changes over time.

The bottom row of Table 2 is computed in the same way as the bottom row of Table 1. In both tables, increases are computed by the ratio of the largest and smallest value along each column. Note that acceptance rates are roughly 20%, with perhaps a 2 to 1 ratio between the largest and smallest value (except for IJCAI). Submissions are different; submissions have larger increases, and those increases are clearly associated with time.

There are lessons to be learned by comparing venues. Different venues are different. Some are larger than others, and some are growing faster than others. As suggested above, I believe that SIGIR is coping relatively well with scale. Can other venues learn from their experience?

Figure 1.

The increase in submissions is not particularly recent (on log scales), with exponential growth extending over more than a decade. Data are based on Table 1.

Figure 2.

Acceptance rates vary more by venue than year. Data are based on Table 2.

9. Cheating, negativity, and other non-issues

The focus of the reviewing process should be positive and constructive: how can we improve the quality of the program? There is a natural tendency to focus too much on negativity (like cheating).

There have always been many ways to cheat, many of which are difficult for reviewers to catch such as inflating citations (Ibrahim et al. Reference Ibrahim, Liu, Zaki and Rahwan2024) and plagiarism. In fact, there is surprisingly little cheating. Eventually, such cheating is likely to be caught and penalties can be severe.Footnote ⁿ We do not expect reviewers to catch much cheating, largely because there is so little to catch, but also because reviewers are not in a good position to catch cheating.

Of course, new technology is creating new ways to cheat (and new ways to catch cheating). There are concerns about bots. EMNLP recently announced an attempt to catch bots.Footnote ^o This policy might sound like a good idea, but the story did not end well.

EMNLP deployed a bot-catching bot that looks for fake references and asked area chairs to verify the hits. In my case, 100% of the hits were false positives. I shared this experience on social media and discovered that other area chairs had a similar experience. How many bots would a bot-catching bot catch, if a bot-catching bot could catch bots? According to Google Antigravity,Footnote ^p

Antigravity should have stopped while it was ahead, but it continued with a reference to the barber’s paradox and infinite sets:

As far as I know, EMNLP’s bot-catching bot was a complete failure. After considerable effort by many area chairs, no bots were caught, as far as I know. The failure should have been predictable from the start. The bot-catching bot was matching authors in the most straightforward way imaginable.Footnote ^q If 50% of the authors mismatch, then report a hit. This heuristic might sound reasonable if you do not have much experience with citations in realistic collections like Semantic Scholar (Kinney et al. Reference Kinney, Anastasiades, Authur, Beltagy, Bragg, Buraczynski, Cachola, Candra, Chandrasekhar and Cohan2023). If there had been even minimal testing, it would have become obvious that the prototype was not ready for deployment.

Unfortunately, the prototype was deployed and hits were posted in Open Review. Area chairs were asked (required) to verify the hits under short notice. The user interface was not designed to make this task easy for area chairs, especially for those of us who were traveling at the time.

In my case, one of the false positives complained about a submission that credited a reference to “X and Y.” According to Google Scholar, the authors of the reference were “X et al.” In fact the submission was citing an unpublished tech report. When I looked at the tech report online, the first page credited the work to X et al., revised by Y.

I have seen hallucinated references, but this is not that. This is merely the kind of chaos that exists in the literature. It is often not so easy to say who wrote what. I might disagree with the reference to “X and Y,” but it is not appropriate to escalate this relatively minor issue to an area chair, especially under time pressure. This issue was neither important nor urgent.

Technology may not be the best way to deal with bots (and cheating). Penalties are more effective. There was a famous case where a lawyer was caught using a bot to cheat, and not surprisingly, the judge was not amused.Footnote ^r Unfortunately, bots are continuing to frustrate judges.Footnote ^s IMHO, public shaming (and severe penalties) are more effective deterrents than a bot-catching bot that wastes time and produces disappointing returns on investments.

Bottom Line: We should all test our code before we release. You (and your org) are responsible for what you release. In this case, the responsible organizations are ACL, ARR, and EMNLP. Unfortunately, instead of apologizing, the responsible organizations started finger pointing, making it all too clear what is wrong with too many organizations with overlapping responsibilities.

The bot-catching bot experience was disrespectful of area chairs in a number of ways:

1. 1. The short notice was unnecessary.

2. 2. Lack of testing (or even plausibility).

3. 3. Yet another bloated task to address yet another non-issue (the prior on bots is low because there are easier ways to cheat).

4. 4. Much ado about nothing (little return on investment, unless one does not value the area chairs very much).

5. 5. We all hate AI slop, but we especially hate it when the AI slop is coming from the program committee.

6. 6. Vibe coding makes it way too easy to deploy crap. We are all responsible for what we ship.

7. 7. After the failed deployment, there should have been an apology.

8. 8. Attempts to catch cheating are a distraction from what should be our primary focus: scale.

10. Constructive suggestion: More effective use of area chairs

The promise to area chairs should be similar to the promise to reviewers:

1. 1. Focus on just two high-value tasks that take advantage of their expertise:

   1. (a) Routing submissions to reviewers, and

   2. (b) Deciding results for a relatively small number of submissions in gray area.

2. 2. A respectful work environment with zero tolerance for toxicity and busy work:

   1. (a) Respectful of one anther’s time: Modest loads and reasonable deadlines.

   2. (b) No nonsense/bloat: No checklists, spam emails, multiple back-and-forth pointless/ineffective discussions with too many additional deadlines.

Use area chairs to route papers to reviewers. That is a better use if their limited time than summarizing results. It is pointless to ask area chairs to summarize reviews from ill-informed, unqualified, and unmotivated reviewers. It is a better use of their time and expertise to become more involved in routing papers. Since area chairs cannot be expected to know everyone and everything in a large community, we should encourage area chairs to specialize in subtopics. Program committees used to be organized by subcommittees on subtopics. We should re-introduce that tradition, as well as mechanisms those venues used to use to route papers to the appropriate subcommittee.

As for summarizing results, area chairs should focus meta reviews on gray area. If reviewers agree, the meta review is adding little value.

Do not bother the area chair with clear rejects and other simple tasks that are below their pay grade. Do not ask area chair to pester late reviewers or find emergency reviews. Send papers to four reviewers and make decisions with the most useful two.

Ac-cent-tchu-ate the positive:Footnote ^t

Avoid averaging grades. Controversial papers are better for the future of the conference than papers that put the audience to sleep. A controversial paper with an advocate and a second is better than an uncontroversial paper with no objections and no advocates.

Positive votes count; abstentions, negative reviews, content-free reviews, and missing reviews do not count, unless they identify a serious flaw that others missed like plagiarism.

Above all, treat volunteers with respect.

11. A constructive suggestion: Nomination round

As mentioned above, the main challenge is scale. There are (a) too many submissions and (b) not enough qualified reviewers. The proposed nomination round is intended to address both (a) and (b).

To qualify for review, a submission needs to be nominated. Qualified reviewers (with h-index of five or more) are encouraged to nominate strong papers for review. Nominations should be used sparingly for strong papers that are truly deserving of the honor. Nominators are expected to review four papers per nomination.

It is understood that nominators are likely to be biased. The point of the nomination round is to filter out submissions that would benefit from more time, as well as AI slop, papers off-topic, and desk rejections. If insiders are not supportive, there is no point asking for more (unbiased) opinions.

I prefer to refer to this process positively as nominations as opposed to desk rejections. The purpose of nominations is to encourage insiders to prioritize limited resources. Nominators are expected to have read the submission and to be willing to stake their reputation on the quality of the work. If a professor is not happy with the work (or if he does not have the time to do his share of reviewing), he is in a better position to set priorities than less-informed outsiders.

When I started at Bell Labs, we were expected to submit papers to an internal review process before we could submit them to conferences and journals. We were told the purpose was to protect the company’s intellectual property, but the real purpose was to discourage junior researchers from publishing too soon (and too much). When researchers start out, they keep hearing “publish or perish,” but actually, that is not exactly right. At the end of the day, reputations hinge on quality as well as quantity.

The proposed nomination process is designed to help professors and managers in industry do what my mentors did at Bell Labs. That is, the nomination process provides an easy excuse for advising a student to take more time when that is appropriate. Since it can be hard for a professor to tell a student what needs to be said, the nomination process provides a convenient excuse. If a professor finds it too difficult to tell a student to take more time, the professor can easily say he is too busy to do the reviews.

The proposed nomination process is also designed to help area chairs select appropriate reviewers. Nominators are encouraged to provide a list of suggested reviewers.

For more discussion of nominations, see section 4.4 of Church et al. (Reference Church, Chandrasekar, Ortega and Ahmad2025).