Crossref Citations
This article has been cited by the following publications. This list is generated based on data provided by
Crossref.
Langsford, Steven
Perfors, Amy
Hendrickson, Andrew T.
Kennedy, Lauren A.
and
Navarro, Danielle J.
2018.
Quantifying sentence acceptability measures: Reliability, bias, and variability.
Glossa: a journal of general linguistics,
Vol. 3,
Issue. 1,
Warstadt, Alex
Singh, Amanpreet
and
Bowman, Samuel R.
2019.
Neural Network Acceptability Judgments.
Transactions of the Association for Computational Linguistics,
Vol. 7,
Issue. ,
p.
625.
Sequeros-Valle, Jose
Hoot, Bradley
and
Cabrelli, Jennifer
2020.
Clitic-Doubled Left Dislocation in Heritage Spanish: Judgment versus Production Data.
Languages,
Vol. 5,
Issue. 4,
p.
47.
Murphy, Elliot
2020.
Language design and communicative competence: The minimalist perspective.
Glossa: a journal of general linguistics,
Vol. 5,
Issue. 1,
Jamieson, E
2020.
Viewing dialect change through acceptability judgments: A case study in Shetland dialect.
Glossa: a journal of general linguistics,
Vol. 5,
Issue. 1,
Abeillé, Anne
and
Winckel, Elodie
2020.
French subject island? Empirical studies ofdontandde qui.
Journal of French Language Studies,
Vol. 30,
Issue. 3,
p.
275.
Cheng, Lisa Lai‐Shen
and
Griffiths, James
2021.
A Companion to Chomsky.
p.
52.
Harris, Jesse
2021.
The Cambridge Handbook of Experimental Syntax.
p.
341.
Miglio, Viola G.
and
Gries, Stefan Th.
2021.
Acceptability of Different Psychological Verbal Constructions by Heritage Spanish Speakers from California.
Languages,
Vol. 6,
Issue. 2,
p.
80.
Cova, Florian
Strickland, Brent
Abatista, Angela
Allard, Aurélien
Andow, James
Attie, Mario
Beebe, James
Berniūnas, Renatas
Boudesseul, Jordane
Colombo, Matteo
Cushman, Fiery
Diaz, Rodrigo
N’Djaye Nikolai van Dongen, Noah
Dranseika, Vilius
Earp, Brian D.
Torres, Antonio Gaitán
Hannikainen, Ivar
Hernández-Conde, José V.
Hu, Wenjia
Jaquet, François
Khalifa, Kareem
Kim, Hanna
Kneer, Markus
Knobe, Joshua
Kurthy, Miklos
Lantian, Anthony
Liao, Shen-yi
Machery, Edouard
Moerenhout, Tania
Mott, Christian
Phelan, Mark
Phillips, Jonathan
Rambharose, Navin
Reuter, Kevin
Romero, Felipe
Sousa, Paulo
Sprenger, Jan
Thalabard, Emile
Tobia, Kevin
Viciana, Hugo
Wilkenfeld, Daniel
and
Zhou, Xiang
2021.
Estimating the Reproducibility of Experimental Philosophy.
Review of Philosophy and Psychology,
Vol. 12,
Issue. 1,
p.
9.
Redl, Theresa
Frank, Stefan L.
de Swart, Peter
de Hoop, Helen
and
Felser, Claudia
2021.
The male bias of a generically-intended masculine pronoun: Evidence from eye-tracking and sentence evaluation.
PLOS ONE,
Vol. 16,
Issue. 4,
p.
e0249309.
Morgan, Adam M.
and
Ferreira, Victor S.
2022.
Still no evidence for audience design in syntax: Resumptive pronouns are not the exception.
Journal of Memory and Language,
Vol. 127,
Issue. ,
p.
104368.
Dentella, Vittoria
Günther, Fritz
and
Leivada, Evelina
2023.
Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias.
Proceedings of the National Academy of Sciences,
Vol. 120,
Issue. 51,
Christensen, Ken Ramshøj
and
Nyvad, Anne Mette
2024.
Complexity, frequency, and acceptability.
Glossa: a journal of general linguistics,
Vol. 9,
Issue. 1,
Murphy, Elliot
Holmes, Emma
and
Friston, Karl
2024.
Natural language syntax complies with the free-energy principle.
Synthese,
Vol. 203,
Issue. 5,
Callesano, Salvatore
Delgado, Rodrigo
Ma, Wenyue
Trybulec, Erin
and
Reyes, Gabriella
2025.
Contextualizing code-switching: testing the effects of reggaetón labels and listening frequency in an Acceptability Judgment Task.
Frontiers in Communication,
Vol. 9,
Issue. ,
Wiltschko, Martina
2025.
The grammar of self-talk. What different modes of talking reveal about language.
Theoretical Linguistics,
Vol. 51,
Issue. 3-4,
p.
167.
Dentella, Vittoria
Günther, Fritz
Leivada, Evelina
and
Rehbein, Malte
2025.
Language in vivo vs. in silico: Size matters but Larger Language Models still do not comprehend language on a par with humans due to impenetrable semantic reference.
PLOS One,
Vol. 20,
Issue. 7,
p.
e0327794.
Colina Fortuny, Laia
D'Alessandro, Roberta
and
Parafita Couto, M. Carmen
2026.
Collecting data in understudied language varieties: A methodological note.
Nordlyd,
Vol. 49,
Issue. 1,
p.
3.
Griffiths, James
2026.
An experimental syntax approach to British English do ellipsis.
Natural Language & Linguistic Theory,
Vol. 44,
Issue. 1,
Branigan & Pickering (B&P) advocate the use of syntactic priming to investigate linguistic representations. We support the use of any data types that scientists find relevant for specific research questions, including syntactic priming. We regret, then, that B&P appear to repeat unsubstantiated claims that paint a relatively misleading picture of acceptability judgments (AJs), a data type that linguists have been using fruitfully for decades. From our perspective, much of the literature criticizing AJs has repeatedly focused on logically possible concerns about their use without investigating whether those concerns are empirically attested. This risks a vicious circle: Articles can cite each other for support, giving the illusion of empirical support. In this commentary, we highlight a number of studies that have pursued this issue head on, which we leverage to examine six of B&P's claims about AJs in detail.
Claim 1: Linguists standardly ask a single informant about the acceptability of a few sentences (sect. 1.2, para. 2)
Claim 1 is a caricature of linguistic methodology that, to our knowledge, has never been supported by evidence. Nonetheless, a charitable interpretation of this claim reveals two separate concerns: (1) the routine use of small sample sizes, and (2) the susceptibility of AJs to investigator bias (Claim 2, below). An obvious consequence of using small samples sizes in research is an increase in errors (probably of all four types identified by Gelman & Carlin Reference Gelman and Carlin2014: I, II, Sign, and Magnitude). By performing a large-scale comparison of the published results in linguistics with retests of those results using large samples of naïve participants, one can evaluate the quality of their convergence rate. This cannot identify specific errors, but it can tell us whether the differences between methods actually produce different results.
Sprouse and Almeida (Reference Sprouse and Almeida2012) tested every English data point from a popular syntax textbook (Adger Reference Adger2003) using large samples of naïve participants. Out of 365 phenomena, they conservatively estimate a minimum convergence rate of 98%. Sprouse et al. (Reference Sprouse, Schütze and Almeida2013) randomly sampled 148 phenomena from a leading linguistics journal (Linguistic Inquiry), and conservatively estimate a convergence rate of 95% ( ± 5% because of the random sampling). These high (conservative) convergence rates suggest that the sample sizes used by linguists (whatever they are) historically have introduced little error to the empirical record for any combination of the following reasons: (1) the samples are larger than what critics claim; (2) the effect sizes are so large that small samples still yield good statistical power; or (3) AJ results are highly replicated before and after publication (e.g., Phillips Reference Phillips, Iwasaki, Hoji, Clancy and Sohn2009).
Claim 2: Acceptability judgments are highly susceptible to theoretical cognitive bias because linguists tend to use professional linguists as participants (sect. 1.2, para. 3)
This can also be addressed by the studies discussed above. Cognitive bias should predict sign reversals between naïve and expert populations. Sprouse and Almeida (Reference Sprouse and Almeida2012) found no sign reversals from the textbook data. Sprouse et al. (Reference Sprouse, Schütze and Almeida2013) reported a 1–3% sign reversal rate in the journal data. Mahowald et al. (Reference Mahowald, Graff, Hartman and Gibson2016a) and Häussler et al. (Reference Häussler, Juzek and Wasow2016) have replicated the latter without reporting an increased sign reversal rate (0–6%). Comparisons of naïve and expert populations also were conducted by Culbertson and Gross (Reference Culbertson and Gross2009), who report high inter- and intra-group correlations on 73 sentence types, and by Dąbrowska (Reference Dąbrowska2010). The latter found that, while experts gave less variable ratings than naïve participants on several sentence types, experts rated certain theoretically interesting syntactic violations as more acceptable than naïve participants, in apparent conflict with their theoretical commitments. Taken together, these results are not what one would expect if AJs were highly susceptible to cognitive bias.
Claim 3: Acceptability judgments are susceptible to differences in instructions (sect. 1.2, para. 3)
Claim 3 has been directly investigated by Cowart (Reference Cowart1997), who reports that the systematic manipulation of instructions does not change the pattern of acceptability judgments for factorial designs.
Claim 4: Acceptability judgments are impacted by sentence processing effects (sect. 1.2, para. 5)
Claim 4 is technically true, but B&P exaggerate its consequences. First, many classic lexical and sentence processing effects have relatively small or negligible effects on acceptability (e.g., Featherston Reference Featherston2009; Phillips Reference Phillips, Iwasaki, Hoji, Clancy and Sohn2009; Sprouse Reference Sprouse2008; Sprouse et al. Reference Sprouse, Wagers and Phillips2012). Second, very few syntactic phenomena have been proposed to be fully reducible to sentence processing effects. The lone exceptions to this appear to be constraints on long-distance dependencies (e.g., Kluender & Kutas Reference Kluender and Kutas1993; Hofmeister & Sag Reference Hofmeister and Sag2010), but in that case, a number of experimental studies have disproven the reductionist predictions (Phillips Reference Phillips2006; Sprouse et al. Reference Sprouse, Wagers and Phillips2012; Yoshida et al. Reference Yoshida, Kazanina, Pablos and Sturt2014). Thus, to the extent that AJs are impacted by sentence processing, it appears as though the effects can be dealt with like any other source of noise in an experimental setting.
Claim 5: Acceptability judgments reveal only set membership (sect. 1.2, para. 7)
Claim 5 is confusing. It is false in the sense that, if one is interested in set membership, this property still needs to be inferred from acceptability data, using a logic that maps that data type back to the relevant cognitive computations. In this, AJs are like any other data type in cognitive science: No data types, including priming, directly reveal the underlying computations of the human brain, and all data types require a linking hypothesis between the observable data and the unobservable cognitive process.
Claim 6: Acceptability judgments have yielded no consensus theory among linguists (sect. 1.2, para. 9)
Claim 6 is a strange criticism to make of any data type, especially AJs. First, the beliefs of scientists are a subjective issue based on how they weigh different kinds of evidence. Second, AJs are, by all accounts, a robust and replicable data type. Whatever disagreements there are in linguistics literature, they appear to obtain mostly at the level of interpreting, not establishing, the data (e.g., Phillips Reference Phillips, Iwasaki, Hoji, Clancy and Sohn2009).
In conclusion, we support B&P's desire to bring new evidence to bear on questions about linguistic representation. We caution, however, that advocacy for one method should not be bolstered by misleading comparisons, especially with methods such as AJs, which yield data that are demonstrably robust, highly replicable, and comparatively convenient and inexpensive to collect.