Hostname: page-component-7b7567c8c-lpg2g Total loading time: 0 Render date: 2026-11-15T12:02:07.253Z Has data issue: false hasContentIssue false
  • English
  • Français

Randomization Tests and Multi-Level Data in U.S. State Politics

Published online by Cambridge University Press:  25 January 2021

Robert S. Erikson,
Pablo M. Pinto and
Kelly T. Rader
Robert S. Erikson
Affiliation:
Columbia University
Pablo M. Pinto
Affiliation:
Columbia University
Kelly T. Rader
Affiliation:
Columbia University
Get accessRights & Permissions [Opens in a new window]

Abstract

Many hypotheses in U.S. state politics research are multi-level, positing that state-level variables affect individual-level behavior. Unadjusted standard errors for state-level variables are too small, leading to overconfidence and possible false rejection of null hypotheses. Primo, Jacobsmeier, and Milyo (2007) explore this problem in their reanalysis of Wolfinger, Highton, and Mullin's (2005) data on the effects of post-registration laws on voter turnout. Primo et al. advocate the use of clustered standard errors to solve the overconfidence problem, but we offer an alternative solution: randomization tests. Randomization tests are non-parametric tests that do not rely on comparisons to theoretical test statistic distributions. Instead, they use distributions tailored to the data, created by randomly scrambling the data many times to simulate what would be observed under the null hypothesis. Unlike with clustering, with the randomization test, U.S. state-level reforms generally fail to be significant both as additive effects and as interactions with individual characteristics.

Information

Type
Research Article
Information
State Politics & Policy Quarterly , Volume 10 , Issue 2 , Summer 2010, pp. 180 - 198
Copyright
Copyright © 2010 by the Board of Trustees of the University of Illinois

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Ai, Chunrong, and Norton, Edward C.. 2003. “Interaction Terms in Logit and Probit Models.” Economics Letters 80:123–9.CrossRefGoogle Scholar
Angrist, Joshua D., and Pischke, Jorn-Steffen. 2009. Mostly Harmless Econometrics: An Empiricist's Companion. Princeton, NJ: Princeton University Press.CrossRefGoogle Scholar
Arceneaux, Kevin, and Nickerson, David W.. 2009. “Modeling Certainty with Clustered Data: A Comparison of Methods.” Political Analysis 17:177–90.CrossRefGoogle Scholar
Donald, Stephen G., and Lang, Kevin. 2007. “Inference with Difference-in-Differences and Other Panel Data.” Review of Economics and Statistics 89:221–33.CrossRefGoogle Scholar
Donohue, John J., and Wolfers, Justin. 2006. “Uses and Abuses of Empirical Evidence in the Death Penalty Debate.” Stanford Law Review 58:791–835.Google Scholar
Edgington, Eugene S., and Onghena, Patrick. 2007. Randomization Tests. 4th ed. Boca Raton, FL: Taylor and Francis Group.CrossRefGoogle Scholar
Fisher, R.A. 1935. The Design of Experiments. Edinburgh, UK: Oliver and Boyd.Google Scholar
Griffin, John D., and Keane, Michael. 2006. “Descriptive Representation and the Composition of African-American Turnout.” American Journal of Political Science 50:998–1012.CrossRefGoogle Scholar
Helland, Eric, and Tabarrok, Alexander. 2004. “Using Placebo Laws to Test ‘More Guns, Less Crime.‘” Advances in Economic Analysis and Policy 4:1–7.CrossRefGoogle Scholar
Hogan, Robert E. 2005. “Gubernatorial Coattail Effects in State Legislative Elections.” Political Research Quarterly 58:587–97.CrossRefGoogle Scholar
Kam, Cindy J., and Franzese, Robert J. Jr. 2007. Modeling and Interpreting Interactive Hypotheses in Regression Analysis. Ann Arbor, MI: University of Michigan Press.Google Scholar
Keele, Luke, McConnaughy, Corrine, and White, Ismail. 2008. “Statistical Inference for Experiments.” Unpublished paper. Available at www.polisci.ohio-state.edu/faculty/lkeele/randtests.pdf (January 4, 2010).Google Scholar
Kennedy, Peter E. 1995. “Randomization Tests in Econometrics.” Journal of Business and Economic Statistics 13:85–94.Google Scholar
Kennedy, Peter E., and Cade, Brian S.. 1996. “Randomization Tests for Multiple Regression.” Communications in Statistics—Simulation and Computation 25:923–9.CrossRefGoogle Scholar
Lawless, Jennifer L. 2004. “Politics of Preference? Congresswomen and Symbolic Representation.” Political Research Quarterly 57:81–99.CrossRefGoogle Scholar
Leoni, Eduardo L. 2009. “Analyzing Multiple Surveys: Results from Monte Carlo Experiments.” Unpublished paper. Available at http://eduardoleoni.com/workingpapers/multilevel.pdf (January 4, 2010).Google Scholar
Manly, Bryan F. J. 1997. Randomization, Bootstrap, and Monte Carlo Methods in Biology. 2nd ed. London, UK: Chapman Hall.Google Scholar
Moore, David S., McCabe, George P., Duckworth, William M., and Sclove, Stanley L.. 2003. The Practice of Business Statistics Companion Chapter 18: Bootstrap Methods and Permutation Tests. New York, NY: W.H. Freeman.Google Scholar
Moulton, Brent R. 1986. “Random Group Effects and the Precision of Regression Estimates.” Journal of Econometrics 32:385–97.CrossRefGoogle Scholar
Moulton, Brent R. 1990. “An Illustration of a Pitfall in Estimating the Effects of Aggregate Variables in Micro Units.” Review of Economics and Statistics 72:334–8.CrossRefGoogle Scholar
O'Gorman, Thomas W. 2005. “The Performance of Randomization Tests that Use Permutations of Independent Variables.” Communications in Statistics—Simulation and Computation 34:895–908.CrossRefGoogle Scholar
Primo, David M., Jacobsmeier, Matthew I., and Milyo, Jeffrey. 2007. “Estimating the Impact of State Policies and Institutions with Mixed-Level Data.” State Politics and Policy Quarterly 7:446–59.CrossRefGoogle Scholar
Soss, Joe, Langbein, Laura, and Metelko, Alan R.. 2003. “Why Do White Americans Support the Death Penalty?” The Journal of Politics 65:397–421.CrossRefGoogle Scholar
Wolfinger, Raymond E., Highton, Benjamin, and Mullin, Megan. 2005. “How Post-registration Laws Affect the Turnout of Citizens Registered to Vote.” State Politics and Policy Quarterly 5:1–23.CrossRefGoogle Scholar
Wolfinger, Raymond E., and Rosenstone, Steven J.. 1980. Who Votes? New Haven, CT: Yale University Press.Google Scholar