Hostname: page-component-7b7567c8c-7pwfn Total loading time: 0 Render date: 2026-11-12T22:24:52.956Z Has data issue: false hasContentIssue false
  • English
  • Français

Running online experiments using web-conferencing software

Published online by Cambridge University Press:  17 January 2025

Jiawei LiOpen the ORCID record for Jiawei Li[Opens in a new window],
Stephen Leider,
Damian Beil and
Izak Duenyas
Jiawei Li*
Affiliation:
The University of Michigan, Ann Arbor, USA
Stephen Leider*
Affiliation:
The University of Michigan, Ann Arbor, USA
Damian Beil*
Affiliation:
The University of Michigan, Ann Arbor, USA
Izak Duenyas*
Affiliation:
The University of Michigan, Ann Arbor, USA
*
e-mail: jiawli@umich.edu
leider@umich.edu
dbeil@umich.edut
duenyas@umich.edu
Get accessRights & Permissions [Opens in a new window]

Abstract

We report the results of a novel protocol for running online experiments using a combination of an online experimental platform in parallel with web-conferencing software in two formats—with and without subject webcams—to improve subjects’ attention and engagement. We compare the results between our online sessions with the offline (lab) sessions of the same experiment. We find that both online formats lead to comparable subject characteristics and performance as the offline (lab) experiment. However, the webcam-on protocol has less noisy data, and hence better statistical power, than the protocol without a webcam. The webcam-on protocol can detect reasonable effect sizes with a comparable sample size as in the offline (lab) protocol.

Keywords

Online experimentExperimental methodologyZTREE unleashedWeb-conferencing softwareWebcamSelection biasSubject performance

JEL classification

C91: Laboratory, Individual BehaviorC90: GeneralD91: Role and Effects of Psychological, Emotional, Social, and Cognitive Factors on Decision Making‡

Information

Type
Original Paper
Information
Journal of the Economic Science Association , Volume 7 , Issue 2: Special Issue: Experiments in the time of COVID 19 – Challenges and Insights (Pages 103-209) , December 2021, pp. 167 - 183
Copyright
Copyright © 2021 The Author(s), under exclusive licence to Economic Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Arechar, A. A., Gächter, S., Molleman, L. (2018). Conducting interactive experiments online. Experimental Economics, 21(1), 99–131. 10.1007/s10683-017-9527-2Google ScholarPubMed
Arechar, A. A., Rand, D. G. (2021). Turking in the time of covid. Behavior Research Methods, 53, 2591–2595. 10.3758/s13428-021-01588-4Google ScholarPubMed
Carpenter, J., Graham, M., Wolf, J. (2013). Cognitive ability and strategic sophistication. Games and Economic Behavior, 80, 115–130. 10.1016/j.geb.2013.02.012Google Scholar
Chandler, J., Mueller, P., Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112–130. 10.3758/s13428-013-0365-7Google ScholarPubMed
Croson, R., Gneezy, U. (2009). Gender differences in preferences. Journal of Economic Literature, 47(2), 448–474. 10.1257/jel.47.2.448Google Scholar
Duch, M. L., Grossmann, M. R., Lauer, T. (2020). z-Tree unleashed: A novel client-integrating architecture for conducting z-Tree experiments over the Internet. Journal of Behavioral and Experimental Finance, 28, 100400. 10.1016/j.jbef.2020.100400Google Scholar
Feltz, C. J., Miller, G. E. (1996). An asymptotic test for the equality of coefficients of variation from k populations. Statistics in Medicine, 15(6), 647–658. 10.1002/(SICI)1097-0258(19960330)15:6<647::AID-SIM184>3.0.CO;2-PGoogle ScholarPubMed
Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4), 25–42. 10.1257/089533005775196732Google Scholar
Gneezy, U., Rustichini, A., Vostroknutov, A. (2010). Experience and insight in the race game. Journal of Economic Behavior & Organization, 75(2), 144–155. 10.1016/j.jebo.2010.04.005Google Scholar
Greiner, B. (2015). Subject pool recruitment procedures: Organizing experiments with ORSEE. Journal of the Economic Science Association, 1(1), 114–125. 10.1007/s40881-015-0004-4Google Scholar
Hauser, D., Paolacci, G., & Chandler, J. (2019). Common concerns with MTurk as a participant pool: Evidence and solutions. Handbook of Research Methods in Consumer Psychology, pages 319–337.Google Scholar
Holt, C. A., Laury, S. K. (2002). Risk aversion and incentive effects. American Economic Review, 92(5), 1644–1655. 10.1257/000282802762024700Google Scholar
Horton, J. J., Rand, D. G., Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425. 10.1007/s10683-011-9273-9Google Scholar
Howell, D. C. (2012). Statistical Methods for Psychology, Cengage Learning.Google Scholar
Kleinman, K. & Huang, S. S. (2016). Calculating power by bootstrap, with an application to cluster-randomized trials. EGEMs, 4(1).Google ScholarPubMed
Kovalchik, S., Camerer, C. F., Grether, D. M., Plott, C. R., Allman, J. M. (2005). Aging and decision making: A comparison between neurologically healthy elderly and young individuals. Journal of Economic Behavior & Organization, 58(1), 79–94. 10.1016/j.jebo.2003.12.001Google Scholar
Paolacci, G., Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188. 10.1177/0963721414531598Google Scholar
Peng, X., Peng, G., & Gonzales, C. (2005). Power analysis and sample size estimation using bootstrap. Phoenix: Paper presented at PharmaSUG.Google Scholar
Zhao, S., Vargas, K., Friedman, D., & Chavez, M. (2020). UCSC LEEPS lab protocol for online economics experiments. SSRN Electronic Journal.Google Scholar
Zhou, H., Fishbach, A. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. Journal of Personality and Social Psychology, 111(4),493. 10.1037/pspa0000056Google ScholarPubMed
Supplementary material: File

Li et al. supplementary material

Li et al. supplementary material
Download Li et al. supplementary material(File)
File 524.3 KB