Hostname: page-component-7b7567c8c-7pwfn Total loading time: 0 Render date: 2026-11-14T00:44:28.168Z Has data issue: false hasContentIssue false
  • English
  • Français

The Wells-Du Bois Protocol for Machine Learning Bias: Building Critical Quantitative Foundations for Third Sector Scholarship

Published online by Cambridge University Press:  01 January 2026

Thema Monroe-WhiteOpen the ORCID record for Thema Monroe-White[Opens in a new window] and
Jesse Lecy
Thema Monroe-White*
Affiliation:
Department of Technology, Entrepreneurship, and Data Analytics, Campbell School of Business, Berry College, 2277 Martha Berry Highway, Mount Berry, GA 30149-5024, USA
Jesse Lecy*
Affiliation:
Arizona State University, Tempe, USA
*
tmonroewhite@berry.edu
jdlecy@asu.edu
Get accessRights & Permissions [Opens in a new window]

Abstract

This paper charts the rapid rise of data science methodologies in manuscripts published in top journals for third sector scholarship, indicating their growing importance to research in the field. We draw on critical quantitative theory (QuantCrit) to challenge the assumed neutrality of data science insights that are especially prone to misrepresentation and unbalanced treatment of sub-groups (i.e., those marginalized and minoritized because of their race, gender, etc.). We summarize a set of challenges that result in biases within machine learning methods that are increasingly deployed in scientific inquiry. As a means of proactively addressing these concerns, we introduce the “Wells-Du Bois Protocol,” a tool that scholars can use to determine if their research achieves a baseline level of bias mitigation. Ultimately, this work aims to facilitate the diffusion of key insights from the field of QuantCrit by showing how new computational methodologies can be improved by coupling quantitative work with humanistic and reflexive approaches to inquiry. The protocol ultimately aims to help safeguard third sector scholarship from systematic biases that can be introduced through the adoption of machine learning methods.

Keywords

Algorithmic biasCritical quantitative methodsMachine learningData scienceThird sector

Information

Type
Research Papers
Information
Voluntas: International Journal of Voluntary and Nonprofit Organizations , Volume 34 , Issue 1 , February 2023, pp. 170 - 184
Copyright
Copyright © International Society for Third-Sector Research 2022

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Agarwal, P. K. (2018). Public administration challenges in the world of AI and Bots. Public Administration Review, 78(6), 917–921. doi: 10.1111/puar.12979.CrossRefGoogle Scholar
Allard, S. W., Wiegand, E. R., Schlecht, C., Datta, A. R., Goerge, R. M., & Weigensberg, E. (2018). State agencies’ use of administrative data for improved practice: Needs, challenges, and opportunities. Public Administration Review, 78(2), 240–250. doi: 10.1111/puar.12883.CrossRefGoogle Scholar
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias. ProPublica, May, 23 (2016), 139–159.Google Scholar
Ayodele, T. O. (2010). Machine learning overview. New Advances in Machine Learning, 2.Google Scholar
Baker, M. (2016). Reproducibility crisis. Nature, 533(26), 353–366.Google Scholar
Banerjee, I., Bhimireddy, A. R., Burns, J. L., Celi, L. A., Chen, L. C., Correa, R., & Gichoya, J. W. (2021). Reading race: AI recognises patient's racial identity in medical images. arXiv preprint https://arxiv.org/abs/2107.10356.Google Scholar
Bansal, A., Agarwal, R., & Sharma, R. K. (2012, November). SVM based gender classification using iris images. In: 2012 fourth international conference on computational intelligence and communication networks (pp. 425–429). IEEE.10.1109/CICN.2012.192CrossRefGoogle Scholar
Battaglio, R. P., & Hall, J. L. (2018). Trinity is still my name: Renewed appreciation for triangulation and methodological diversity in public administration. Public Administration Review, 78(6), 825–827. doi: 10.1111/puar.13010.CrossRefGoogle Scholar
Beck, A. J. (2018). Race and ethnicity of violent crime offenders and arrestees, 2018. Bureau of Justice Statistics.Google Scholar
Bozeman, B. (2002). Public-value failure: When efficient markets may not do. Public Administration Review, 62(2), 145–161. doi: 10.1111/0033-3352.00165.CrossRefGoogle Scholar
Buolamwini, J., & Gebru, T. (2018, January). Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Conference on fairness, accountability and transparency (pp. 77–91).Google Scholar
Carnochan, S., Samples, M., Myers, M., & Austin, M. J. (2014). Performance measurement challenges in nonprofit human service organizations. Nonprofit and Voluntary Sector Quarterly, 43(6), 1014–1032. doi: 10.1177/0899764013508009.CrossRefGoogle Scholar
Chang, R. M., Kauffman, R. J., & Kwon, Y. (2014). Understanding the paradigm shift to computational social science in the presence of big data. Decision Support Systems, 63, 67–80. doi: 10.1016/j.dss.2013.08.008.CrossRefGoogle Scholar
Chen, Y. W., & Nakazawa, M. (2017). Emotions and Pan-Asian organizing in the US Southwest: Analyzing interview discourses via sentiment analysis. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 28(6), 2785–2806.CrossRefGoogle Scholar
Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163. doi: 10.1089/big.2016.0047.CrossRefGoogle ScholarPubMed
Christian, B. (2020). The alignment problem: Machine learning and human values. WW Norton & Company.Google Scholar
Conte, R., Gilbert, N., Bonelli, G., Cioffi-Revilla, C., Deffuant, G., Kertesz, J., & Nowak, A. (2012). Manifesto of computational social science. The European Physical Journal Special Topics, 214(1), 325–346. doi: 10.1140/epjst/e2012-01697-8.CrossRefGoogle Scholar
Corry, O. (2010). Defining and theorizing the third sector. In Third sector research (pp. 11–20). Springer, New York, NY.10.1007/978-1-4419-5707-8_2CrossRefGoogle Scholar
Davenport, T. H., & Patil, D. J. (2012). Data scientist: The sexiest job of the 21st century. Harvard Business Review, 90(5), 70–76.Google Scholar
Dhar, V. (2013). Data science and prediction. Communications of the ACM, 56(12), 64–73. doi: 10.1145/2500499.CrossRefGoogle Scholar
D'Ignazio, C., & Klein, L. F. (2020). Data feminism. MIT Press doi: 10.7551/mitpress/11805.001.0001.CrossRefGoogle Scholar
Dressel, J., & Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1), eaao5580.10.1126/sciadv.aao5580CrossRefGoogle ScholarPubMed
Du Bois, W. E. B. (1900). African American photographs assembled for 1900 Paris Exposition. Library of Congress,Washington, DC. Retrieved from http://www.loc.gov/pictures/collection/anedub/dubois.html.Google Scholar
Ellegaard, O., & Wallin, J. A. (2015). The bibliometric analysis of scholarly production: How great is the impact?. Scientometrics, 105(3), 1809–1831. doi: 10.1007/s11192-015-1645-z.CrossRefGoogle ScholarPubMed
Faircloth, S. C., Alcantar, C. M., & Stage, F. K. (2015). Use of large-scale data sets to study educational pathways of American Indian and Alaska Native students. New Directions for Institutional Research, 2014(163), 5–24. doi: 10.1002/ir.20083.CrossRefGoogle Scholar
Fyall, R., Moore, M. K., & Gugerty, M. K. (2018). Beyond NTEE codes: Opportunities to understand nonprofit activity through mission statement content coding. Nonprofit and Voluntary Sector Quarterly, 47(4), 677–701. doi: 10.1177/0899764018768019.CrossRefGoogle Scholar
Garcia, N. M., Lopez, N., & Velez, V. N. (2018). QuantCrit: Rectifying quantitative methods through critical race theory. Race Ethnicity and Education, 21(2), 149–157. doi: 10.1080/13613324.2017.1377675.CrossRefGoogle Scholar
Gawande, A. (2009). The checklist manifesto: How to Get things right. Metropolitan Books, New York.Google Scholar
Gawande, A. (2011). The checklist manifesto: How to get things right. Journal of Nursing Regulation, 1(4), 64. doi: 10.1016/S2155-8256(15)30310-0.CrossRefGoogle Scholar
Gawande, A. A., Kwaan, M. R., Regenbogen, S. E., Lipsitz, S. A., & Zinner, M. J. (2007). An Apgar score for surgery. Journal of the American College of Surgeons, 204(2), 201–208. doi: 10.1016/j.jamcollsurg.2006.11.011.CrossRefGoogle ScholarPubMed
Gillborn, D., Warmington, P., & Demack, S. (2018). QuantCrit: Education, policy, Big Data and principles for a critical race theory of statistics. Race Ethnicity and Education, 21(2), 158–179. doi: 10.1080/13613324.2017.1377417.CrossRefGoogle Scholar
Guo, C., & Saxton, G. D. (2018). Speaking and being heard: How nonprofit advocacy organizations gain attention on social media. Nonprofit and Voluntary Sector Quarterly, 47(1), 5–26. doi: 10.1177/0899764017713724.CrossRefGoogle Scholar
Hardwick, R., Anderson, R., & Cooper, C. (2015). How do third sector organisations use research and other knowledge? A systematic scoping review. Implementation Science, 10(1), 84. doi: 10.1186/s13012-015-0265-6.CrossRefGoogle ScholarPubMed
Harris, M. (2001). The place of self and reflexivity in third sector scholarship: An exploration. Nonprofit and Voluntary Sector Quarterly, 30(4), 747–760. doi: 10.1177/0899764001304007.CrossRefGoogle Scholar
Heath, C., & Heath, D. (2013). Decisive: How to make better choices in life and work. Random House.Google Scholar
Hodgkinson, V., & Painter, A. (2003). Third sector research in international perspective: The role of ISTR. Voluntas: International Journal of Voluntary and Nonprofit Organizations, 14(1), 1–14.CrossRefGoogle Scholar
ISTR. (2019). International society for third-sector research strategic plan 2019–2024. https://cdn.ymaws.com/www.istr.org/resource/resmgr/docs/istr_strategic_plan_sept_201.pdfGoogle Scholar
Jones, R. (2019). Eugenics in education policy and the impact on African American students.Google Scholar
Kincheloe, J., & McLaren, P. (1994). Rethinking critical thinking and qualitative research. Handbook of qualitative research. Thousand Oaks, CA: Sage PublicationsGoogle Scholar
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores. arXiv preprint https://arxiv.org/abs/1609.05807Google Scholar
LaValle, S., Lesser, E., Shockley, R., Hopkins, M. S., & Kruschwitz, N. (2011). Big data, analytics and the path from insights to value. MIT Sloan Management Review, 52(2), 21–32.Google Scholar
Lavertu, S. (2016). We all need help: “Big data” and the mismeasure of public administration. Public Administration Review, 76(6), 864–872. doi: 10.1111/puar.12436.CrossRefGoogle Scholar
Lazer, D., Pentland, A. S., Adamic, L., Aral, S., Barabasi, A. L., Brewer, D., & Jebara, T. (2009). Life in the network: The coming age of computational social science. Science (new York, NY), 323(5915), 721. doi: 10.1126/science.1167742.CrossRefGoogle Scholar
Lecy, J., & Thornton, J. (2016). What big data can tell us about government awards to the nonprofit sector: Using the FAADS. Nonprofit and Voluntary Sector Quarterly, 45(5), 1052–1069. doi: 10.1177/0899764015620902.CrossRefGoogle Scholar
Lewis, D. L., & Willis, D. (2010). A small nation of people: WEB Du Bois and African American portraits of progress. Zondervan.Google Scholar
Liao, H., Tang, M., Luo, L., Li, C., Chiclana, F., & Zeng, X. J. (2018). A bibliometric analysis and visualization of medical big data research. Sustainability, 10(1), 166. doi: 10.3390/su10010166.CrossRefGoogle Scholar
Litofcenko, J., Karner, D., & Maier, F. (2020). Methods for classifying nonprofit organizations according to their field of activity: A report on semi-automated methods based on text. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 31(1), 227–237.10.1007/s11266-019-00181-wCrossRefGoogle Scholar
Marshall, B., and Geier, S., (2019) Targeted curricular innovations in data science, In: Proceedings of the IEEE Frontiers in Education Conference. 10.1109/FIE43999.2019.9028491CrossRefGoogle Scholar
Mergel, I., Rethemeyer, R. K., & Isett, K. (2016). Big data in public affairs. Public Administration Review, 76(6), 928–937. doi: 10.1111/puar.12625.CrossRefGoogle Scholar
Monroe-White, T. (2021). Emancipatory data science: a liberatory framework for mitigating data harms and fostering social transformation. In Proceedings of the 2021 on Computers and People Research Conference (pp. 23–30).10.1145/3458026.3462161CrossRefGoogle Scholar
Monroe-White, T. and Marshall, B. (2019). Data science intelligence: Mitigating public value failures using PAIR principles. Proceedings of the 2019 Pre-ICIS SIGDSA Symposium. 4._ https://aisel.aisnet.org/sigdsa2019/4Google Scholar
Mullainathan, S., & Spiess, J. (2017). Machine learning: An applied econometric approach. Journal of Economic Perspectives, 31(2), 87–106. doi: 10.1257/jep.31.2.87.CrossRefGoogle Scholar
Noble, S. U. (2018). Algorithms of oppression: How search engines reinforce racism. NYU Press.10.18574/nyu/9781479833641.001.0001CrossRefGoogle Scholar
Norris-Tirrell, D., Rinella, J., & Pham, X. (2018). Examining the career trajectories of nonprofit executive leaders. Nonprofit and Voluntary Sector Quarterly, 47(1), 146–164. doi: 10.1177/0899764017722023.CrossRefGoogle Scholar
Nwakpuda, E. I. (2020). Major donors and higher education: Are STEM donors different from other donors?. Nonprofit and Voluntary Sector Quarterly, 0899764020907153.Google Scholar
O’Neil, C. (2016). Weapons of math destruction: How big data increases inequality and threatens democracy. Broadway Books.Google Scholar
Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., & Weinberger, K. Q. (2017). On fairness and calibration. arXiv preprint https://arxiv.org/abs/1709.02012.Google Scholar
Regenbogen, S. E., Ehrenfeld, J. M., Lipsitz, S. R., Greenberg, C. C., Hutter, M. M., & Gawande, A. A. (2009). Utility of the surgical apgar score: Validation in 4119 patients. Archives of Surgery, 144(1), 30–36. doi: 10.1001/archsurg.2008.504.CrossRefGoogle ScholarPubMed
Roll-Hansen, N., & Broberg, G. (Eds.). (2005). Eugenics and the welfare state: Sterilization policy in Denmark, Sweden, Norway, and Finland. Michigan State University Press.Google Scholar
Sablan, J. R. (2019). Can you really measure that? Combining critical race theory and quantitative methods. American Educational Research Journal, 56(1), 178–203. doi: 10.3102/0002831218798325.CrossRefGoogle Scholar
Santos, M. R., Laureano, R. M., & Moro, S. (2020). Unveiling research trends for organizational reputation in the nonprofit sector, VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 31(1), 56–70CrossRefGoogle Scholar
Schneble, C. O., Elger, B. S., & Shaw, D. (2018). The cambridge analytica affair and Internet-mediated research. EMBO Reports, 19(8), e46579. doi: 10.15252/embr.201846579.CrossRefGoogle ScholarPubMed
Scurlock, R., Dolsak, N., & Prakash, A. (2020) Recovering from scandals: Twitter coverage of oxfam and save the children scandals, VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 31(1), 94–110.10.1007/s11266-019-00148-xCrossRefGoogle Scholar
Smith, S. M. (2000). " Looking at one's self through the eyes of others": WEB Du Bois's photographs for the 1900 paris exposition. African American Review, 34(4), 581–599. doi: 10.2307/2901420.CrossRefGoogle Scholar
Treaster, J. B. (2017). Will you graduate? Ask big data. The New York Times, 2.Google Scholar
Van Winkle, K. (2022). Above all made by themselves: The visual rhetoric of WEB Du Bois’s data visualizations. Technical Communication Quarterly, 31(1), 17–32. doi: 10.1080/10572252.2021.1906450.CrossRefGoogle Scholar
Viterna, J., Clough, E., & Clarke, K. (2015). Reclaiming the third sector from civil society a new agenda for development studies. Sociology of Development, 1(1), 173–207. doi: 10.1525/sod.2015.1.1.173.CrossRefGoogle Scholar
Walker, R. M., Chandra, Y., Zhang, J., & van Witteloostuijn, A. (2019). Topic modeling the research-practice gap in public administration. Public Administration Review, 79(6), 931–937. doi: 10.1111/puar.13095.CrossRefGoogle Scholar
Wasif, R. (2020). Does the media’s anti-western bias affect its portrayal of NGOs in the muslim world? Assessing newspapers in Pakistan. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 1–16.10.1007/s11266-020-00242-5CrossRefGoogle Scholar
Weizenbaum, J. (1972). On the impact of the computer on society. Science, 176(4035), 609–614. doi: 10.1126/science.176.4035.609.CrossRefGoogle ScholarPubMed
Wells, I. B. (1895). A red record. Tabulated statistics and alleged causes of lynchings in the United States, 1892–1893–1894. Respectfully submitted to the nineteenth century civilization in the land of the free and the home of the brave.Google Scholar
Wells, R. S., & Stage, F. K. (2015). Past, present, and future of critical quantitative research in higher education. New Directions for Institutional Research, 2014(163), 103–112. doi: 10.1002/ir.20089.CrossRefGoogle Scholar
Wilkerson, I. (2020). Caste: The Origins of Our Discontents. Random House.Google Scholar
Williamson, A. K., Luke, B., & Furneaux, C. (2020). Ties that bind: Public foundations in dyadic partnerships. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 1–13.Google Scholar
Zhu, J., Huang, H., & Zhang, D. (2019). Big Tigers, Big Data: Learning social reactions to China's anticorruption campaign through online feedback. Public Administration Review, 79(4), 500–513. doi: 10.1111/puar.12866.CrossRefGoogle Scholar
Zuberi, T., & Bonilla-Silva, E. (Eds.). (2008). White logic, white methods: Racism and methodology. Rowman & Littlefield Publishers.10.5040/9798216386278CrossRefGoogle Scholar