Book contents
- The Cambridge Handbook of English Corpus Linguistics
- Cambridge Handbooks in Language and Linguistics
- The Cambridge Handbook of English Corpus Linguistics
- Copyright page
- Contents
- Figures
- Tables
- Contributors
- Major Corpora Cited in the Handbook
- Introduction
- Part I Methodological Considerations
- Part II Corpus Analysis of Linguistic Characteristics
- 4 Intonation and Other Prosodic Features
- 5 Keyword Analysis
- 6 Phraseology and Collocation
- 7 Lexical and Phraseological Complexity in Learner Language
- 8 Grammatical Variation
- 9 Grammatical Change
- 10 Discourse Analysis Using Corpus-Assisted Methodologies
- 11 Corpus Pragmatics
- 12 Multimodal Corpora
- Part III Corpus Analysis of Varieties
- Part IV Other Applications of Corpus Analysis
- Index
- References
12 - Multimodal Corpora
from Part II - Corpus Analysis of Linguistic Characteristics
Published online by Cambridge University Press: 08 July 2026
Book contents
- The Cambridge Handbook of English Corpus Linguistics
- Cambridge Handbooks in Language and Linguistics
- The Cambridge Handbook of English Corpus Linguistics
- Copyright page
- Contents
- Figures
- Tables
- Contributors
- Major Corpora Cited in the Handbook
- Introduction
- Part I Methodological Considerations
- Part II Corpus Analysis of Linguistic Characteristics
- 4 Intonation and Other Prosodic Features
- 5 Keyword Analysis
- 6 Phraseology and Collocation
- 7 Lexical and Phraseological Complexity in Learner Language
- 8 Grammatical Variation
- 9 Grammatical Change
- 10 Discourse Analysis Using Corpus-Assisted Methodologies
- 11 Corpus Pragmatics
- 12 Multimodal Corpora
- Part III Corpus Analysis of Varieties
- Part IV Other Applications of Corpus Analysis
- Index
- References
Summary
This chapter provides an overview of multimodal corpora, highlighting their significance in analyzing communication that integrates various modes such as speech, gesture, and gaze. The chapter defines both a mode and a multimodal corpus before introducing key types of multimodal corpora, including video-recorded interactions, corpora of signed languages, and datasets combining text and images. Prominent examples of multimodal corpora such as the ViMELF, FreMIC, and AMI corpora showcase the variety of multimodal corpora that exist and the various techniques employed to construct multimodal corpora. Challenges in constructing and annotating multimodal corpora are discussed, including temporal alignment, manual annotation efforts, and data-sharing limitations due to privacy concerns. The chapter emphasizes the importance of tailoring multimodal corpora to specific research questions while acknowledging the trade-offs of specialization. Advancements in technology, such as automated tagging and collaborative tools, are proposed as potential solutions to enhance accessibility and interoperability. A case study on turn management in virtual meetings illustrates the application of multimodal analysis, offering insights into how participants request and allocate turns in mediated contexts. The chapter serves as a guide for researchers navigating the methodological complexities of multimodal corpus-based studies.
Keywords
Information
- Type
- Chapter
- Information
- The Cambridge Handbook of English Corpus Linguistics , pp. 247 - 264Publisher: Cambridge University PressPrint publication year: 2026
References
Adolphs, S., & Carter, R. 2013. Spoken corpus linguistics: From monomodal to multimodal. London: Routledge.CrossRefGoogle Scholar
Akker, R., Theune, M., Truong, K., & Kok, I. 2010. The organisation of floor in meetings and the relation with speaker addressee Patterns. In Proceedings of the 2nd international workshop on social signal processing (pp. 35–40). New York: ACM. https://doi.org/10.1145/1878116.1878129.CrossRefGoogle Scholar
Allwood, J. 2002. Bodily communication dimensions of expression and content. In Granström, B., House, D., & Karlsson, I. (Eds.), Multimodality in language and speech systems (pp. 7–76). Dordrecht: Kluwer Academic.CrossRefGoogle Scholar
Allwood, J., Cerrato, L., Dybkjaer, L., Jokinen, K., Navarretta, C., & Paggio, P. 2004. The MUMIN multimodal coding scheme. Technical report, Nordic Network for MultiModal Interfaces (MUMIN). www.cst.dk/mumin/.Google Scholar
Aran, O., Hung, H., & Gatica-Perez, D. 2010. A multimodal corpus for studying dominance in small group conversations. LREC workshop on Multimodal Corpora, Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality. www.lrec-conf.org/proceedings/lrec2010/workshops/W6.pdf.Google Scholar
Baker, P., & Collins, L. 2023. Creating and analysing a multimodal corpus of news texts with Google Cloud Vision’s automatic image tagger. Applied Corpus Linguistics, 3(1), 100043.CrossRefGoogle Scholar
Bateman, J. A., & Tseng, C. I. 2023. Multimodal discourse analysis as a method for revealing narrative strategies in news videos. Multimodal Communication, 12(3), 261–285.CrossRefGoogle Scholar
Bateman, J. A., Wildfeuer, J., & Hiippala, T. 2017. Multimodality: Foundations, research and analysis: A problem-oriented introduction. Berlin: de Gruyter.Google Scholar
Bednarek, M., & Caple, H. 2017. The discourse of news values: How news organizations create newsworthiness. Oxford: Oxford University Press.CrossRefGoogle Scholar
Bezemer, J., & Kress, G. 2008. Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication, 25(2), 166–195.CrossRefGoogle Scholar
Boakye, K., Trueba-Hornero, B., Vinyals, O., & Friedland, G. 2008. Overlapped speech detection for improved speaker diarization in multiparty meetings. In 2008 IEEE international conference on acoustics, speech and signal processing (pp. 4353–4356). New York: IEEE.CrossRefGoogle Scholar
Brunner, M. L., & Diemer, S. 2021. Multimodal meaning making: The annotation of nonverbal elements in multimodal corpus transcription. Research in Corpus Linguistics, 9(1), 63–88.CrossRefGoogle Scholar
Caple, H., Huan, C., & Bednarek, M. 2020. Multimodal news analysis across cultures. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Carletta, J. 2007. Unleashing the killer corpus: Experiences in creating the multi-everything AMI meeting corpus. Language Resources & Evaluation, 41, 181–190. https://doi.org/10.1007/s10579-007-9040-x.CrossRefGoogle Scholar
Chen, Y., & Adolphs, S. 2022. Towards a corpus-based description of speech-gesture units of meaning: The case of the circular gesture. International Journal of Corpus Linguistics, 28(2), 172–201.CrossRefGoogle Scholar
Christiansen, A., Dance, W., & Wild, A. 2020. An introduction to visual constituent analysis. In Dayer, D. & Rüdiger, S. (Eds.), Corpus approaches to social media (pp. 149–175). Amsterdam: John Benjamins.CrossRefGoogle Scholar
Cohn, N. 2016. A multimodal parallel architecture: A cognitive framework for multimodal interactions. Cognition, 146, 304–323.CrossRefGoogle ScholarPubMed
Collet, C., Diemer, S., & Brunner, M. L. 2021. So in video‐mediated communication in the expanding circle. World Englishes, 40(4), 594–610.CrossRefGoogle Scholar
Collins, L. 2019. Corpus linguistics for online communication: A guide for research. London: Routledge.CrossRefGoogle Scholar
Collins, L. 2020. Working with images and emoji in the Dukki Facebook Corpus. In Rüdiger, S. & Dayter, D. (Eds.), Corpus approaches to social media (pp. 175–196). Amsterdam: John Benjamins.CrossRefGoogle Scholar
Diamantopoulou, S., & Ørevik, S. (Eds.). 2022. Multimodality in English language learning. London: Routledge.Google Scholar
Fernández Polo, F. J. 2021. Backchannels in video-mediated ELF conversations: A case study. Journal of English as a Lingua Franca, 10(1), 113–140.CrossRefGoogle Scholar
Goodwin, C. 1981. Conversational organisation: Interaction between speakers and hearers. New York: Academic Press.Google Scholar
Grzyb, B., Frank, S. L., & Vigliocco, G. 2024. Communicative efficiency in multimodal language directed at children and adults. Journal of Experimental Psychology, 153(7), 1904–1919. https://doi.org/10.1037/xge0001588.CrossRefGoogle ScholarPubMed
Janssens, J., & Van De Mieroop, D. 2024. The importance of multimodal resources for micro-oriented analyses of interactions: A case study of emergent leadership in a hybrid meeting. International Journal of Business Communication, 62(4), 659–684. https://doi.org/10.1177/23294884241234885.CrossRefGoogle Scholar
Jefferson, G. 2004. Glossary of transcript symbols with an introduction. In Lerner, G. H. (Ed.), Conversation analysis: Studies from the first generation (pp. 13–31). Amsterdam: John Benjamins.CrossRefGoogle Scholar
Knapp, M., & Hall, J. 2002 Nonverbal communication in human interaction. Belmont, CA: Wadsworth.Google Scholar
Knight, D. 2011. Multimodality and active listenership: A corpus approach. London: Continuum.Google Scholar
Knight, D., & Adolphs, S. 2008 Multi-modal corpus pragmatics: The case of active listenership. In Romero Trillo, J. (Ed.), Pragmatics and corpus linguistics: A mutualistic entente (pp. 175–190). Berlin: Mouton de Gruyter.CrossRefGoogle Scholar
Knight, D., O’Keeffe, A., Fitzgerald, C., McNamara, J., Mark, G., Peraldi, S., Fahey Palma, T., Farr, F., Cowan, B., & Adolphs, S. 2024. Corpus linguistics for virtual workplace discourse. London: Routledge.CrossRefGoogle Scholar
Kress, G., & Van Leeuwen, T. 2002. Colour as a semiotic mode: Notes for a grammar of colour. Visual Communication, 1(3), 343–368.CrossRefGoogle Scholar
Lycan, W. G. 1977. Conversation, politeness, and interruption. Paper in Linguistics, 10(1–2), 23–53.CrossRefGoogle Scholar
McCowan, I., Carletta, J., Kraaij, W., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Karaiskos, V., Kronenthal, M., Lathoud, G., Lincoln, M., Lisowska, A., Post, W., Reidsma, D., & Wellner, P. (2005). The AMI meeting corpus. In Noldus, L. P. J. J., Grieco, F., Loijens, L. W. S., & Zimmerman, P. H. (Eds.), Proceedings of measuring behavior 2005: 5th international conference on methods and techniques in behavioral research (pp. 137–140). Wageningen:Noldus Information Technology.Google Scholar
McGlashan, M. 2015. The representation of same-sex parents in children’s picturebooks: A corpus-assisted multimodal critical discourse analysis. PhD dissertation, Lancaster University.Google Scholar
Mills, S. 2011. Communities of practice and politeness. In Davies, B., Haugh, M., & Merrsion, A. J. (Eds.), Situated politeness (pp. 73–87). London: Continuum.Google Scholar
Mondada, L. 2007. Multimodal resources for turn-taking: Pointing and the emergence of possible next speakers. Discourse studies, 9(2), 194–225.CrossRefGoogle Scholar
O’Keeffe, A., Knight, D., Mark, G., Fitzgerald, C., McNamara, J., Adolphs, S., Fahey Palma, T., & Peraldi, S. 2024. “We’ve lost you Ian”: Multi-modal corpus innovations in capturing, processing and analysing professional online spoken interactions. Research in Corpus Linguistics, 12(2), 1–23.CrossRefGoogle Scholar
Paggio, P., Agirrezabal, M., Navarretta, C., & Vitasovic, L. 2024. Multimodal behaviour in an online environment: The GEHM Zoom corpus collection. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (pp. 11890–11900). https://aclanthology.org/2024.lrec-main.1038/.CrossRefGoogle Scholar
Peraldi, S., Fahey-Palma, T., O’Keeffe, A., Knight, D., Mark, G., Fitzgerald, C., McNamara, J., & Farr, F. In press. Doing meetings online: understanding variation in virtual workplace meetings. Register Studies.Google Scholar
Prendergast, M. 2019. Political cartoons as carnivalesque: A multimodal discourse analysis of Argentina’s Humor Registrado magazine, Social Semiotics, 29(1), 45–67.CrossRefGoogle Scholar
Reece, A., Cooney, G., Bull, P., Chung, C., Dawson, B., Fitzpatrick, C., Glazer, T., Knox, D., Liebscher, A., & Marin, S. 2023. The CANDOR corpus: Insights from a large multimodal dataset of naturalistic conversation. Science Advances, 9(13), eadf3197.CrossRefGoogle ScholarPubMed
Renals, S., Hain, T., & Bourlard, H. 2007. Recognition and understanding of meetings the AMI and AMIDA projects. In 2007 IEEE workshop on Automatic Speech Recognition & Understanding (ASRU) (pp. 238–247). New York: IEEE.CrossRefGoogle Scholar
Rühlemann, C., & Ptak, A. 2023. Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus. Open Linguistics, 9(1), 20220245.CrossRefGoogle Scholar
Sacks, H., Schegloff, E. A., & Jefferson, G. 1974. A simplest systematics for the organization of turn taking for conversation. Language, 50, 696–735.CrossRefGoogle Scholar
Sahlender, M., & ten Hagen, I. 2023. Do teachers adapt their gestures in linguistically heterogeneous second language teaching to learners’ language proficiencies? Gesture, 22(2), 189–226.CrossRefGoogle Scholar
Seuren, L. M., Wherton, J., Greenhalgh, T., & Shaw, S. E. 2021. Whose turn is it anyway? Latency and the organization of turn-taking in video-mediated interaction. Journal of Pragmatics, 172, 63–78. https://doi.org/10.1016/j.pragma.2020.11.005.CrossRefGoogle ScholarPubMed
Todd, R. W., Vungthong, S., Trakulkasemsuk, W., Pojanapunya, P., & Towns, S. G. 2023. Love is blue: Designing and using a multimodal corpus analysis tool. Applied Corpus Linguistics, 3(1), 100042.CrossRefGoogle Scholar
ViMELF 2018. Corpus of Video-Mediated English as a Lingua Franca Conversations. Birkenfeld: Trier University of Applied Sciences.Google Scholar
Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H. 2006. ELAN: A professional framework for multimodality research. In Proceedings of LREC 2006, Fifth International Conference on Language Resources and Evaluation (pp. 1556–1559). Genoa: European Language Resources Association (ELRA). https://aclanthology.org/L06-1082/.Google Scholar
Zhu, W., Hessel, J., Awadalla, A., Gadre, S. Y., Dodge, J., Fang, A., Yu, Y., Schmidt, L., Wang, W. Y., & Choi, Y. 2024. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 1–17.Google Scholar
Accessibility standard: Inaccessible, or known limited accessibility
Why this information is here
This section outlines the accessibility features of this content - including support for screen readers, full keyboard navigation and high-contrast display options. This may not be relevant for you.Accessibility Information
The HTML of this chapter is known to have missing or limited accessibility features.
We may be reviewing its accessibility for future improvement, but final compliance is
not yet assured and may be subject to legal exceptions. If you have any questions,
please contact
accessibility@cambridge.org.
Content Navigation
Table of contents navigation
Allows you to navigate directly to chapters, sections, or non‐text items through a linked table of contents, reducing the need for extensive scrolling.
Allows you to navigate directly to chapters, sections, or non‐text items through a linked table of contents, reducing the need for extensive scrolling.
Index navigation
Provides an interactive index, letting you go straight to where a term or subject appears in the text without manual searching.
Provides an interactive index, letting you go straight to where a term or subject appears in the text without manual searching.
Reading Order & Textual Equivalents
Single logical reading order
You will encounter all content (including footnotes, captions, etc.) in a clear, sequential flow, making it easier to follow with assistive tools like screen readers.
You will encounter all content (including footnotes, captions, etc.) in a clear, sequential flow, making it easier to follow with assistive tools like screen readers.
Visual Accessibility
Use of colour is not sole means of conveying information
You will still understand key ideas or prompts without relying solely on colour, which is especially helpful if you have colour vision deficiencies.
You will still understand key ideas or prompts without relying solely on colour, which is especially helpful if you have colour vision deficiencies.
Structural and Technical Features
ARIA roles provided
You gain clarity from ARIA (Accessible Rich Internet Applications) roles and attributes, as they help assistive technologies interpret how each part of the content functions.
You gain clarity from ARIA (Accessible Rich Internet Applications) roles and attributes, as they help assistive technologies interpret how each part of the content functions.