Video, multimodality and the embodied organisation of action

A study that records only sound has decided what counts. It has decided that talk carries the action, that hands, eyes and objects are context, and that a transcript holds what the analysis needs. The unit before it read talk as a sequence. This eighth of ten units adds the body and the setting, and argues the opposite. Gaze, gesture and the manipulation of objects are organised in time and across modalities. Claims about them can be warranted only by analysis of the moving record, and a transcript of talk alone cannot carry a multimodal claim.

The argument takes six steps: what the moving record records, why the body belongs to the action, why the analyst's categories are part of the phenomenon, how reference is settled in the body, what a design claim inherits from the record, and what a transcript must become. It closes on which modes count, and who decides.

The record is collected before the analytic question settles

Heath, Hindmarsh and Luff (2010) write the practical guide, and its order argues. The chapters run through access, ethics and planning, collecting data, developing preliminary observations from the video, matters of context, and the presentation of findings, with transcription notations in an appendix. The apparatus that makes it inspectable is a support rather than the evidence.

Jordan and Henderson (1995) state the method. Interaction analysis investigates how people interact with each other and with objects. It covers talk, non-verbal interaction and the use of artefacts, with video as the primary record and replay as the instrument. The authors refuse predetermined categories. They offer foci instead, among them the structure of events, the temporal organisation of activity, turn taking and participation structures. The list continues with trouble and repair, the spatial organisation of activity, and artefacts and documents. Heath, Hindmarsh and Luff (2010) put access and ethics before collection, and reprint an introductory letter seeking access as an appendix. What the record can contain is therefore settled with participants before the analysis starts. A study that reports its codes but not its record has replaced the instrument with a summary of it.

Why the body belongs to the action

Goodwin (2000) gives the reason. From video recordings of girls playing hopscotch and of archaeologists classifying colour, he argues that action is assembled from semiotic fields that mutually elaborate each other. His unit is the contextual configuration, the array of fields participants demonstrably orient to. He contrasts that array with a set an analyst might impose to code context. Relevance is therefore an empirical finding about a course of action, and the record is where it is found.

Streeck, Goodwin and LeBaron (2011) show how far the apparatus travels. They gather video-based studies of linguistic structure coordinated with gaze, gesture and the material setting, across family interaction, aircraft, clinics, architects' offices and musical performances. The volume has chapters on participants with aphasia, blindness and deafness. The body is the ordinary medium of action.

Seeing is skilled, and the analyst's categories are part of it

Goodwin (1994) makes the analyst's own apparatus an object of study. From videotapes of archaeologists making maps and of lawyers working with the Rodney King videotape, he identifies three practices. A profession shapes what its members see through coding schemes, highlighting, and the articulation of graphic representations. The archaeological case is small and complete. A student classifying dirt holds the sample under holes cut in a Munsell colour chart and squirts it with water to bring out the colour. A category on a form thus becomes a physical procedure any trained student can repeat. The same videotape supported two readings, and each side worked to make its own visible. A code or a frame grab is therefore work rather than a neutral report. The record must stay inspectable for that work to be disputed.

Reference is settled in the body

Hindmarsh and Heath (2000) give the case that costs a transcript its standing. From audio-visual recordings in a telecommunications control centre, they examine how colleagues establish mutual orientation towards documents and screens. A deictic expression has its referent fixed by gaze, gesture and bodily orientation. The words alone leave it indeterminate.

vom Lehn, Heath and Hindmarsh (2001) extend the point to a public setting. From video recordings in museums and galleries, they show that visitors, alone and together, encounter exhibits in ways organised with regard to the actions of others. Their fragment is one visitor. The pace of his walk and his successive glances organise what he sees, each print read in the light of the one before, until he stops opposite a particular print. Looking is a public and sequentially organised achievement, which no visitor log records.

A design claim and a transcript both inherit what the record shows

Heath and Luff (2000) carry the method into systems. Their video-based field studies run from information systems in general medical practice to media spaces, news production, the London Underground control room and computer aided design. The finding for design is that systems are encountered inside courses of action that studies of adoption and self-report miss. A design claim about a system's use inherits the evidence of the record behind it.

Mondada (2018) shows that the step from record to claim is the hard part. She asks how multimodal conversation analysis can be operationalised in a transcription system, using video recordings of French and Swiss German encounters. She names the phenomena that stretch a transcript: body arrangements in interactional spaces, material environments, mobile settings, silent activities, haptic contact and camera movement. Transcription choices, she concludes, carry conceptual consequences. Mondada (2016) had already repositioned language as one modality among others, warning against logo-centric and visuo-centric views. The transcript stays a representation. It must be tied back to the record, and it settles in advance what can be claimed.

Contested question: which modes count, and who decides?

The first position puts the analyst's framework in charge. Its evidence is framework exposition. Norris (2004) offers modal density, the modal intensity and modal complexity through which a higher-level action is built from chains of lower-level actions, weighting embodied modes alongside modes not carried by the body. Jewitt (2014) shows a plural field in a handbook of thirty-four chapters, in which different approaches sit alongside one another. Bezemer and Kress (2016) treat modes as culturally shaped resources and analyse learning and communication from video recordings, photographs, screenshots and print materials.

The second position puts the participants in charge. Goodwin (2000) defines relevance by what participants demonstrably orient to, expressly not by what an analyst imposes to code context, and his evidence is close analysis of video. Hindmarsh and Heath (2000) apply the same discipline to reference, where the referent is settled in bodily conduct. Jordan and Henderson (1995) refuse predetermined categories in a foundational methods article. A mode the analyst ranks highly but no participant treats as relevant has been measured, but not found.

The third position turns on the instrument. Mondada (2018) shows from video corpora that what a transcription system can represent changes what an analysis can claim. The apparatus therefore sits inside the finding rather than outside it. What would settle it is the three checks that follow. The first is prediction: take segments where a framework ranks one mode above another, and ask whether the participants' next action displays orientation to the mode ranked highest. The second is convergence: give the same corpus to analysts from the sequential and the social-semiotic traditions, and compare their claims. The third is deviant cases: ask each framework what would count against its weighting.

The strongest counter-argument to this unit's own claim is that it states one tradition's standard as a general one. A social-semiotic claim about what a mode affords does not require sequential evidence at all, and Bezemer and Kress (2016) work from still materials. The moving record is the warrant the sequential tradition requires, not the only warrant there is.

What to do while the dispute is open

Say which mode the claim rests on, and show the segment. Where modes are weighted analytically, test the weighting against displayed orientation before treating it as measurement. Keep the record and the clips, so a reader can inspect the moment the claim turns on, and keep a lighter copy for a reader on a metered connection.

Where the argument is heading

The record is becoming mobile, because Mondada (2018) lists mobile settings and camera movement among what a transcription system must handle. That makes the analyst's camera part of the object. The phenomena are widening. Streeck, Goodwin and LeBaron (2011) gather studies of participants with aphasia, blindness and deafness, and Heath and Luff (2000) keep the analysis close to systems in use. The record is also becoming a shared object, because Heath, Hindmarsh and Luff (2010) treat data sessions and access as part of the method. The standard that follows is modest: name the segment, say what the record shows there, and let the reader inspect it.

The sources this unit rests on

• Heath, C., Hindmarsh, J. and Luff, P. (2010). Video in Qualitative Research: Analysing Social Interaction in Everyday Life. London: Sage. doi:10.4135/9781526435385 The guide runs from access and planning through collecting data to developing preliminary observations and presenting findings, with transcription notations in an appendix. It gives the unit video as primary data rather than illustration.
• Hindmarsh, J. and Heath, C. (2000). Embodied reference: A study of deixis in workplace interaction. Journal of Pragmatics 32(12), 1855-1878. doi:10.1016/s0378-2166(99)00122-8 From recordings in a telecommunications control centre, they examine how colleagues establish mutual orientation towards documents and screens. The referent of a deictic expression is fixed by gaze, gesture and bodily orientation, which talk alone leaves indeterminate.
• Goodwin, C. (1994). Professional vision. American Anthropologist 96(3), 606-633. doi:10.1525/aa.1994.96.3.02a00100 Goodwin uses videotapes of archaeologists making maps and of lawyers working with the Rodney King videotape to identify three practices that shape professional seeing: coding schemes, highlighting, and graphic representations. It gives the unit its reason to treat a code or frame grab as work, not a neutral report.
• Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics 32(10), 1489-1522. doi:10.1016/s0378-2166(99)00096-x From video recordings of girls playing hopscotch and of archaeologists classifying colour, Goodwin argues that action is assembled from semiotic fields that mutually elaborate one another, and defines the contextual configuration as the array of fields participants demonstrably orient to, not a set the analyst imposes. Relevance is therefore an empirical finding, which is how the unit uses the paper.
• Jordan, B. and Henderson, A. (1995). Interaction analysis: Foundations and practice. Journal of the Learning Sciences 4(1), 39-103. doi:10.1207/s15327809jls0401_2 They define interaction analysis as a method for investigating how people interact with each other and with objects, covering talk, non-verbal interaction and the use of artefacts, with video as the primary record and replay as the instrument. They refuse predetermined categories and offer foci: the structure of events, the temporal organisation of activity, turn taking, participation structures, trouble and repair, the spatial organisation of activity, and artefacts.
• Mondada, L. (2016). Challenges of multimodality: Language and the body in social interaction. Journal of Sociolinguistics 20(3), 336-366. doi:10.1111/josl.1_12177 Mondada asks what follows from studying language and the body together, arguing that multimodal analysis reconceptualises action, cognition and temporality and that language should be repositioned as one modality among others. She warns against logo-centric and visuo-centric views.
• Mondada, L. (2018). Multiple temporalities of language and body in interaction: Challenges for transcribing multimodality. Research on Language and Social Interaction 51(1), 85-106. doi:10.1080/08351813.2018.1413878 Using video recordings of French and Swiss German encounters and of animal interactions, Mondada asks how multimodal conversation analysis can be operationalised in transcription and names what stretches it: body arrangements in interactional spaces, material environments, mobile settings, silent activities and camera movement. Transcription choices, she finds, carry conceptual consequences.
• Norris, S. (2004). Analysing Multimodal Interaction: A Methodological Framework. New York: Routledge. doi:10.4324/9780203379493 Norris proposes modal density, made of modal intensity and modal complexity, through which higher-level actions are built from chains of lower-level actions, with embodied modes such as speech, gaze and gesture weighted alongside modes not carried by the body. It gives the unit its analyst-led position.
• Streeck, J., Goodwin, C. and LeBaron, C. (2011). Embodied Interaction: Language and Body in the Material World. Cambridge: Cambridge University Press. https://assets.cambridge.org/97805218/95637/frontmatter/9780521895637_frontmatter.pdf The collection gathers video-based studies of linguistic structure coordinated with gaze, gesture and the material setting, across family interaction, day care centres, work settings such as aircraft and clinics, and performances, with chapters on participants with aphasia, blindness and deafness. It shows the apparatus travelling to impaired and material settings.
• Heath, C. and Luff, P. (2000). Technology in Action. Cambridge: Cambridge University Press. doi:10.1017/cbo9780511489839 Heath and Luff use video-based field studies of how complex systems affect work and interaction, from information systems in general medical practice to media spaces, news production, the London Underground control room and computer aided design. They argue that systems are encountered inside courses of action that studies of adoption miss, and that such studies can inform design.
• Bezemer, J. and Kress, G. (2016). Multimodality, Learning and Communication: A Social Semiotic Frame. London: Routledge. doi:10.4324/9781315687537 They analyse sign making in museums, hospitals, schools and the home using video recordings, photographs, screenshots and print materials, and treat modes as culturally shaped resources carrying meaning. The frame asks what a mode affords rather than where it sits in a sequence, the unit's strongest objection to requiring the moving record.
• vom Lehn, D., Heath, C. and Hindmarsh, J. (2001). Exhibiting interaction: Conduct and collaboration in museums and galleries. Symbolic Interaction 24(2), 189-216. doi:10.1525/si.2001.24.2.189 From video recordings in museums and galleries, the authors show that visitors, alone and together, encounter exhibits in ways organised with regard to the actions of others, companions and strangers alike. Looking is a public and sequentially organised achievement, a case where the phenomena are gaze and body, not talk.
• Jewitt, C. (2014). The Routledge Handbook of Multimodal Analysis. 2nd edition. London: Routledge. https://www.routledge.com/The-Routledge-Handbook-of-Multimodal-Analysis/Jewitt/p/book/9780415714808 The second edition presents a research toolkit of thirty-four chapters, twelve of them new, setting different approaches to multimodality alongside one another. It shows that multimodality is a family of approaches rather than a single method, so an analyst must say which one is in use.