Temporal Relations of Verbal and Non-Verbal Behavior in Storytelling
Datum
2019-01-11
Autor:innen
Zeitschriftentitel
ISSN der Zeitschrift
Bandtitel
Verlag
University of Oregon
Zusammenfassung
This dissertation takes a ‘big data’ approach to analyzing a corpus of multimodal storytelling with the goal of providing data for researchers interested in developing more holistic models of production that integrate verbal and non-verbal behavior. Rather than approaching the data with a specific hypothesis in mind, I approach the data with a set of methods that analyze the temporal relationship between two behaviors and apply the methods to every single possible pair of behaviors. Rather than using the data to test hypotheses, I am using it to formulate them.
The methods used in this dissertation examine covariation between behaviors (how much do two behaviors overlap with each other, and is this more or less likely than we would expect, given a random distribution of the two behaviors), the sequential patterns of behaviors (the multimodal n-grams of behaviors that are most strongly associated), and the frequency distribution of behavior boundaries (the timing of behavior onsets and offsets near other behavior onsets and offsets).
The analyses examine all possible pairs of behaviors from four modalities (head gesture, manual gesture, eye-gaze, and speech), as well as looking within and across roles of speaker and listener. A list of testable hypotheses is given, based on the findings in the data.