Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition

Bousmalis, Konstantinos and Morencey, Louis–Philippe and Pantic, Maja (2011) Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition. In: IEEE International Conference on Automatic Face & Gesture Recognition and Workshops, FG 2011, 21-25 March 2011, Santa Barbara, CA.

PDF
Restricted to UT campus only: Request a copy
647Kb

Abstract:	This paper attempts to recognize spontaneous agreement and disagreement based only on nonverbal multi-modal cues. Related work has mainly used verbal and prosodic cues. We demonstrate that it is possible to correctly recognize agreement and disagreement without the use of verbal context (i.e. words, syntax). We propose to explicitly model the complex hidden dynamics of the multimodal cues using a sequential discriminative model, the Hidden Conditional Random Field (HCRF). In this paper, we show that the HCRF model is able to capture what makes each of these social attitudes unique. We present an efficient technique to analyze the concepts learned by the HCRF model and show that these coincide with the findings from social psychology regarding which cues are most prevalent in agreement and disagreement. Our experiments are performed on a spontaneous dataset of real televised debates. The HCRF model outperforms conventional approaches such as Hidden Markov Models and Support Vector Machines.
Item Type:	Conference or Workshop Item
Copyright:	© 2011 IEEE
Faculty:	Electrical Engineering, Mathematics and Computer Science (EEMCS)
Research Group:	Human Media Interaction (HMI)
Link to this item:	http://purl.utwente.nl/publications/79505
Official URL:	http://dx.doi.org/10.1109/FG.2011.5771341
Export this item as:	BibTeX EndNote HTML Citation Reference Manager

Show download statistics for this publication

Repository Staff Only: item control page