String-based audiovisual fusion of behavioural events for the assessment of dimensional affect


Eyben, Florian and Wöllmer, Martin and Valstar, Michel F. and Gunes, Hatice and Schuller, Björn and Pantic, Maja (2011) String-based audiovisual fusion of behavioural events for the assessment of dimensional affect. In: IEEE International Conference on Automatic Face & Gesture Recognition and Workshops, FG 2011, 21-25 March 2011, Santa Barbara, CA (pp. pp. 322-329).

[img] PDF
Restricted to UT campus only
: Request a copy
Abstract:The automatic assessment of affect is mostly based on feature-level approaches, such as distances between facial points or prosodic and spectral information when it comes to audiovisual analysis. However, it is known and intuitive that behavioural events such as smiles, head shakes or laughter and sighs also bear highly relevant information regarding a subject's affective display. Accordingly, we propose a novel string-based prediction approach to fuse such events and to predict human affect in a continuous dimensional space. Extensive analysis and evaluation has been conducted using the newly released SEMAINE database of human-to-agent communication. For a thorough understanding of the obtained results, we provide additional benchmarks by more conventional feature-level modelling, and compare these and the string-based approach to fusion of signal-based features and string-based events. Our experimental results show that the proposed string-based approach is the best performing approach for automatic prediction of Valence and Expectation dimensions, and improves prediction performance for the other dimensions when combined with at least acoustic signal-based features.
Item Type:Conference or Workshop Item
Copyright:© 2011 IEEE
Electrical Engineering, Mathematics and Computer Science (EEMCS)
Research Group:
Link to this item:
Official URL:
Export this item as:BibTeX
HTML Citation
Reference Manager


Repository Staff Only: item control page