Speech Transcript Evaluation for Information Retrieval


Werff, Laurens van der and Kraaij, Wessel and Jong, Franciska de (2011) Speech Transcript Evaluation for Information Retrieval. In: 12th Annual Conference of the International Speech Communication Association, Interspeech 2011, 27-31 Aug 2011, Florence, Italy (pp. pp. 1525-1528).

open access
Abstract:Speech recognition transcripts are being used in various fields of research and practical applications, putting various demands on their accuracy. Traditionally ASR research has used intrinsic evaluation measures such as word error rate to determine transcript quality. In non-dictation-type applications such as speech retrieval, it is better to use extrinsic (or task specific) measures. Indexation and the associated processing may eliminate certain errors, whereas the search query may reveal others. In this work, we argue that the standard extrinsic speech retrieval measure average precision is unpractical for ASR evaluation. As an alternative we propose the use of ranked correlation measures on the output of the speech retrieval task, with the goal of predicting relative mean average precision. The measures we used showed a reasonably high correlation with average precision, but require much less human effort to calculate and can be more easily deployed in a variety of real-life settings.
Item Type:Conference or Workshop Item
Electrical Engineering, Mathematics and Computer Science (EEMCS)
Research Group:
Link to this item:http://purl.utwente.nl/publications/78194
Official URL:http://mickey.ifp.uiuc.edu/speech/akantor/proceedings/interspeech2011/IS2011/PDF/AUTHOR/IS110170.PDF
Conference URL:http://www.interspeech2011.org/
Export this item as:BibTeX
HTML Citation
Reference Manager


Repository Staff Only: item control page