Speech Transcript Evaluation for Information Retrieval

Werff van der, Laurens and Kraaij, Wessel and Jong de, Franciska (2011) Speech Transcript Evaluation for Information Retrieval. In: 12th Annual Conference of the International Speech Communication Association, Interspeech 2011, 27-31 Aug 2011, Florence, Italy.

Preview

PDF
193Kb

Abstract:	Speech recognition transcripts are being used in various fields of research and practical applications, putting various demands on their accuracy. Traditionally ASR research has used intrinsic evaluation measures such as word error rate to determine transcript quality. In non-dictation-type applications such as speech retrieval, it is better to use extrinsic (or task specific) measures. Indexation and the associated processing may eliminate certain errors, whereas the search query may reveal others. In this work, we argue that the standard extrinsic speech retrieval measure average precision is unpractical for ASR evaluation. As an alternative we propose the use of ranked correlation measures on the output of the speech retrieval task, with the goal of predicting relative mean average precision. The measures we used showed a reasonably high correlation with average precision, but require much less human effort to calculate and can be more easily deployed in a variety of real-life settings.
Item Type:	Conference or Workshop Item
Faculty:	Electrical Engineering, Mathematics and Computer Science (EEMCS)
Research Group:	Human Media Interaction (HMI)
Link to this item:	http://purl.utwente.nl/publications/78194
Official URL:	http://mickey.ifp.uiuc.edu/speech/akantor/proceedings/interspeech2011/IS2011/PDF/AUTHOR/IS110170.PDF
Conference URL:	http://www.interspeech2011.org/
Export this item as:	BibTeX EndNote HTML Citation Reference Manager

Show download statistics for this publication

Repository Staff Only: item control page