Concept Detectors: How Good is Good Enough?


Aly, Robin and Hiemstra, Djoerd (2009) Concept Detectors: How Good is Good Enough? In: 17th ACM International Conference on Multimedia, 19-22 October 2009, Beijing, China (pp. pp. 233-242).

[img] PDF
Restricted to UT campus only
: Request a copy
Abstract:Today, semantic concept based video retrieval systems often show insufficient performance for real-life applications. Clearly, a big share of the reason is the lacking performance of the detectors of these concepts. While concept detectors are on their endeavor to improve, following important questions need to be addressed: ``How good do detectors need to be to produce usable search systems?'' and ``How does the detector performance influence different concept combination methods?''. We use Monte Carlo Simulations to provide answers to the above questions. The main contribution of this paper is a probabilistic model of detectors which outputs confidence scores to indicate the likelihood of a concept to occur. This score is also converted into a posterior probability and a binary classification. We investigate the influence of changes to the model's parameters on the performance of multiple concept combination methods. Current web search engines produce a mean average precision (MAP) of around $0.20$. Our simulation reveals that the best performing video search method achieve this performance using detectors with $0.60$ MAP and is therefore usable in real-life. Furthermore, perfect detection allows the best performing combination method to produce $0.39$ search MAP in an artificial environment with Oracle settings. We also find that MAP is not necessarily a good evaluation measure for concept detectors since it is not always correlated with search performance.
Item Type:Conference or Workshop Item
Copyright:© 2009 ACM
Electrical Engineering, Mathematics and Computer Science (EEMCS)
Research Group:
Link to this item:
Official URL:
Export this item as:BibTeX
HTML Citation
Reference Manager


Repository Staff Only: item control page

Metis ID: 264419