Relying on topic subsets for system ranking estimation


Share/Save/Bookmark

Hauff, Claudia and Hiemstra, Djoerd and Jong, Franciska de and Azzopardi, Leif (2009) Relying on topic subsets for system ranking estimation. In: 18th ACM Conference on Information and Knowledge Management, CIKM'09, 2-6 November 2009, Hong Kong, China (pp. pp. 1859-1862).

[img] PDF
Restricted to UT campus only
: Request a copy
341kB
Abstract:Ranking a number of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments was first explored by Soboroff et al [6]. Over the years, a number of alternative approaches have been proposed. We perform a comprehensive analysis of system ranking estimation approaches on a wide variety of TREC test collections and topics sets. Our analysis reveals that the performance of such approaches is highly dependent upon the topic or topic subset, used for estimation. We hypothesize that the performance of system ranking estimation approaches can be improved by selecting the "right" subset of topics and show that using topic subsets improves the performance by 32% on average, with a maximum improvement of up to 70% in some cases.
Item Type:Conference or Workshop Item
Copyright:© 2009 ACM
Faculty:
Electrical Engineering, Mathematics and Computer Science (EEMCS)
Research Group:
Link to this item:http://purl.utwente.nl/publications/69482
Official URL:http://doi.acm.org/10.1145/1645953.1646249
Export this item as:BibTeX
EndNote
HTML Citation
Reference Manager

 

Repository Staff Only: item control page