A comparison of user and system query performance predictions


Hauff, Claudia and Kelly, Diane and Azzopardi, Leif (2010) A comparison of user and system query performance predictions. In: 19th ACM International Conference on Information and Knowledge Management, CIKM '10, 26-30 Oct 2010, Toronto, Canada (pp. pp. 979-988).

[img] PDF
Restricted to UT campus only
: Request a copy
Abstract:Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.
Item Type:Conference or Workshop Item
Copyright:© 2010 ACM
Electrical Engineering, Mathematics and Computer Science (EEMCS)
Research Group:
Link to this item:http://purl.utwente.nl/publications/74334
Official URL:https://doi.org/10.1145/1871437.1871562
Export this item as:BibTeX
HTML Citation
Reference Manager


Repository Staff Only: item control page