|
|
|
                                          
|
Return to Session IIa: Test collections
Test collections model use cases in ways that facilitate evaluation of information retrieval systems. This paper describes the use of search-guided relevance assessment to create a test collection for retrieval of spontaneous conversational speech. Approximately 10,000 thematically coherent segments were manually identified in 625 hours of oral history interviews with 246 individuals. Automatic speech recognition results, manually prepared summaries, controlled vocabulary indexing, and name authority control are available for every segment. Those features were leveraged by a team of four relevance assessors to identify topically relevant segments for 28 topics developed from actual user requests. Search-guided assessment yielded sufficient inter-annotator agreement to support formative evaluation during system development. Baseline results for ranked retrieval are presented to illustrate use of the collection.
@inproceedings{1009002, author = {Douglas W. Oard and Dagobert Soergel and David Doermann and Xiaoli Huang and G. Craig Murray and Jianqiang Wang and Bhuvana Ramabhadran and Martin Franz and Samuel Gustman and James Mayfield and Liliya Kharevych and Stephanie Strassel}, title = {Building an information retrieval test collection for spontaneous conversational speech}, booktitle = {SIGIR '04: Proceedings of the 27th annual international conference on Research and development in information retrieval}, year = {2004}, isbn = {1-58113-881-4}, pages = {41--48}, location = {Sheffield, United Kingdom}, doi = {http://doi.acm.org/10.1145/1008992.1009002}, publisher = {ACM Press}, }  ©2005 Association for Computing Machinery |
