Evaluating retrieval: relevance, test collections and the limits of measurement

A ranked list is only good if it matches a judgement of what should have been returned, and that judgement is made by a person, about a document, at a moment, for a purpose.

This lecture argues that a retrieval score measures the test collection and its assessors as much as the system. It argues that pooled evaluation under-credits a system which retrieves relevant documents the pool never held. It also argues that the honest repair is a measure that answers to the judgement rather than a pretence that the judgement is objective. This eighth unit asks what that estimate is measured against.

A score means nothing until a collection fixes the judgements

Cleverdon (1967) established the method. The Cranfield tests compared index languages against a fixed document collection and a fixed set of queries, with relevance judged once. They reported that the specificity of the index terms mattered most: single-word terms were more effective than concept terms or a controlled vocabulary under the conditions of the test. Fixing documents, queries and judgements lets a score difference be read as a system difference.

The collection Cranfield built was about 1,400 documents with roughly 220 queries, each with its relevance judgements recorded once, and the small set of assessors who paid for that judging set the standard against which every later score was measured (Cleverdon, 1967). Nothing records who those assessors were, so a reader working in a second language is not among the judgements they made.

Sparck Jones (1974) shows what that design measured in its early years. She reviews automatic techniques for analysing and describing documents, for manipulating those descriptions in search, and for generating the index language across the literature of 1968 to 1973.

Relevance is a judgement with several senses

Saracevic (1975) reviewed the thinking on relevance and interpreted its different views within one framework of communication. He treats relevance as a measure of the effectiveness of contact between a source and a destination. Different views arose because relevance was considered at different points in the process of knowledge communication. Relevant, then, is not one thing, and a score computed under one sense need not agree with a score computed under another.

Mizzaro (1997) showed the disagreement was structural. He reviewed the history of relevance through about 160 papers, divided it into three periods and analysed the papers under seven aspects, among them the kinds of relevance and agreement among judges. A concept that needs a framework to be discussed at all is not a property a document carries.

Borlund (2003) carried the point into evaluation, arguing that the multidimensional and dynamic nature of relevance explains why no consensus definition has emerged, so the definition a measure embeds is a choice about what the measure reports.

The pooling assumption and what it hides

A test collection cannot judge every document, so it judges a pool of the documents returned by the systems under test. Zobel (1998) examined both stages of large-scale IR measurement, the gathering of documents for assessment and the use of the assessments. He found the measured relative performance of systems reliable while recall was overestimated, because many relevant documents had never been found and so could not be credited. He proposed a pooling strategy that finds more relevant documents for a given effort without compromising fairness. Relative and absolute claims come apart, and the inflation belongs to the pool rather than to any one system.

Buckley and Voorhees (2004) turned that into a result about the measures. Current measures are not robust to gross violations of the completeness assumption, the assumption that all relevant documents have been identified and are present. Their replacement correlates highly with the existing measures when judgements are complete and more robust when they are not. Larger or dynamic collections built on current pooling practices remain viable even with incomplete relevance information.

A constructed case shows the mechanism. Two systems are entered in an evaluation and the assessors judge the top hundred documents each returned. A document relevant to a topic sits at rank three hundred for both, but a third system, entered later, returns it at rank five. No system in the pool returned it, so no assessor saw it, and the third system is charged for having found something nobody judged. The illustration is constructed to exhibit the mechanism Buckley and Voorhees (2004) measure, not a measured case, and the size of the effect depends on the depth of the pool.

Judge variation moves the scores without moving the ranking

Assessors disagree, so the measurement cannot be trusted. Voorhees (1998) tested it by redoing relevance judgements with different assessors and tracking how far effectiveness scores moved. She found the relative ranking of systems comparatively stable even where individual judgements varied.

Sanderson and Zobel (2005) made the statistical side explicit. They found the t-test highly reliable, more reliable than the sign or Wilcoxon tests and far more reliable than reporting a percentage difference with no test. They argued that assessor effort is better spent on more topics assessed in less detail. A comparison without a significance test is therefore not evidence.

Measuring a comparison rather than a score

Two systems return different documents, so their ranked lists are not conjoint, and evaluation often happens at a depth where the lists are incomplete. Webber, Moffat and Zobel (2010) proposed rank-biased overlap, a measure of similarity between incomplete ranked lists that handles non-conjointness. It weights high ranks more heavily than low, and it is monotonic with increasing depth of evaluation, a combination no earlier measure satisfied.

Voorhees, Samarov and Soboroff (2017) sharpened the comparison from the other end. Randomly partitioning the collection gives replicates, and bootstrap ANOVA extracts the system-topic interactions unavailable without them, yielding a more precise system effect and a narrower confidence interval, with more significant pairwise differences found.

Contested question: does the score measure the system or the collection?

The literature disagrees, and both sides are right about a different quantity.

The first position holds that judge variation and pooling are noise and that comparison survives them. Voorhees (1998) supplies its evidence, since relative rankings held when judgements were redone, and Sanderson and Zobel (2005) show the standard test is reliable enough to assert a difference. On this account the test collection is a serviceable instrument.

The second position holds that the pool is a systematic bias the measures inherit. Zobel (1998) shows recall is overestimated because relevant documents outside the pool are never found, and Buckley and Voorhees (2004) show the standard measures are not robust once completeness is violated grossly. Saracevic (1975), Mizzaro (1997) and Borlund (2003) reinforce the critique, since a score built on one sense of relevance cannot stand for the senses it ignores.

The third position, which this lecture takes, is that these are statements about different quantities. A relative claim can survive judge variation while an absolute claim such as recall cannot. What would settle it is an evaluation that varies the collection and the system independently. It would report absolute and comparative measures alongside the pooling depth behind each collection. It would show whether an advantage survives on collections never used to build the pool. The robust measures (Buckley and Voorhees, 2004; Webber, Moffat and Zobel, 2010) and the replicate designs (Voorhees, Samarov and Soboroff, 2017) are the toolkit such an evaluation would use.

Rate each side by its evidence. Voorhees (1998) and Sanderson and Zobel (2005) measured the reliability of the comparison itself, so the defence rests on experiment. Zobel (1998) recorded an inflated recall from a study of the pooled experiments, and Buckley and Voorhees (2004) measured how the standard measures fail under incomplete judgements. The strongest objection is that the critique of pooling is itself produced from pooled collections, so a defect found with the instrument cannot be repaired by measuring harder with it. While the dispute is open, a designer reports the pooling depth and the measure behind any comparison, and treats an absolute recall figure as an estimate.

Where the argument is heading

Ferro and Peters (2019) gather the state of the field at book length. They review evaluation over twenty years of CLEF, covering the design of campaigns, the measures used and the challenges of new retrieval tasks. The robustness line replaces a measure that silently assumes completeness with one that reports what it was computed over (Buckley and Voorhees, 2004; Webber, Moffat and Zobel, 2010). The sensitivity line sharpens the comparison by design rather than by adding more systems to the same shallow pool (Voorhees, Samarov and Soboroff, 2017). It is continuous with Sanderson and Zobel's (2005) preference for more topics assessed in less detail. A third line asks whether the judgement itself can be produced by a machine. Faggioli and colleagues (2023) set out a spectrum of judgement strategies by how far the machine is trusted, from fully human to fully automated, and report a pilot comparison of a language model's judgements against trained assessors. The open question is whether a fixed judged collection can stand for the queries a deployed system will receive. Cleverdon's (1967) design assumed the judgements were a fixed standard, and that assumption is still under test.

The sources this unit rests on

• Borlund, P. (2003). The concept of relevance in IR. Journal of the American Society for Information Science and Technology. Borlund surveys the multidimensional and dynamic nature of relevance and argues that this multidimensionality explains why no consensus definition has emerged, with particular attention to situational relevance. It connects the philosophical disagreement to practice, since the definition a measure embeds decides what it reports.
• Buckley, C. and Voorhees, E. M. (2004). Retrieval evaluation with incomplete information. Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval. The authors show that current evaluation measures are not robust to gross violations of the assumption that all relevant documents have been identified, and introduce a measure that is more robust when judgements are incomplete.
• Cleverdon, C. (1967). The Cranfield tests on index language devices. Aslib Proceedings. Cleverdon compared index languages on a fixed collection and query set with relevance judged once, finding that the specificity of the index terms mattered most and that single-word terms outperformed concept terms and a controlled vocabulary. It is the origin of the test-collection method the unit evaluates.
• Faggioli, G., Dietz, L., Clarke, C. L. A. et al. (2023). Sets out a spectrum of relevance-judgement strategies by how far the machine is trusted, from fully human to fully automated, and reports a pilot comparison of a language model's judgements against trained assessors. It opens a third route to a test collection, judgements generated rather than collected. It is a perspectives paper with a small pilot, not a validated substitute.
• Ferro, N. and Peters, C. (eds.) (2019). Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF. The Information Retrieval Series. This edited volume reviews information retrieval evaluation across twenty years of CLEF, covering experimental evaluation and its theory, evaluation infrastructures, the campaigns and the challenges that remain; the publisher's description and the editor list were read, and the chapters sit behind a subscription. It supplies the book-length state of the field where this unit's arguments have arrived.
• Mizzaro, S. (1997). Relevance: the whole history. Journal of the American Society for Information Science. Mizzaro reviews the history of relevance through about 160 papers, defines a framework for the positions, and presents it in three chronological periods under seven aspects, including kinds of relevance and agreement among judges.
• Sanderson, M. and Zobel, J. (2005). Information retrieval system evaluation: effort, sensitivity, and reliability. Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval. The authors revisit how significance tests should be used when comparing retrieval systems and find the t-test highly reliable, more so than the sign or Wilcoxon tests and far more reliable than reporting a percentage difference with no test.
• Saracevic, T. (1975). Relevance: a review of and a framework for the thinking on the notion in information science. Journal of the American Society for Information Science. Saracevic reviews the thinking on relevance and interprets its different views within a framework of communication, treating relevance as a measure of the effectiveness of contact between a source and a destination.
• Sparck Jones, K. (1974). Automatic indexing: a state-of-the-art report. Journal of Documentation. Sparck Jones reviews automatic indexing from 1968 to 1973, covering document description, search manipulation and index-language generation. It shows the indexing half of retrieval rested on comparative experiment, so evaluation and ranking meet in the test collection.
• Voorhees, E. M. (1998). Variations in relevance judgments and the measurement of retrieval effectiveness. Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. Voorhees redid relevance judgements with different assessors and tracked how far effectiveness scores moved, finding the relative ranking of systems comparatively stable even where individual judgements varied. It carries the defence side of the unit's contested question: scores vary, but system comparisons may still be reliable.
• Voorhees, E. M., Samarov, D. and Soboroff, I. (2017). Using replicates in information retrieval evaluation. ACM Transactions on Information Systems. Randomly partitioning the test collection gives replicates, and bootstrap ANOVA extracts the system-topic interactions unavailable without them, yielding a more precise system effect and a narrower confidence interval. It carries the claim that evaluation can be made more sensitive by design.
• Webber, W., Moffat, A. and Zobel, J. (2010). A similarity measure for indefinite rankings. ACM Transactions on Information Systems. The authors propose rank-biased overlap, a measure of similarity between incomplete ranked lists that handles non-conjointness, weights high ranks more heavily than low, and is monotonic with increasing evaluation depth. It carries the modern answer to comparing rankings when the lists are incomplete.
• Zobel, J. (1998). How reliable are the results of large-scale information retrieval experiments? Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. Zobel examines both the pooling of documents for assessment and the use of those assessments in the TREC experiments, and finds the measured relative performance of systems reliable while recall is overestimated because many relevant documents were never found. It carries both sides at once: comparisons are reliable, but an absolute measure such as recall is inflated by pooling.