Evaluating with expensive ground truth: benchmarks, metrics and red teams

An evaluation is an argument that a system will work somewhere else, and its evidence is almost always a sample: items with known answers, or judgements collected from people. The score that comes out of it is portable in a way the sample is not. This eighth of the ten units takes how a system is evaluated. Benchmarks and automatic metrics measure agreement with a reference sample drawn from one distribution, so they can be optimised without competence in the setting the product runs in. Honest evaluation therefore needs construct validity and adversarially generated cases. A red-team report which finds nothing is an existence proof about the search that was run, not an estimate of safety.

A benchmark score is evidence about the test set

Raji and colleagues (2021) observe that a small collection of influential benchmarks stands in across subfields for a range of general problems, so state-of-the-art performance on them is read as progress towards goals they were never designed to measure.

Bowman and Dahl (2021) sharpen it for one task family. Evaluation for natural language understanding is broken, they argue, because unreliable systems score so highly that groups who build better ones cannot show an improvement. They set out four criteria and find most benchmarks failing them. They add that abandoning in-distribution benchmarks for adversarially constructed out-of-distribution sets guarantees poor performance while obscuring the abilities the benchmark was meant to measure.

Their question has no safe default: over which population of items was the score computed, and is that population the one the product will meet? Gururangan and colleagues (2018) trained a text classifier on the hypothesis sentence alone, without the premise, in two inference datasets built by crowdworkers. The classifier recovered the label in about 67 per cent of SNLI and 53 per cent of MultiNLI cases.

A metric holds only for the task it was built for

Reiter (2018) states what bounds a metric's validity. He reviewed 284 correlations reported across 34 papers as evidence on whether BLEU measures the utility and user satisfaction of language processing systems. He concluded that it is supported for diagnostic machine translation evaluation, what it was proposed for, and not outside it.

Liu and colleagues (2016) ran the equivalent test for dialogue, correlating metrics imported from machine translation with human judgements of dialogue responses. The correlations were very weak in the Twitter domain and absent in the technical Ubuntu domain.

Novikova and colleagues (2017) supply the distinction: word-based and grammar-based metrics only weakly reflect human judgements of individual outputs, and their performance is specific to data and system, so they remain reliable only for ranking systems.

Sai and colleagues (2022) group metrics by what they compare and what they assume.

The score is also a product of its construction and its incentives

The measurement problem starts before the metric does. Blodgett and colleagues (2020) surveyed 146 papers analysing bias in language processing and found their motivations often vague and their techniques poorly matched to those motivations. A quantity whose meaning is unfixed has nothing for validation to be against.

Incentives do the rest. Ethayarajh and Jurafsky (2020) show that leaderboard competition under-invests in the qualities practitioners value, including compactness, fairness and energy efficiency. What is measured is what is optimised.

Bender and colleagues (2021) extend the point to the material the evaluation sits on. They document the cost of training large models, the biases of web-derived data, and the difficulty of auditing systems whose datasets cannot be documented.

Human judgement is the ground truth and it is not gold

When the target is human judgement, the ground truth is a measuring instrument too. Clark and colleagues (2021) asked non-experts to distinguish human- from machine-authored text in stories, news and recipes, and found them at chance level on GPT-3 output. Training improved accuracy up to 55 per cent, but not significantly across all three domains, and the reasons given were often contradictory.

van der Lee and colleagues (2021) find substantial variation in design, annotator recruitment, instructions and reporting across published human evaluations, and set out best-practice guidelines. A reported human evaluation is interpretable only when those choices are stated.

Chiang and colleagues (2024) show the instrument built at scale. Chatbot Arena collects pairwise preference judgements from a large user base, and had over 240,000 votes at the time of writing. It reports both the statistics that turn preference into a ranking and checks that its questions are diverse and that crowd votes agree well with expert raters. That crowd is a sample of whoever had the bandwidth to take part.

A clean red-team report is a statement about the search

Adversarial generation attacks the static sample directly. Kiela and colleagues (2021) built Dynabench, in which annotators write examples a target model misclassifies but another person does not, so the test set moves as models improve.

The same logic produces the unit's claim about red teams. A successful probe shows a failure exists, so it is an existence proof. A probe that finds nothing licenses only a statement about the search. Ganguli and colleagues (2022) ran such an effort and found reinforcement learning from human feedback made their models harder to red team by non-adversarial attempts. Nothing in this unit's list measures the coverage of an adversarial search in an interface, so the red-team claim remains an argument, not a rate.

Contested question: does adversarial construction repair a broken benchmark?

Both papers agree about the diagnosis, and they disagree about the repair. Kiela and colleagues (2021) hold that dynamic adversarial benchmarks answer the critical need, because models reach outstanding scores on static tasks while failing on simple challenge examples. Bowman and Dahl (2021) hold the opposite. Adversarial collection does not address the causes of benchmark failure, so the repair lies in dataset design, annotation reliability, size and the handling of social bias.

What would settle it is a comparison neither paper runs. Take a named deployment setting and a fixed task, build a static and a dynamic adversarial benchmark for it, and ask which predicts performance in the setting.

A second contest sits underneath the first: whether a crowd is a usable judge. Clark and colleagues (2021) found untrained non-experts at chance with contradictory justifications, while Chiang and colleagues (2024) report crowd votes in good agreement with expert raters. A same-or-different judgement on one paragraph is not a pairwise preference between two systems on a chosen prompt.

What to do while the dispute is open

Both sides of the main dispute are position papers, so the disagreement is argued rather than measured. Kiela and colleagues (2021) report a platform and its use; Bowman and Dahl (2021) run no experiment either, and their four criteria are stated rather than validated. The second dispute is better evidenced and splits by task, because one experiment found non-experts at chance while a platform reports crowd votes agreeing with expert raters. The strongest objection to this unit's thesis is that its own recommendation has the same standing. The unit asks for construct validity and adversarial cases, and no study here shows that either improves performance in the setting a product runs in. While the dispute is open, a designer should report the population of items a score came from, keep an adversarial set beside the static one, and read a clean red-team report as a statement about the search rather than about the system.

Where the argument is heading

The direction is towards evaluation as a stated argument rather than a reported number. Gehrmann, Clark and Sellam (2023) surveyed the flaws in human and automatic evaluation of generated text, and in its datasets, pointed out over the past 20 years. They found the central problem is not that the flaws are unknown but that improved approaches are rarely widely adopted. They then scored 66 generation papers from recent conferences against their own recommendations.

The sources this unit rests on

• Bender, E. M., Gebru, T., McMillan-Major, A., et al. (2021). On the Dangers of Stochastic Parrots. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3442188.3445922 Documents the cost of training large language models, the biases of web-derived data and the difficulty of auditing systems whose datasets are undocumented. It grounds the unit's point that an evaluation inherits the history of its data.
• Blodgett, S. L., Barocas, S., III, H. D., et al. (2020). Language (Technology) is Power: A Critical Survey of "Bias" in NLP. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.485 Surveyed 146 papers analysing bias in language processing and found their motivations often vague and their techniques poorly matched. It shows the measurement problem begins in an unstated definition.
• Bowman, S. R. and Dahl, G. (2021). What Will it Take to Fix Benchmarking in Natural Language Understanding? Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. doi:10.18653/v1/2021.naacl-main.385 Argues that improvement cannot be shown on benchmarks that unreliable systems already score highly on, and that most benchmarks fail four stated criteria, while adversarial sets obscure the abilities benchmarks should measure.
• Chiang, W., Zheng, L., Sheng, Y., et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv preprint arXiv:2403.04132. doi:10.48550/arXiv.2403.04132 Ranks language models by pairwise human preference from a large user base, with over 240,000 votes at the time of writing, and reports that crowd votes agree with expert raters. It is the counterweight to the finding that untrained evaluators cannot judge text.
• Clark, E., August, T., Serrano, S., et al. (2021). All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). doi:10.18653/v1/2021.acl-long.565 Found non-experts distinguishing human- from machine-authored text at chance on GPT-3 output, with training gains of up to 55 per cent not significant across domains. It undermines the assumption that an untrained crowd is a ground truth.
• Ethayarajh, K. and Jurafsky, D. (2020). Utility is in the Eye of the User: A Critique of NLP Leaderboards. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). doi:10.18653/v1/2020.emnlp-main.393 Shows that leaderboard competition under-invests in the qualities practitioners value, including compactness, fairness and energy efficiency, because the benchmark does not score them. It explains why a poor metric persists: what is measured is what is optimised.
• Gehrmann, S., Clark, E. and Sellam, T. (2023). Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text. Journal of Artificial Intelligence Research. doi:10.1613/jair.1.13715 Surveys the flaws in human and automatic evaluation of generated text pointed out over 20 years and reports that improved approaches are rarely adopted. The authors score 66 recent papers against their own recommendations.
• Kiela, D., Bartolo, M., Nie, Y., et al. (2021). Dynabench: Rethinking Benchmarking in NLP. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. doi:10.18653/v1/2021.naacl-main.324 Introduces a platform in which annotators write examples a target model misclassifies but another person does not, so the test set moves as models improve, and argues that static benchmarks saturate. It is the repair the unit examines.
• Liu, C. W., Lowe, R., Serban, I., et al. (2016). How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. doi:10.18653/v1/d16-1230 Correlated metrics imported from machine translation with human judgements of responses and found the correlations very weak in Twitter and absent in Ubuntu. It is the empirical case of a metric uncorrelated with the judgement it represents.
• Novikova, J., Dusek, O., Curry, A. C., et al. (2017). Why We Need New Evaluation Metrics for NLG. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. doi:10.18653/v1/d17-1238 Found word-based and grammar-based metrics only weakly reflect human judgements of individual outputs, and that metric performance is data- and system-specific, while remaining reliable for ranking systems. It supplies the distinction the unit uses.
• Raji, I. D., Bender, E. M., Paullada, A., et al. (2021). AI and the Everything in the Whole Wide World Benchmark. arXiv preprint arXiv:2111.15366. doi:10.48550/arXiv.2111.15366 Argues that a small set of influential benchmarks functions as a stand-in for general capability, so benchmark performance is treated as evidence of progress towards goals it was never designed to measure. It states the unit's thesis generally.
• Ganguli, D., Lovitt, L., Kernion, J., et al. (2022). Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviours, and Lessons Learned. arXiv preprint arXiv:2209.07858. doi:10.48550/arXiv.2209.07858 A red-teaming effort across three model sizes and four training regimes found reinforcement learning from human feedback made models harder to red team by non-adversarial attempts.
• Gururangan, S., Swayamdipta, S., Levy, O., et al. (2018). Annotation Artefacts in Natural Language Inference Data. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). doi:10.18653/v1/n18-2017 The crowdworker protocol for building inference data leaves clues in the hypotheses, so a classifier reading the hypothesis alone identified the label in about 67 per cent of SNLI cases and 53 per cent of MultiNLI cases.
• Reiter, E. (2018). A Structured Review of the Validity of BLEU. Computational Linguistics. doi:10.1162/coli_a_00322 Reviews 284 correlations reported across 34 papers as evidence on whether BLEU measures the utility and user satisfaction of language processing systems, and concludes it is supported for diagnostic machine translation and not elsewhere. A metric's validity is bounded by task and population.
• Sai, A. B., Mohankumar, A. K. and Khapra, M. M. (2022). A Survey of Evaluation Metrics Used for NLG Systems. ACM Computing Surveys. doi:10.1145/3485766 Surveys automatic evaluation metrics for natural language generation with a taxonomy grouping them by what they compare and assume. It shows how metric proliferation outran validation, so a favourable metric needs justifying.
• van der Lee, C., Gatt, A., van Miltenburg, E., et al. (2021). Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech & Language. doi:10.1016/j.csl.2020.101151 Reviews how human evaluation of generated text is conducted, finding substantial variation in design, annotator recruitment, instructions and reporting, and sets out best-practice guidelines. It is the standard for judging whether a human evaluation is interpretable.