Accuracy is a single-stakeholder metric wearing a neutral name

A recommender usually carries one headline number with a plain word beside it: accuracy. This unit argues that the word is not neutral. Accuracy measures agreement with what one party did, so a system tuned to it answers a question that already names a beneficiary. When the metric set holds only that number, the design has decided whose interest the system serves, in the evaluation code rather than in the product meeting.

The claim is narrower than it sounds. Accuracy is not a bad measure: a list that surfaces nothing relevant serves nobody. What the unit claims is that accuracy is one objective among several, that the others belong to other parties, and that the relations between them are measurable. The sixth of ten units, this one reads the instruments the field built to express competing objectives, and the results that show what an aggregate hides. The seventh unit asks whether the narrowing personalisation is blamed for sits in what platforms expose or in what people select.

A metric is a claim about what counts as good

Vargas and Castells (2011) found that measures for novelty and diversity had no consolidated ground. They took different forms, their relations were unstudied, and most ignored an item's rank and its relevance. They defined rank-aware and relevance-aware measures so that a team could compute the competing qualities on the same ranked list as accuracy.

Castells, Hurley and Vargas (2015) show how far the redefinition had travelled by the middle of the decade. Their handbook chapter gathers the novelty and diversity work into one account, and draws a distinction the unit needs. The diversity of a single person's list is a different quantity from the aggregate diversity of the catalogue. The latter asks how much of the whole inventory the system ever exposes. The two arise from the same recommendations and can move apart, since a list tuned to one user can leave the catalogue untouched. The chapter is a synthesis rather than a new experiment, so it shows that the quantities exist and are measurable, not that either improves a user's experience.

A single figure cannot see a group failure or a feedback loop

Ekstrand and colleagues (2018) evaluated popularity-based recommendation by demographic group. They found that popularity bias disadvantaged particular groups, while the disadvantage was invisible in aggregate accuracy, and the categories a study cuts by decide whether an older user or a disabled user is visible at all. Zhao and colleagues (2018) ran a field experiment with 1,508 users over at least a month, comparing six algorithms and measuring both activity and perception. The implicit-optimised model engaged users more than the rating-optimised one, which the authors read as the historical reason the field moved to implicit feedback. It also raised negative engagement, which correlated with lower satisfaction.

Two results show why the receiver's own long-run interest is not captured by the engagement number either. Chaney, Stewart and Engelhardt (2018) simulated a recommender that both shapes and learns from user behaviour. Training on data confounded in this way homogenises behaviour without improving utility. Jiang and colleagues (2019) analysed the dynamics, separating the echo chamber from the filter bubble within one model of user and system, and offered measures to slow the degeneration. A system optimised for the measured signal can tighten the loop that produces it.

The score can satisfy one party and worsen another

Sonboli and colleagues (2022) set out the multi-sided structure directly. A recommender sits between parties, job applicants and employers, or artists and listeners, and satisfying one side's fairness criterion can worsen another's. Milano, Taddeo and Floridi (2020) reached the same gap from the ethical literature. They identify six areas of concern, and observe that user-centred accounts consider the receiver of a recommendation and leave other stakeholders out. Ekstrand and colleagues (2022) place this inside information access generally. They name what complicates any transfer of fairness definitions from classification: the multi-stakeholder setting, the rank-based problem, the centrality of personalisation and the role of user response.

The provider side is clearest in what recommendation does to the long tail. Abdollahpouri, Burke and Mobasher (2017) showed that learning-to-rank algorithms amplify popularity bias, so the systems recommend popular items frequently and the rest rarely. They then introduced a regularisation framework that controls how much exposure goes to popular and long-tail items. On two datasets it improved long-tail coverage without substantial loss of ranking performance.

Vrijenhoek and colleagues (2021) make the conflict explicit by importing an institution's objectives. Assessing news recommenders against the diversity newsrooms want, they argue that current evaluation measures clicks and short-term engagement. It does not measure a reader's longer-term interest in diverse and important information. Once an editor rather than a click stream states what a good recommendation is, the metric set is visibly a decision about values, taken outside the engineering team.

The contested question: is the objective conflict structural or tunable?

The first position treats the conflict as structural. Kleinberg, Mullainathan and Raghavan (2016) formalise three fairness conditions for a risk score. They prove that no method satisfies all three simultaneously except in highly constrained cases. Even approximate satisfaction requires the data to lie in an approximate special case. Chaney and colleagues (2018) supply the recommender instance, where optimising on confounded data raises homogeneity without raising utility. Sonboli and colleagues (2022) report the multi-sided version, in which one party's criterion is met at another's expense. On this reading no better formula repairs the metric set, because any single scalar encodes one party's preference.

The second position treats the conflict as a tuned parameter. Abdollahpouri and colleagues (2017) show a regulariser that trades accuracy against coverage continuously, improving long-tail coverage at little cost to ranking quality on two datasets. Beutel and colleagues (2019) built a fairness metric from pairwise comparisons in randomised experiments, and a regulariser trained on it. They applied it in a large-scale production recommender, and improved pairwise fairness. If the cost of the intervention is small and measurable, the metric set is a dial a team turns rather than a verdict on whose interests the system serves.

A dial still has a default, and the default is a statement about whose interest the shipped system serves; the tunable position claims only that the alternative is affordable. What would settle it is a shared benchmark. It would report the utility cost of each diversity or fairness intervention, alongside the stakeholder objectives that intervention improves. Deldjoo and colleagues (2023) find such reporting missing after reviewing more than 160 publications. It would also need field measurement of outcomes beyond engagement, for which Zhao and colleagues (2018) provide one comparison rather than a standard. Until then, an accuracy figure is a report of one party's interest with the rest left unmeasured.

The two positions rest on different kinds of evidence. The structural position rests on a formal impossibility proof (Kleinberg, Mullainathan and Raghavan, 2016). It rests on a simulation of a confounded feedback loop (Chaney, Stewart and Engelhardt, 2018). The tunable position rests on two measured interventions. One is a regulariser evaluated on two datasets (Abdollahpouri, Burke and Mobasher, 2017). The other is a fairness regulariser deployed in a large-scale production recommender (Beutel et al., 2019). The deployed case is the stronger evidence that a cost is affordable. While the dispute is open, report the utility cost of each intervention beside the stakeholder it helps. The strongest objection is that the thesis and the tunable position agree, so the dispute is about affordability.

Where this is heading

The direction indicated by this list is towards measurement that names its beneficiary. Stray and colleagues (2024) synthesise an interdisciplinary programme. They set out what it requires: processes for stating values, values-driven measurements that reach past engagement, optimisation for long-term outcomes, and causal inference of recommender effects. Deldjoo and colleagues (2023) call for interdisciplinary work because the fairness literature operationalises abstract problems while leaving the normative claim unstated. Ekstrand and colleagues (2022) frame shared open problems for information access, and Vrijenhoek and colleagues (2021) show a newsroom held to its stated objectives. The open question is whether an evaluation will report the interests a recommender serves and the interests it costs, or continue to report one number under a neutral name.

Sources

• Vargas, S. and Castells, P. (2011). Rank and relevance in novelty and diversity metrics for recommender systems. Proceedings of the fifth ACM conference on Recommender systems. doi:10.1145/2043932.2043955 Found that novelty and diversity metrics had no consolidated ground and that most ignored rank and relevance, then defined rank-aware measures for both. That makes the competing objectives comparable with accuracy on one list.
• Castells, P., Hurley, N. J. and Vargas, S. (2015). Novelty and Diversity in Recommender Systems. Recommender Systems Handbook. doi:10.1007/978-1-4899-7637-6_26 Gathered the novelty and diversity work into one account and separated the diversity of one user's list from the aggregate diversity of the catalogue. The two are computed on the same recommendations and can move apart. A synthesis rather than an experiment.
• Ekstrand, M. D., Tian, M., Azpiazu, I. M., Ekstrand, J. D., Anuyah, O., McKitrick, M. S. and Pera, M. S. (2018). All the cool kids, how do they fit in? Popularity and demographic biases in recommender evaluation and effectiveness. Proceedings of the Conference on Fairness, Accountability and Transparency. https://proceedings.mlr.press/v81/ekstrand18b.html Evaluated popularity-based recommendation by demographic group and found that the bias disadvantaged particular groups while the disadvantage was invisible in aggregate accuracy. A population average can conceal a group-level failure.
• Zhao, Q., Harper, F. M., Adomavicius, G. and Konstan, J. A. (2018). Explicit or implicit feedback? engagement or satisfaction? Proceedings of the 33rd Annual ACM Symposium on Applied Computing. doi:10.1145/3167132.3167275 Ran a field experiment with 1,508 users over at least a month, comparing six algorithms and collecting both activity and perception. The implicit-optimised model engaged users more but also raised negative engagement, which correlated with lower satisfaction.
• Sonboli, N., Burke, R., Ekstrand, M. and Mehrotra, R. (2022). The multisided complexity of fairness in recommender systems. AI Magazine. doi:10.1002/aaai.12054 Review multi-sided fairness in recommendation, using employment listings and music as settings, and show that satisfying one side's criterion can worsen another's. A single-party metric is the one-party case of a multi-party problem.
• Milano, S., Taddeo, M. and Floridi, L. (2020). Recommender systems and their ethical challenges. AI and Society. doi:10.1007/s00146-020-00950-y Review the ethical challenges of recommender systems systematically, identify six areas of concern, and observe that user-centred approaches assess the receiver while leaving other stakeholders out. That gap is the ethical form of the unit's thesis.
• Ekstrand, M. D., Das, A., Burke, R. and Diaz, F. (2022). Fairness in information access systems. Foundations and Trends in Information Retrieval. doi:10.1561/1500000079 Present a taxonomy of fair information access and name what complicates borrowing fairness definitions from classification: multi-stakeholder settings, the rank-based problem, the centrality of personalisation and the role of user response. The monograph places the debate inside a wider argument about what search is for.
• Abdollahpouri, H., Burke, R. and Mobasher, B. (2017). Controlling Popularity Bias in Learning-to-Rank Recommendation. Proceedings of the Eleventh ACM Conference on Recommender Systems. doi:10.1145/3109859.3109912 Show that learning-to-rank recommenders amplify popularity bias, then introduce a regularisation framework controlling the balance between popular and long-tail exposure; on two datasets it improved long-tail coverage without substantial loss of ranking performance. The bias is a property of the objective, and its correction is a dial.
• Chaney, A. J. B., Stewart, B. M. and Engelhardt, B. E. (2018). How algorithmic confounding in recommendation systems increases homogeneity and decreases utility. Proceedings of the 12th ACM Conference on Recommender Systems. doi:10.1145/3240323.3240370 Simulate a recommender that both shapes and learns from user behaviour and show that training on data confounded in this way homogenises behaviour without improving utility. The long-run cost is a property of the feedback loop.
• Jiang, R., Chiappa, S., Lattimore, T., Gyorgy, A. and Kohli, P. (2019). Degenerate Feedback Loops in Recommender Systems. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. doi:10.1145/3306618.3314288 Analyse the feedback loop theoretically, separating user dynamics from system behaviour and disentangling the echo chamber from the filter bubble, and offer measures to slow degeneration. The paper gives the unit its mechanism for harm that no single ranking shows.
• Vrijenhoek, S., Kaya, M., Metoui, N., Moller, J., Odijk, D. and Helberger, N. (2021). Recommenders with a mission: assessing diversity in news recommenders. Proceedings of the 2021 Conference on Human Information Interaction and Retrieval. doi:10.1145/3406522.3446019 Assess news recommenders against the diversity newsrooms want and argue that current evaluation measures clicks and short-term engagement rather than longer-term interest in diverse and important information. The editorial setting turns the objective conflict into a stated institutional decision.
• Kleinberg, J., Mullainathan, S. and Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807. doi:10.48550/arXiv.1609.05807 Formalise three fairness conditions for a probabilistic risk score and prove that no method satisfies all three simultaneously except in highly constrained cases. The impossibility result is the structural position in the contested question.
• Beutel, A., Chen, J., Doshi, T., Qian, H., Wei, L., Wu, Y. et al. (2019). Fairness in Recommendation Ranking through Pairwise Comparisons. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. doi:10.1145/3292500.3330745 Build a fairness metric from pairwise comparisons in randomised experiments and a regulariser trained on it, and apply it in a large-scale production recommender, improving pairwise fairness. The deployed case is the tunable position's strongest evidence.
• Deldjoo, Y., Jannach, D., Bellogin, A., Difonzo, A. and Zanzonelli, D. (2023). Fairness in recommender systems: research landscape and future directions. User Modelling and User-Adapted Interaction. doi:10.1007/s11257-023-09364-z Review more than 160 publications on fairness in recommendation and find many works using abstract problem formulations while leaving the normative claim undiscussed. The missing shared benchmarks and stated values are what would have to change for the contested question to be settled.
• Stray, J., Halevy, A., Assar, P., Hadfield-Menell, D., Boutilier, C., Ashar, A. et al. (2024). Building Human Values into Recommender Systems: An Interdisciplinary Synthesis. ACM Transactions on Recommender Systems. doi:10.1145/3632297 Synthesise theory and practice from several disciplines to ask how recommenders can enact the values of the people they serve, and set out open problems including multi-stakeholder processes for resolving trade-offs, values-driven measurement and causal inference of recommender effects. The programme is the constructive form of the unit's closing claim.