Offline accuracy on logged data is not evidence of benefit

A team judges a recommender by a number computed on a log of interactions it helped to produce. The offline experiment withholds part of that log, fits a model on the rest and scores it on what was withheld. It rewards agreement with past exposure, and the audits show that much of the reported progress does not survive a correction to the protocol.

The claim is narrow. Offline evaluation is not worthless: a comparison under a declared, fixed protocol can still rule some hypotheses out. The narrower point is that the protocol supplies much of the reported difference, and that the difference comes in units no user experiences. The fifth of ten units, this one prices the offline protocol. The sixth unit asks whose interest the single number beside the word accuracy serves.

The protocol, not the algorithm, decides the answer

Gunawardana and Shani (2015) set out the design space: offline experiments with no user interaction, user studies where a small group reports on the experience, and large-scale online experiments with a real population. For each they describe the questions it can answer and the protocols that make an answer trustworthy.

Meng and colleagues (2020) compared three common strategies for splitting interactions into training and test data, across seven state-of-the-art models on two datasets, and found the split an important confounding variable that can markedly alter the ranking of the models.

Krichene and Rendle (2020) found a sharper defect in a widespread shortcut: scoring ranking metrics on a random sample of the catalogue rather than the whole of it. Sampled metrics are inconsistent with their exact versions and do not preserve relative statements such as one recommender being better than another, not even in expectation.

Canamares, Castells and Moffat (2020) walk the whole chain of an offline experiment, from filtering and splitting the ratings, through selecting users and handling missing scores, to the scoring of empty recommendations. Varying the split, metric, target items or treatment of empty recommendations produced different or opposite outcomes.

Valcarce and colleagues (2020) turn the metric choice into a measured object, comparing ranking metrics for top-N recommendation on two properties: robustness to sparsity and popularity bias, and the power to separate algorithms. Precision was the most robust and normalised discounted cumulative gain the most discriminating, so the two properties did not sit in one metric.

The audits made the criticism standing

Ferrari Dacrema, Cremonesi and Jannach (2019) turned the worry into a measurement. Of eighteen neural recommendation algorithms from top-level conferences, the team could reproduce only seven with reasonable effort. Simple heuristic methods such as nearest-neighbour or graph-based approaches can often outperform six of those seven. The seventh was not consistently better than a well-tuned non-neural linear method.

Ferrari Dacrema and colleagues (2021) widened the audit. Conceptually simple methods can outperform eleven of the twelve reproducible neural approaches, and none was consistently better than matrix factorisation or linear models. The audits do not show neural recommendation is useless; they show the size of the advantage depends on the baselines and the tuning.

A more recent audit widens the target from the algorithm to the setup. Hidasi and Czapp (2023) catalogue four flaws they find widespread across a decade of published evaluation, adding dataset-task mismatch and information leaking through time to the sampling problems already named, and show that each can change the ordering of models. The flaws spread because later work copies an earlier setup without rechecking it, so the criticism attaches to practice rather than to any one paper. The protocol also decides who is in the log, so a reader without the device or stable connection it assumes leaves no trace.

An accuracy metric rewards what people have already chosen

Canamares and Castells (2018) find a bias towards items many people have already chosen, and model when popularity is a genuine signal and when it is an artefact of measurement. They test the predictions on a crowdsourced dataset built to lack the usual biases, where the accuracy a conventional offline setting measures contradicts the accuracy observable without the bias.

Cremonesi, Koren and Turrin (2010) show the practical face of this on the top-N task: algorithms tuned to minimise rating error do not perform as expected on a ranked list, where a naive non-personalised algorithm can outperform some common approaches and almost match sophisticated ones.

Herlocker and colleagues (2004) gave two papers room to disagree, finding that over one content domain the tested accuracy metrics collapsed into roughly three equivalence classes, correlated within a class and uncorrelated across classes.

A measure built on the user sees what the accuracy figure cannot. Knijnenburg and colleagues (2012) built the machinery an accuracy metric cannot replace, linking objective system aspects to behaviour through perceptual and evaluative constructs. Perceptions of recommendation quality and variety mediated the effect of the system's properties on the user's experience of the process, the system and the outcome.

Reviewing published field tests, Jannach and Jugovac (2019) record which business measures real deployments used, finding the literature scattered and open questions remaining about quantifying business effects.

The contested question: can the offline protocol be repaired, or is the gap structural?

The first position treats the protocol as repairable. Canamares, Castells and Moffat (2020) enumerate the choice points and exhibit the pitfalls, which presupposes that a study declaring and justifying each choice is defensible. Gunawardana and Shani (2015) argue that a well-designed offline experiment answers a narrow question well and locate the error in inferences drawn beyond it.

The second treats the gap as structural. Beel and colleagues (2013) evaluated a set of recommendation approaches under both offline and online protocols and found the results often contradicted each other, with offline frequently failing to predict the online outcome. They name the ignorance of human factors and argue it may never be determinable when those factors matter, concluding that offline evaluation may be inappropriate for research paper recommenders in many settings.

Cremonesi and colleagues (2010) treat the strength of non-personalised baselines as an artefact that biases top-N results, and warn practitioners to choose the test set so as not to favour them. Canamares and Castells (2018) argue popularity can be a genuine signal under identifiable conditions, so a reader must separate real effectiveness from measurement bias rather than assume it.

What would settle it is a pre-registered study that runs the same algorithms under a fully declared offline protocol and then in a controlled online experiment on the same population, reporting the offline-online correlation per design choice. Settling the popularity dispute needs datasets with a known exposure process. Until then the honest report of an offline result is a result about the log.

The repair position rests on experiments that vary the choice points of a protocol (Canamares, Castells and Moffat, 2020) and a handbook account that separates the regimes by the question each answers (Gunawardana and Shani, 2015). The structural position rests on one comparative study that ran the same approaches offline and online and often found them contradictory (Beel et al., 2013). While the dispute is open, declare the split, the metric and the aggregation, and treat an offline win as a hypothesis for an online test. The strongest objection is that the structural claim rests on one domain, so the unit generalises it beyond its evidence.

Where this is heading

The audits ask for consistent baselines and comparable tuning and work as a template for checking a claim (Ferrari Dacrema et al., 2019, 2021), Canamares, Castells and Moffat (2020) supply the components a reader can inspect, and Gunawardana and Shani (2015) set out which regime answers which question.

Sources

• Herlocker, J. L., Konstan, J. A., Terveen, L. G. and Riedl, J. T. (2004). Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems. doi:10.1145/963770.963772 Reviewing the decisions that make up a collaborative-filtering evaluation, they observed that recommenders had been evaluated in many incomparable ways. In one content domain the tested accuracy metrics collapsed into roughly three equivalence classes, correlated within a class and uncorrelated across classes, so two studies can report different winners.
• Beel, J., Genzmehr, M., Langer, S., Nurnberger, A. and Gipp, B. (2013). A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation. Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation. doi:10.1145/2532508.2532511 Evaluated a set of approaches under both offline and online protocols and found the results often contradicted each other, with offline frequently failing to predict the online outcome. The authors attribute this to ignored human factors, argue it will probably never be determinable when those matter, and conclude offline evaluation may be inappropriate for research paper recommenders in many settings.
• Ferrari Dacrema, M., Cremonesi, P. and Jannach, D. (2019). Are we really making much progress? A worrying analysis of recent neural recommendation approaches. Proceedings of the 13th ACM Conference on Recommender Systems. doi:10.1145/3298689.3347058 Attempted to reproduce eighteen neural recommendation algorithms from top-level conferences; only seven reproduced with reasonable effort. Six of those seven can often be outperformed by simple heuristic methods, and the seventh was not consistently better than a well-tuned non-neural linear method. A suspicion about research practice became a reproducible audit.
• Meng, Z., McCreadie, R., Macdonald, C. and Ounis, I. (2020). Exploring Data Splitting Strategies for the Evaluation of Recommendation Models. Fourteenth ACM Conference on Recommender Systems. doi:10.1145/3383313.3418479 Compared three splitting strategies across seven models on two datasets and found no standard strategy in use. The split acted as an important confounding variable that can markedly alter the ranking of the models, making much published work non-comparable even when datasets and metrics are fixed. The error sits beneath the metric.
• Krichene, W. and Rendle, S. (2020). On Sampled Metrics for Item Recommendation. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. doi:10.1145/3394486.3403226 Showed that ranking metrics computed on a random sample of the catalogue are inconsistent with their exact versions: they do not preserve relative statements such as one recommender being better than another, not even in expectation. Smaller samples make metrics converge until they collapse to one measure, so the authors recommend avoiding sampling or correcting it.
• Ferrari Dacrema, M., Boglio, S., Cremonesi, P. and Jannach, D. (2021). A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research. ACM Transactions on Information Systems. doi:10.1145/3434185 Extended the audit to recent neural collaborative-filtering approaches and found eleven of the twelve reproducible methods can be outperformed by conceptually simple alternatives, and that none was consistently better than matrix factorisation. The works audited appeared between 2015 and 2018. Repeating the finding makes the critique a property of practice.
• Gunawardana, A. and Shani, G. (2015). Evaluating Recommender Systems. Recommender Systems Handbook. doi:10.1007/978-1-4899-7637-6_8 Sets out the evaluation design space, separating offline experiments from user studies and large-scale online experiments, and describes for each the questions it can answer and the protocols that support a trustworthy conclusion. The chapter supports offline testing for the narrower question it fits, which is what the criticism turns on.
• Jannach, D. and Jugovac, M. (2019). Measuring the Business Value of Recommender Systems. ACM Transactions on Management Information Systems. doi:10.1145/3370082 Reviews published field tests of recommender systems and reports which business-related measures were used in real deployments, finding the literature scattered and open questions remaining about the realistic quantification of business effects. The authors also question the value of algorithmic improvements and offline experiments as commonly run.
• Canamares, R., Castells, P. and Moffat, A. (2020). Offline evaluation options for recommender systems. Information Retrieval Journal. doi:10.1007/s10791-020-09371-3 Examines each step of an offline experiment, from filtering and splitting the ratings, through selecting users and handling missing scores, to the scoring of condensed lists and the weighting of significance tests. Experiments across those choice points show that varying the split, metric, target items or treatment of empty recommendations can give different or even opposite outcomes, which makes a defensible protocol a matter of declared choices.
• Valcarce, D., Bellogin, A., Parapar, J. and Castells, P. (2020). Assessing ranking metrics in top-N recommendation. Information Retrieval Journal. doi:10.1007/s10791-020-09377-x Compared ranking metrics for top-N recommendation on robustness to sparsity and popularity bias and on discriminative power, finding precision the most robust and normalised discounted cumulative gain the most discriminating, and the geometric mean more robust than the arithmetic mean for cold users. A metric can be stable or separating, rarely both.
• Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H. and Newell, C. (2012). Explaining the user experience of recommender systems. User Modelling and User-Adapted Interaction. doi:10.1007/s11257-011-9118-4 Proposes a user-centric framework linking objective system aspects to behaviour through perceptual and evaluative constructs and validates it with four field trials and two controlled experiments analysed by structural equation modelling. Perceptions of recommendation quality and variety mediated the effect of the system's properties on the user's experience of the process, the system and the outcome.
• Canamares, R. and Castells, P. (2018). Should I follow the crowd? A probabilistic analysis of the effectiveness of popularity in recommender systems. The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. doi:10.1145/3209978.3210014 Models the conditions under which recommending popular items is genuinely effective and when it is instead an artefact of experimental bias, through dependencies between whether an item is rated, discovered and relevant. On a crowdsourced dataset built to lack the usual biases, the accuracy a conventional offline setting measures contradicts the accuracy observable without that bias.
• Cremonesi, P., Koren, Y. and Turrin, R. (2010). Performance of recommender algorithms on top-n recommendation tasks. Proceedings of the fourth ACM conference on Recommender Systems. doi:10.1145/1864708.1864721 Shows that algorithms optimised to minimise rating error do not perform as expected on the top-N task, where a short ranked list is what the user sees. A naive non-personalised algorithm can outperform some common approaches and almost match sophisticated ones, and a few popular items can skew performance, so the test set must be chosen carefully.
• Hidasi, B. and Czapp, A. T. (2023). Widespread Flaws in Offline Evaluation of Recommender Systems. Proceedings of the 17th ACM Conference on Recommender Systems. doi:10.1145/3604915.3608839 Catalogued four flaws widespread across a decade of published evaluation, among them dataset-task mismatch and information leaking through time, and showed that each can change the ordering of models. The flaws spread because later work copies an earlier setup without rechecking it, and the paper proposes no complete replacement protocol.