A model is a fitted guess: data, generalisation and distribution shift

This second of the ten units takes the model behind the interface. A machine learning model's competence is a property of the distribution its training data was drawn from, not of its code. The failure that reaches the interface is a confident answer that is wrong for this person in this setting, and the design settles it long before the interface, in collection and labelling choices the people building the feature rarely see. Documenting them is part of the interface's supply chain, not a preliminary to it.

Generalisation, not accuracy, is the product

Domingos (2012), setting out twelve lessons for applied machine learning, makes generalisation rather than training accuracy the goal. A model that scores well where it was trained and poorly anywhere else memorised the wrong thing; one that generalises learned a property surviving a change of sample. Quinonero-Candela and colleagues (2009) name the assumption made at deployment, that training and test data share a distribution, and argue it is the exception, because shift arises from ordinary experimental design, measurement and non-reproducible testing. A model in front of a person is betting on a distribution it has not seen.

Every benchmark is a biased sample of the world

Torralba and Efros (2011) measured the bias in standard object-recognition datasets. A classifier trained on one dataset and tested on another performs poorly, so well known datasets are distinguishable from each other and from the visual world. Shankar and colleagues (2017) found an observable Amerocentric and Eurocentric representation bias in two widely used open datasets, on which classifiers trained poorly for under-represented regions. Recht and colleagues (2019) put a number on the gap. Rebuilding the CIFAR-10 and ImageNet test sets by the original procedures lowered accuracy by 3 to 15 points on CIFAR-10 and 11 to 14 on ImageNet. Gains on the old sets still translated into larger gains on the new ones, so the authors read the drop as generalisation to harder images rather than overfitting.

Shift is the default condition, not an exception

Gama and colleagues (2014) survey concept drift, where the relation between inputs and targets changes over time. Koh and colleagues (2021) assembled WILDS, a benchmark of ten datasets covering shifts across hospitals, camera traps, time and location. The datasets that dominated the field under-represented the shifts deployment produces. Standard training yielded substantially lower out-of-distribution than in-distribution performance. The gap persisted for existing methods built to address shift. The technical answer is domain adaptation. Csurka (2017) surveys the shallow and deep method families for visual applications, a mature toolkit rather than comparative success, and when the domain gap is a sampling frame that omitted half the world adaptation treats a symptom. Zech and colleagues (2018) supply the deployment-side case. They trained pneumonia classifiers on 158,323 chest radiographs from three hospital systems whose prevalence of the finding ranged from 34.2 per cent to 1.0 per cent. Sorting radiographs by hospital system alone reached an AUC of 0.861. The best internally scoring model fell from 0.931 to 0.815 at a third site. A model can learn the site rather than the disease, which the benchmark results above cannot show.

The choices that decide the failure are taken in data collection

Sambasivan and colleagues (2021) interviewed 53 AI practitioners across three continents. They described data cascades: compounding downstream effects triggered by data issues conventional practice does not treat as urgent, and reported consistently that data work is devalued relative to model work, which is why these failures surface late and expensively. Paullada and colleagues (2021) survey the critical literature on dataset collection and use and advocate changes to how datasets are built, documented and evaluated. Together they answer the question a designer should ask of any labelled set: who collected it, under what conditions, and what work was deferred?

Documentation is an interface practice, not a preliminary

If the failure is a collection decision, disclosing that decision is a design artefact. Gebru and colleagues (2021) propose datasheets for datasets. A datasheet records the dataset's motivation, composition, collection process and recommended uses. Bender and Friedman (2018) extend the idea from a dataset's contents to the population it claims to represent. They propose data statements of population, language variety and curation, on the argument that disclosure changes which scientific claims a system can legitimately be said to support. Hutchinson and colleagues (2021) adapt practices from software engineering and infrastructure to the dataset lifecycle. Longpre and colleagues (2024) then measured the scale of the gap, auditing more than 1,800 text datasets on popular hosting sites, tracing their sources, creators and licences, and finding licence omission rates above 70 per cent and error rates above 50 per cent. None of the proposals, though, measures whether a datasheet changes what a team builds, so documentation is an accountability practice whose effect is asserted rather than shown. Sculley and colleagues (2015) apply the notion of technical debt to machine learning systems, cataloguing risk factors including data dependencies, hidden feedback loops and changes in the external world, so a quick model win can carry a large ongoing cost in the data pipeline.

Contested question: is the failure a data problem or a modelling problem?

Three positions are live, and the disagreement is about where the fix belongs.

The first holds that the failure is a data problem and the fix is collection and documentation. Torralba and Efros (2011), Shankar and colleagues (2017), Sambasivan and colleagues (2021), Paullada and colleagues (2021) and Gebru and colleagues (2021) all locate the cause in the sampling frame and the labelling process. On this view no model work repairs a training population that excludes the people who will use the feature.

The second holds that the failure is a modelling problem, addressable by better methods. Csurka (2017) surveys a mature toolkit for adaptation. Recht and colleagues (2019) attribute their drops not to overfitting but to models' inability to generalise to harder inputs, a property of the learner. The objection to the data-first view is that a shift-robust model is a demonstrated option. The practitioners' account of devalued data work is an account of culture, not of where the error is.

The third holds that the dispute cannot yet be settled because the field has not run the comparison. Koh and colleagues (2021) supply the measurement instrument, and it reports accuracy rather than the interface outcome a user meets. No study here varies a data-collection decision and a model-side adaptation on the same deployment, so none measures the joint effect on the error a person actually experiences. What would settle it is that experiment: hold the interface and the deployment population constant, vary one data decision and one adaptation method, and report the error rate on the target population in each.

What to do while the dispute is open

The evidence on the two sides is unequal in kind rather than in volume. The data-first position rests on measured dataset bias and interview studies of practitioners, so it establishes a cause and describes a culture. The modelling position rests on one survey of adaptation methods and one re-collection experiment, and only Zech and colleagues (2018) measure a deployment, at three sites and one disease. The strongest objection to this unit's thesis is that its own remedy is unmeasured. Gebru and colleagues (2021) argue for a datasheet and supply a template, and no study here reports a release that changed because of one, so the prescription asks of practice a measurement it has not made of itself. While the dispute is open, a designer should ask for the error rate on the population the feature will meet and record the sampling frame behind it, rather than accept a score from the training distribution. Ask whether that population reads in a second language or is on a metered connection.

Where the argument is heading

The direction of travel is from a private modelling choice towards a stated, documented fact. WILDS (Koh et al., 2021) made shift a benchmark rather than a footnote. The datasheet, data statement and accountability framework of Gebru and colleagues (2021), Bender and Friedman (2018) and Hutchinson and colleagues (2021) made the dataset a document. Paullada and colleagues (2021) present the critique as settled enough to change practice.

The sources this unit rests on

• Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM. doi:10.1145/2347736.2347755 Domingos sets out twelve lessons, among them that learning decomposes into representation, evaluation and optimisation and that generalisation, not training accuracy, is the goal. It is an exposition of practice rather than a measurement.
• Quinonero-Candela, J., Sugiyama, M., Schwaighofer, A. and Lawrence, N. D. (2009). Dataset Shift in Machine Learning. MIT Press. doi:10.7551/mitpress/9780262170055.001.0001 This volume states the assumption under which a learned model transfers: that training and test distributions are the same. Its argument is that shift is present in most applications, so every deployed interface bets on a distribution.
• Torralba, A. and Efros, A. A. (2011). Unbiased look at dataset bias. IEEE Conference on Computer Vision and Pattern Recognition. doi:10.1109/cvpr.2011.5995347 The authors quantify the bias in several standard object-recognition datasets and show that a classifier trained on one and tested on another performs poorly, so a benchmark result partly measures the test set.
• Shankar, S., Halpern, Y., Breck, E., et al. (2017). No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World. arXiv preprint arXiv:1711.08536. doi:10.48550/arXiv.1711.08536 The authors assess two widely used open image datasets, find an observable Amerocentric and Eurocentric representation bias, and show classifiers trained on them perform poorly for under-represented regions. It links a sampling frame directly to a deployed model's error pattern.
• Recht, B., Roelofs, R., Schmidt, L. and Shankar, V. (2019). Do ImageNet Classifiers Generalise to ImageNet? arXiv preprint arXiv:1902.10811. doi:10.48550/arXiv.1902.10811 The authors rebuild the CIFAR-10 and ImageNet test sets by the original collection procedures and re-evaluate a broad range of models, finding drops of 3 to 15 points on CIFAR-10 and 11 to 14 on ImageNet, which they attribute to harder images rather than overfitting.
• Gama, J., Zliobaite, I., Bifet, A., et al. (2014). A survey on concept drift adaptation. ACM Computing Surveys. doi:10.1145/2523813 The survey covers concept drift in online supervised learning, its adaptation strategies and the shared evaluation conventions. It supplies the field's taxonomy for a change in the relation between inputs and targets, and notes that some drift is a change in the world rather than a defect in the data.
• Koh, P. W., Sagawa, S., Marklund, H., et al. (2021). WILDS: A benchmark of in-the-wild distribution shifts. Proceedings of the 38th International Conference on Machine Learning. https://proceedings.mlr.press/v139/koh21a.html The authors assemble WILDS, a curated benchmark of ten datasets covering shifts across hospitals, camera traps, time and location, because the field's dominant datasets under-represent real deployments. Standard training yielded substantially lower out-of-distribution than in-distribution performance, and the gap persisted for existing shift methods.
• Csurka, G. (2017). Domain Adaptation for Visual Applications: A Comprehensive Survey. arXiv preprint arXiv:1702.05374. doi:10.48550/arXiv.1702.05374 The survey positions domain adaptation within transfer learning for visual applications and describes the shallow and deep method families and their scenarios. It catalogues a mature technical toolkit rather than reporting comparative success.
• Zech, J. R., Badgeley, M. A., Liu, M., et al. (2018). Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Medicine. doi:10.1371/journal.pmed.1002683 Radiographs from three hospital systems taught the model the site rather than the disease, and the best internally scoring model fell from an AUC of 0.931 to 0.815 at a third site.
• Sambasivan, N., Kapania, S., Highfill, H., et al. (2021). "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI. CHI Conference on Human Factors in Computing Systems. doi:10.1145/3411764.3445518 From interviews with 53 practitioners in India, East and West Africa and the USA, the authors describe data cascades, compounding downstream effects triggered by data issues conventional practice does not treat as urgent, and report that data work is devalued relative to model work.
• Paullada, A., Raji, I. D., Bender, E. M., et al. (2021). Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns. doi:10.1016/j.patter.2021.100336 The authors survey the literature critical of dataset collection and use, covering societal harm, filtering and augmentation, and the norms that sustain current practice. It organises an existing critique rather than adding measurements, so the turn is established rather than emerging.
• Gebru, T., Morgenstern, J., Vecchione, B., et al. (2021). Datasheets for datasets. Communications of the ACM. doi:10.1145/3458723 The authors propose datasheets for datasets, by analogy with the electronic-component datasheet, recording a dataset's motivation, composition, collection process and recommended uses. The paper argues the case and offers a template, not evidence of downstream effect.
• Bender, E. M. and Friedman, B. (2018). Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics. doi:10.1162/tacl_a_00041 Bender and Friedman propose data statements for natural language processing, disclosing the population, language variety and curation process behind a dataset. Their argument is that disclosure changes which scientific claims a system can legitimately be said to support.
• Hutchinson, B., Smart, A., Hanna, A., et al. (2021). Towards accountability for machine learning datasets: Practices from software engineering and infrastructure. ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3442188.3445918 The authors adapt practices from software engineering to dataset development, proposing documentation and accountability mechanisms spanning the dataset lifecycle. It is built from fieldwork rather than tested for its effect.
• Sculley, D., Holt, G., Golovin, D., et al. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems 28. https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html Sculley and colleagues apply the software-engineering notion of technical debt to machine learning systems and catalogue the risk factors, among them data dependencies, hidden feedback loops and changes in the external world. It names the failure classes without measuring their frequency.
• Longpre, S., Mahari, R., Chen, A., et al. (2024). A large-scale audit of dataset licensing and attribution in AI. Nature Machine Intelligence 6(8), 975-987. doi:10.1038/s42256-024-00878-8 A legal and machine learning team audited more than 1,800 text datasets on popular hosting sites, tracing sources, creators and licences. Licence omission exceeded 70 per cent and errors exceeded 50 per cent, so the documentation the datasheet papers propose was largely absent from the datasets in use.