Fairness: why the definitions conflict, and what a product must state

This sixth of the ten units takes which errors the system may make against whom. Fairness in a machine learning product is usually treated as a quality to be achieved, as though a better metric would settle it. Several fairness criteria are each defensible on their own terms. No decision procedure satisfies all of them at once except in narrow cases, because they make incompatible demands about which errors a system may make against whom. The obligation that follows is not to be fair in the abstract. It is to state which criterion is used, measure it on the population the system affects, and accept in public the errors the choice assigns to identifiable people.

The harm is fixed before any criterion is chosen

Barocas and Selbst (2016) show how early the decision is made. Data mining reproduces discrimination while appearing neutral in three ways. It inherits the prejudices of decision makers whose judgements became the labels. It reflects inequalities already present in what it measures. It finds regularities that are only patterns of exclusion. Reading American disparate impact doctrine as the best route for a claimant, they show the business necessity defence sitting beside a method built to find predictive correlations.

Obermeyer and colleagues (2019) show the consequence in a deployed system. A commercial algorithm used by health systems to select patients for extra care did well on its own terms. At a given risk score, however, Black patients were considerably sicker than White patients. Removing the disparity would have raised the share selected for additional help from 17.7 to 46.5 per cent. The algorithm predicted health care cost rather than illness. Because less is spent on Black patients for the same burden of disease, cost was a biased proxy. A related failure excludes the disabled person whose record is too thin to be counted.

Buolamwini and Gebru (2018) supply the measurement discipline that follows. They built a facial analysis benchmark balanced by gender and skin type. On it, three commercial gender classifiers reached error rates up to 34.7 per cent for darker-skinned females against 0.8 per cent for lighter-skinned males. The disaggregated figure was the finding, not a caveat.

Three conditions that cannot hold at once

The impossibility results are narrower than they are often reported. Kleinberg, Mullainathan and Raghavan (2017) formalise three conditions a risk score might be asked to meet: calibration within each group, balance for the positive class, and balance for the negative class. They prove no method satisfies all three simultaneously except in highly constrained special cases. Approximate satisfaction still requires the data to lie near one of those cases. This is arithmetic under unequal base rates, not a defect of any algorithm.

Chouldechova (2017) reaches the same obstacle from the applied side. The criteria applied to recidivism prediction instruments cannot all hold when the prevalence of reoffending differs between groups, and disparate impact arises when a score fails error rate balance. Angwin and colleagues (2016), whose analysis the paper takes as its starting point, found that a non-recidivating Black defendant was nearly twice as likely to be scored high risk as a comparable White defendant. Chouldechova (2017) shows the two observations are compatible. When reoffending is more common in one group, matching the proportion of high-risk scores that are correct forces the false positive rates apart. One instrument therefore satisfies predictive parity and still flags innocent members of one group at a higher rate.

Hardt, Price and Srebro (2016) show the impossibility is not the end of the design task. They define equalised odds, requiring true positive and false positive rates to match across groups, and give a post-processing step that adjusts any learned predictor to satisfy it. The criterion is oblivious, resting on joint statistics rather than individual cases, and the authors state its limits.

Friedler, Scheidegger and Venkatasubramanian (2021) supply the vocabulary the rest of the unit needs. Alongside observed features and the decision space of outputs they posit a construct space of unobservable but meaningful variables. They show that different fairness mechanisms assume different mappings from constructs to observations. Jacobs and Wallach (2019) give the reason the conflict is not merely technical. Treating constructs such as risk of recidivism as unobservable quantities inferred through a measurement model, they call fairness an essentially contested construct.

Contested question: state a criterion, or abandon the formalism?

Three positions are live.

The first holds that the impossibility is a specification problem whose answer is to state a criterion and measure it. Kleinberg and colleagues (2017) prove the conditions conflict. Chouldechova (2017) asks for base rates alongside the measure. Hardt and colleagues (2016) show a named criterion can be enforced by post-processing.

The second holds that the formal constraints should be abandoned rather than chosen. Corbett-Davies and colleagues (2018) sort the definitions into those constraining disparities in outcomes and those constraining the influence of protected characteristics on decisions. They show analytically and in an admissions example that both families typically yield policies strongly Pareto dominated by a simple threshold rule. Requiring such definitions can, they argue, harm the groups they were written to protect.

The third holds that the argument is at the wrong level of description. Selbst and colleagues (2019) argue that abstraction and modular design make technical interventions ineffective or misleading in fairness work, because harms arise in social, organisational and historical context the abstractions discard. Wachter, Mittelstadt and Russell (2021) press the point from law. EU non-discrimination law and the case law of the Court of Justice treat equality as contextual, decided case by case.

What would separate them is a comparison none of these papers runs. Hold the data and model fixed, vary only the constraint, and measure accuracy together with harm to the population it protects.

The three positions are differently supported. The case for stating and enforcing a criterion is formal, resting on an impossibility proof, an account of disparate impact under unequal base rates and a post-processing method. The case against the formal constraints is also formal, and it adds one admissions example. The third position is an argument from law and sociology rather than a measurement of harm. The strongest objection to this unit's thesis is that disclosure can be the weaker duty. Corbett-Davies and colleagues (2018) show that a stated criterion is typically Pareto dominated by a simple threshold rule, so a product that states the wrong criterion has documented a harm rather than prevented one. While the dispute is open, a designer should state which criterion is enforced and record the errors it assigns to identifiable people, rather than report an average over the population.

What to do while the dispute is open

The obligation is to disclose the choice. State the construct and the target variable, because a proxy can be predictively accurate and still encode the inequality it measures (Obermeyer et al., 2019). State which criterion is enforced, because the alternatives are not equivalent (Kleinberg et al., 2017; Chouldechova, 2017). State the measurement assumptions, because one criterion means different things under different construct theories (Friedler et al., 2021; Jacobs and Wallach, 2019). Report by subgroup and intersection, because an average can hide a group the system fails (Buolamwini and Gebru, 2018). Pagano and colleagues (2023) screened 128 articles and found different fairness metrics give non-uniform results for the same use case. Caton and Haas (2024) close a survey with five dilemmas rather than a standard.

The one piece of evidence that disclosure changes behaviour is also the weakest. Raji and Buolamwini (2019) tracked the commercial effect of the Gender Shades audit. All three targeted companies released new API versions within seven months. Overall benchmark error fell by 5.72 to 8.3 per cent, with the largest reduction for the darker-skinned female subgroup. A company the audit did not name still carried an error rate above 31 per cent for that subgroup.

Where the argument is heading

The direction is from picking a metric towards making a measured, auditable claim. Wachter and colleagues (2021) propose conditional demographic disparity as an evidential baseline meant to carry the law's procedural requirements without pretending the normative judgement away. Pagano and colleagues (2023) show why one figure will not travel between use cases. Corbett-Davies and colleagues (2018) insist that any stated criterion must answer their charge that it is Pareto dominated by context-specific policy. The open question is one Raji and Buolamwini (2019) raise and cannot answer. Does an audited criterion change the outcome for the people the system scores, or only the announcements of the company that scores them?

The sources this unit rests on

• Barocas, S. and Selbst, A. D. (2016). Big Data's Disparate Impact. California Law Review 104(3), 671-732. doi:10.2139/ssrn.2477899 Argues that data mining reproduces discrimination while appearing neutral, inheriting the prejudices of prior decision makers, reflecting existing inequality, or finding regularities that are only patterns of exclusion. Examines American disparate impact doctrine, and the business necessity defence, as the claimant's route.
• Buolamwini, J. and Gebru, T. (2018). Gender shades: intersectional accuracy disparities in commercial gender classification. Proceedings of the 1st Conference on Fairness, Accountability and Transparency (PMLR 81). https://proceedings.mlr.press/v81/buolamwini18a.html Found two facial analysis benchmarks overwhelmingly lighter-skinned and built a benchmark balanced by gender and skin type. Three commercial classifiers reached error rates up to 34.7 per cent for darker-skinned females against 0.8 per cent for lighter-skinned males.
• Caton, S. and Haas, C. (2024). Fairness in Machine Learning: A Survey. ACM Computing Surveys. doi:10.1145/3616865 Organises fairness approaches into pre-processing, in-processing and post-processing methods with eleven method areas, and notes that regression, recommender systems and unsupervised learning are thinner than binary classification. It closes with five dilemmas, not a standard.
• Chouldechova, A. (2017). Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data. doi:10.1089/big.2016.0047 Shows the fairness criteria applied to recidivism instruments cannot all hold when the prevalence of reoffending differs across groups, and how disparate impact arises when a score fails error rate balance. A reported measure must be read with the base rates.
• Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., et al. (2018). The Measure and Mismeasure of Fairness. arXiv preprint arXiv:1808.00023. doi:10.48550/arXiv.1808.00023 Sorts fairness definitions into those constraining disparities in outcomes and those constraining the influence of protected characteristics on decisions, and shows analytically and empirically that both families typically yield strongly Pareto dominated policies. In an admissions example they lowered diversity and academic preparation.
• Friedler, S. A., Scheidegger, C. and Venkatasubramanian, S. (2021). The (Im)possibility of fairness. Communications of the ACM. doi:10.1145/3433949 Introduces a construct space of unobservable but meaningful variables alongside observed features and the decision space of outputs, and shows that different fairness mechanisms require different assumptions about how constructs map onto observations. Fairness work should state them explicitly.
• Hardt, M., Price, E. and Srebro, N. (2016). Equality of Opportunity in Supervised Learning. arXiv preprint arXiv:1610.02413. doi:10.48550/arXiv.1610.02413 Proposes equalised odds and a post-processing method that adjusts any learned predictor to satisfy it. The criterion is oblivious, resting on joint statistics, and the framework shifts the cost of poor classification from the disadvantaged group to the decision maker.
• Jacobs, A. Z. and Wallach, H. (2019). Measurement and Fairness. arXiv preprint arXiv:1912.05511. doi:10.48550/arXiv.1912.05511 Proposes measurement modelling from the social sciences, treating constructs such as risk of recidivism as unobservable quantities inferred through a measurement model whose assumptions decide which fairness properties can hold. Fairness is an essentially contested construct, so disputes about it are about the construct.
• Kleinberg, J., Mullainathan, S. and Raghavan, M. (2017). Inherent Trade-Offs in the Fair Determination of Risk Scores. LIPIcs, Volume 67, ITCS 2017. doi:10.4230/LIPIcs.ITCS.2017.43 Formalises three fairness conditions for a probabilistic risk score, covering calibration within groups and balance for the positive and negative classes, and proves no method satisfies all three simultaneously except in highly constrained special cases.
• Obermeyer, Z., Powers, B., Vogeli, C., et al. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science. doi:10.1126/science.aax2342 Analysed a commercial risk algorithm used to select patients for extra care: at a given risk score Black patients were sicker than White patients, and removing the disparity would raise the share receiving extra help from 17.7 to 46.5 per cent. The bias arose because the algorithm predicted cost rather than illness.
• Pagano, T. P., Loureiro, R. B., Lisboa, F. V. N., et al. (2023). Bias and Unfairness in Machine Learning Models: A Systematic Review on Datasets, Tools, Fairness Metrics, and Identification and Mitigation Methods. Big Data and Cognitive Computing. doi:10.3390/bdcc7010015 A PRISMA review that screened 128 articles published between 2017 and 2022 and selected 45. Different fairness metrics do not give uniform results for a given use case.
• Raji, I. D. and Buolamwini, J. (2019). Actionable Auditing. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. doi:10.1145/3306618.3314244 Investigated the commercial impact of the Gender Shades audit: all three targeted companies released new API versions within seven months and overall benchmark error fell by 5.72 to 8.3 per cent, most for the darker-skinned female subgroup. One untargeted company carried 31 per cent error.
• Angwin, J., Larson, J., Mattu, S. and Kirchner, L. (2016). Machine Bias. ProPublica, 23 May 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing The investigation Chouldechova (2017) takes as her starting point: the instrument was almost twice as likely to label Black defendants higher risk when they did not reoffend. It puts the recidivism disparity on primary data.
• Selbst, A. D., Boyd, D., Friedler, S. A., et al. (2019). Fairness and Abstraction in Sociotechnical Systems. Proceedings of the Conference on Fairness, Accountability, and Transparency. doi:10.1145/3287560.3287598 Argues that importing abstraction and modular design from computer science into fairness work makes technical interventions ineffective, because harms arise in social, organisational and historical context the abstractions discard. The authors set out five traps and argue for refocusing design on process.
• Wachter, S., Mittelstadt, B. and Russell, C. (2021). Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI. Computer Law & Security Review. doi:10.1016/j.clsr.2021.105567 Analyses EU non-discrimination law and the case law of the Court of Justice, finding an incompatibility between European notions of discrimination and statistical fairness, the gap between contextual equality and embedded metrics. Automating fairness under EU law may be impossible because the concepts require case-by-case judgement.