Fairness has no single definition, so an audit reports a value choice rather than a verdict

Earlier units treat consent, transparency and accountability as obligations a designer can meet. This eighth of ten units argues that fairness is not one of them, as practitioners usually assume. It has no single definition, the definitions conflict, and the conflict is not a dispute better engineering will close. An audit can show that a system performs differently across groups. It cannot show that the system is fair, because that judgement depends on which error you will accept and on whom it falls.

Fairness criteria conflict, and the conflict is proved rather than observed

Kleinberg, Mullainathan and Raghavan (2017) formalise three fairness conditions for a risk score and prove that, except in highly constrained special cases, no method satisfies all three at once. Two are the conditions public argument uses without naming: calibration within each group, and equal error rates across groups.

Chouldechova (2017) reaches the same conclusion from a deployed class of instrument. She shows that the fairness criteria applied to recidivism prediction instruments cannot all be satisfied at once when the prevalence of reoffending differs across groups.

ProPublica compared COMPAS risk scores for more than 7,000 people arrested in Broward County, Florida, with reoffending over two years. Defendants who were Black and did not reoffend were labelled higher risk at 44.9 per cent against 23.5 per cent for White defendants. White defendants who did reoffend were labelled low risk at 47.7 per cent against 28.0 per cent (Angwin, Larson, Mattu and Kirchner 2016). The vendor answered that its score was calibrated within each group, and both statements can hold at once. That is Chouldechova's result in a shipped instrument, not a theorem.

Friedler, Scheidegger and Venkatasubramanian (2021) sort the definitions into families. One family demands equal treatment across groups independently of outcomes, and the other demands equal error rates. The authors show that no single mechanism satisfies both, because they encode different value systems. To choose a metric is to choose which error falls on whom. That is the unit's thesis in one object.

The deployed cases show the label carries the bias

Buolamwini and Gebru (2018) found that three commercial gender classification systems missed darker-skinned females most, at error rates up to 34.7 per cent against 0.8 per cent for lighter-skinned males.

Obermeyer, Powers, Vogeli and Mullainathan (2019) show the second route by which bias enters. In a widely used commercial algorithm that health systems use to select patients for extra care, they found that at a given risk score Black patients were considerably sicker than White patients. Remedying the disparity would have raised the share of Black patients receiving additional help from 17.7 to 46.5 per cent. The algorithm predicted health care costs as a proxy for illness, and less is spent on Black patients with the same need, so the proxy recorded them as healthier. The model did what it was asked. The label was wrong.

An outside audit sees outputs, and outputs support only weak inference

Sandvig, Hamilton, Karahalios and Langbort (2014) set out five designs for auditing a platform from outside: the code audit, the noninvasive user audit, the scraping audit, the sock puppet audit, and the collaborative or crowdsourced audit. They differ by the access and evidence each requires, and the code audit would settle the question directly but needs consent that is rarely granted, so most evidence comes from outputs, which supports weaker inferences about cause. Metaxa, Park, Robertson and colleagues (2021) add that algorithmic content changes silently with no record of past exposure, so an audit measures a version of a system rather than the system.

One audit changed a vendor, and the fuller audit sits inside the firm

Raji and Buolamwini (2019) revisited the three vendors targeted by Gender Shades. Within seven months all three, IBM, Microsoft and Megvii (Face++), released new API versions and reduced accuracy disparities. The largest reduction fell on the darker-skinned female subgroup, whose error fell by 17.7 to 30.4 per cent, while overall benchmark error fell by 5.72 to 8.3 per cent. Two non-target vendors, Amazon and Kairos, lagged at 31.37 and 22.50 per cent darker-female error. Disclosure moved the named firms and left the unnamed behind.

A law has now made the audit compulsory. New York City's Local Law 144 requires an employer using an automated employment decision tool to obtain an independent bias audit and to publish a summary of its results. Gerchick and colleagues (2025) collected the 116 audits published from July 2023 to November 2024 and found them incomplete, hampered by missing demographic data, opaque aggregation and metrics that do not match how the tools are used. Fifty-three per cent carried at least one impact ratio below the four-fifths line, the law sets no threshold, and the authors knew of no enforcement action. A regulator can mandate the metric and still not deliver a verdict.

Raji, Smart, White and colleagues (2020) propose a framework for auditing a system end to end across the internal development life cycle, each stage yielding documents that form the audit report. The framework concedes the central point: the organisation that would be judged produces the evidence a fairness claim needs.

Contested question: should fairness be fixed in advance or judged by consequences?

The literature disagrees about whether the formal definitions are the right tool at all. The impossibility position holds that the criteria conflict, so the choice must be made openly and defended. Kleinberg and colleagues (2017), Chouldechova (2017) and Friedler and colleagues (2021) each treat a metric as a value judgement about which error matters.

Corbett-Davies, Gaebler, Nilforoshan, Shroff and Goel (2018) hold the opposed position. They assemble the definitions into two families, those constraining disparities in decisions and those constraining the effect of protected characteristics, and show analytically and empirically that both typically produce strongly Pareto dominated decision policies. Requiring the definitions to hold can harm the groups they were designed to protect, so equitable design must engage with context-specific consequences.

Wachter, Mittelstadt and Russell (2021) add a third axis from law. They classify fairness metrics as bias preserving or bias transforming by how each handles pre-existing inequality. They argue that a bias-preserving metric can itself give rise to prima facie indirect discrimination.

What would settle it is not a better argument but a prospective study. It would hold a decision task fixed, adopt each family of definitions in turn, and measure both what happens to the groups the constraint protects and the cost in overall accuracy. The field has not run the trial.

The sides do not carry equal evidence. The impossibility position has a proof in Kleinberg, Mullainathan and Raghavan (2017) and a deployed instrument in Chouldechova (2017). The consequences position has analytics and one worked admissions example in Corbett-Davies and colleagues (2018), and the legal axis is a classification of metrics in Wachter, Mittelstadt and Russell (2021). The strongest objection to this unit's thesis is that the impossibility results hold for binary prediction from an imperfect score, so many systems can satisfy several definitions at once. While the dispute is open, an audit should report several metrics over several subgroups, state which error it accepts, and name who bears it.

The single-axis audit misses the groups most affected

A further disagreement concerns granularity. Wang, Ramaswamy and Russakovsky (2022) show that fairness research has historically considered a single binary attribute while the reality is more complicated. Their US Census evaluation recommends evaluating on the full set of attributes and warns against imbalance techniques used without considering their normative consequences. Every deployed case in this unit turns on race or sex, and the audits built to answer them inherit those axes, so a system can be audited and still never be tested on age or disability. Pagano, Loureiro, Lisboa and colleagues (2023), in a PRISMA review that selected 45 of 128 articles, report that most of the tools they identified were not used in empirical experiments, and that different fairness metrics do not give uniform results for a given use case.

Where the argument is heading

Two directions are named in the recent work rather than guessed. The first is evaluation that matches the claim. The second is scope. Ferrara (2023) surveys the sources and impacts of bias, extends the account to generative models that reproduce stereotypes in synthetic media, and reports that group fairness metrics may not consider intersectionality. The direction is documented, multi-axis evaluation performed while a system is built rather than a verdict pronounced after it ships. The ninth unit asks who answers for the audit.

The sources this unit rests on

Each note summarises what the cited document did and found; none is a quotation.

• Kleinberg, J., Mullainathan, S. and Raghavan, M. (2017). Inherent Trade-Offs in the Fair Determination of Risk Scores. ITCS. doi:10.4230/lipics.itcs.2017.43 Proofs that three fairness conditions cannot be satisfied simultaneously except in highly constrained cases, so the conflict holds for any method rather than for a weak one.
• Chouldechova, A. (2017). Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data. doi:10.1089/big.2016.0047 Shows the fairness criteria cannot all hold at once when the prevalence of reoffending differs across groups, demonstrating the conflict on a deployed instrument and linking disparate impact to error rate balance.
• Friedler, S. A., Scheidegger, C. and Venkatasubramanian, S. (2021). The (Im)possibility of fairness. Communications of the ACM. doi:10.1145/3433949 Separates the definitions into families demanding equal treatment and equal error rates, and shows that no single mechanism satisfies both because they encode different value systems, which makes the metric a value choice.
• Buolamwini, J. and Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. PMLR 81. https://proceedings.mlr.press/v81/buolamwini18a.html Audited two benchmarks and three commercial gender classification systems; the benchmarks were 79.6 and 86.2 per cent lighter-skinned and the systems missed darker-skinned females at rates up to 34.7 per cent against 0.8 per cent for lighter-skinned males.
• Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science. doi:10.1126/science.aax2342 Found that a widely used health algorithm used cost as a proxy for illness, so Black patients at a given risk score were considerably sicker; remedying the disparity would raise the share receiving additional help from 17.7 to 46.5 per cent.
• Raji, I. D. and Buolamwini, J. (2019). Actionable Auditing. AIES. doi:10.1145/3306618.3314244 Measures whether public disclosure changed the targeted vendors; within seven months all three released new API versions and reduced disparities, with darker-female error falling by 17.7 to 30.4 per cent while two non-targets lagged at 31.37 and 22.50 per cent.
• Gerchick, M. K., Encarnacion, R., Tanigawa-Lau, C. and colleagues (2025). Auditing the audits: lessons for algorithmic accountability from Local Law 144's bias audits. FAccT. doi:10.1145/3715275.3732004 Collected the 116 bias audits published under New York City's Local Law 144 between July 2023 and November 2024 and found them incomplete, hampered by missing demographic data, opaque aggregation and metrics that do not match practice; 53 per cent carried at least one impact ratio below four-fifths, and no enforcement action was known.
• Raji, I. D., Smart, A., White, R. N. et al. (2020). Closing the AI accountability gap. FAccT. doi:10.1145/3351095.3372873 Proposes an audit inside the organisation across the whole development life cycle, with documents from each stage forming the audit report; it concedes that the evidence has to be produced where the system is built.
• Sandvig, C., Hamilton, K., Karahalios, K. and Langbort, C. (2014). Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. ICA Data and Discrimination preconference. http://www-personal.umich.edu/~csandvig/research/Auditing%20Algorithms%20--%20Sandvig%20--%20ICA%202014%20Data%20and%20Discrimination%20Preconference.pdf Sets out five ways to audit a platform from outside, the code audit, noninvasive user audit, scraping audit, sock puppet audit and collaborative or crowdsourced audit, distinguished by the access and evidence each needs, which is why most evidence comes from outputs.
• Metaxa, D., Park, J. S., Robertson, R. E. et al. (2021). Auditing Algorithms: Understanding Algorithmic Systems from the Outside In. Foundations and Trends in HCI. doi:10.1561/1100000083 Reviews audit methodology and two decades of audit studies, and states that algorithmic content changes silently with no record of past exposure, so an audit measures one version of a system.
• Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H. et al. (2018). The Measure and Mismeasure of Fairness. arXiv; Journal of Machine Learning Research, 2023. doi:10.48550/arxiv.1808.00023 Sorts fairness definitions into two families and shows analytically and empirically that both typically yield strongly Pareto dominated decision policies; its college admissions example has formal definitions produce lower diversity and a less prepared class.
• Wachter, S., Mittelstadt, B. and Russell, C. (2021). Bias Preservation in Machine Learning: The Legality of Fairness Metrics Under EU Non-Discrimination Law. West Virginia Law Review. doi:10.2139/ssrn.3792772 Classifies fairness metrics as bias preserving or bias transforming by how each handles pre-existing inequality, and argues that a bias-preserving metric can itself raise prima facie indirect discrimination under EU non-discrimination law.
• Wang, A., Ramaswamy, V. V. and Russakovsky, O. (2022). Towards Intersectionality in Machine Learning. FAccT. doi:10.1145/3531146.3533101 Works through three pipeline stages where intersectionality changes the answer and recommends evaluating on the full set of demographic attributes; its US Census evaluation introduces metrics built for the intersectional setting.
• Pagano, T. P., Loureiro, R. B., Lisboa, F. V. N. et al. (2023). Bias and Unfairness in Machine Learning Models: A Systematic Review. Big Data and Cognitive Computing. doi:10.3390/bdcc7010015 A PRISMA review that selected 45 of 128 articles published between 2017 and 2022, finding that many tools were not used empirically, that different metrics do not give uniform results, and that multiclass and multimetric studies are rare.
• Ferrara, E. (2023). Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies. Sci. doi:10.3390/sci6010003 Surveys the sources and impacts of bias and extends the account to generative models, which reproduce and amplify stereotypes in synthetic media, noting that group fairness metrics may not consider intersectionality.
• Angwin, J., Larson, J., Mattu, S. and Kirchner, L. (2016). Machine bias. ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing An analysis of COMPAS risk scores for more than 7,000 people arrested in Broward County, Florida, in 2013 and 2014, checked against reoffending over two years. Defendants who were Black and did not reoffend were labelled higher risk at 44.9 per cent against 23.5 per cent for White defendants, while White reoffenders were labelled low risk at 47.7 per cent against 28.0 per cent.