The politics of quantification: fairness, exclusion and research ethics

Every unit before this one has treated the measurement decision as technical work: name the construct, check the instrument, fix the analysis before the data arrive. This tenth of ten units argues that the decision is also political, and that the technical frame conceals it. What you measure decides which harm becomes visible. Whom you measure decides whose experience the number describes. What you do with the result decides who carries the cost of being optimised.

Choosing a metric is choosing who counts

Friedman and Nissenbaum gave the argument its first vocabulary. From actual cases they distinguish three kinds of bias. Preexisting bias has its roots in social institutions and attitudes. Technical bias comes from the constraints of a design. Emergent bias appears in a context of use. Freedom from bias, they argue, belongs among the criteria for judging a system's quality, beside reliability and accuracy (Friedman and Nissenbaum, 1996). A measure can faithfully record a distribution that society produced, and it can introduce a distortion of its own.

Buolamwini and Gebru made the concealment measurable. Two facial analysis benchmarks were overwhelmingly lighter-skinned, at 79.6 per cent for IJB-A and 86.2 per cent for Adience. Three commercial gender classifiers, evaluated on a balanced dataset, reached 34.7 per cent error for darker-skinned women against a maximum of 0.8 per cent for lighter-skinned men (Buolamwini and Gebru, 2018). Accuracy averaged over the benchmark would have reported these systems as close to working. The aggregate was not wrong. It was an average over a population that the systems did not serve equally.

Mack and colleagues coded 506 accessibility papers at CHI and ASSETS from 2010 to 2019, and 836 over 26 years. They found that over 43 per cent of the recent papers concerned blind and low-vision people, with a median of 13 participants for groups with disabilities and older adults (Mack et al., 2021). A field's choice of whom to study can leave whole disabilities under-represented.

Fairness is a normative choice wearing a formula's clothes

Kleinberg, Mullainathan and Raghavan formalise three fairness conditions for risk scores. They prove that no method satisfies all three except in highly constrained special cases (Kleinberg et al., 2016).

Corbett-Davies and colleagues show what the choice costs. They sort formal definitions into those that constrain the effects of decisions on disparities and those that constrain the effects of protected characteristics. Both families typically yield strongly Pareto dominated policies. In their admissions example, adhering to popular notions of fairness would produce both a less diverse student body and a less academically prepared class. Enforcing a definition can therefore harm the groups it was meant to protect (Corbett-Davies et al., 2018).

A lender builds two models, one equalising approval rates across groups and one equalising the rate of wrongful refusal. On the same data they lend to different people, and the choice is a decision about which error is worse, taken by whoever owns the model. Caton and Haas survey the field into pre-processing, in-processing and post-processing methods and 11 method areas. They close with five open dilemmas rather than a settled procedure (Caton and Haas, 2023).

An average is the wrong instrument for an unequal population

Where access is already unequal, the sample describes the better-served group more closely, and the average inherits that tilt. So does a survey delivered only to a fast connection, or a metric completed only by users who can afford the device it needs. Aggregate accuracy is the average, and subgroup error is the distribution (Buolamwini and Gebru, 2018).

D'Ignazio and Klein turn this into a method. Data science is a form of power, they argue, whose narratives are overwhelmingly white, male and techno-heroic. The response is organised around three questions: data science by whom, data science for whom, data science with whose interests in mind. Their claim that the data never, ever speak for themselves is the measurement argument stated as ethics (D'Ignazio and Klein, 2020).

Bennett and Keyes push past measurement. They draw on two computer-vision case studies of technologies meant to help disabled people. They argue that common fairness metrics are insufficient for disability, and that they are likely to produce harms that a fairness framing does not address. They call for justice, a question about relations between people rather than a property of a model (Bennett and Keyes, 2020).

The data are a political object before the analysis begins

Every number is computed on data that someone decided to collect, so provenance belongs in the argument. Hutchinson and colleagues propose documenting the decisions behind a dataset at each stage of its construction, so that a reader can see what was chosen and contest it (Hutchinson et al., 2021). Olteanu and colleagues catalogue the objects that such a study inherits, and each is a decision taken before any model runs. Population bias is present when the users who produce the data are not the population of interest. Behavioural bias is present when the platform shapes what is expressed. Content bias is present when the measurement treats groups differently. Item wording above the respondent's reading level is a content bias. Linking bias is present when the network used to connect records is itself unrepresentative. It does not rank them or say which is largest in a given study, so the provenance document still has to name the one that matters.

The ethics of a study cannot be separated from its design

Vitak, Shilton and Ashktorab surveyed 263 researchers using online data. They documented the practices on which social computing researchers converge, and the disagreements, which are not explained by discipline, method or workplace (Vitak et al., 2016). Brown, Weilenmann, McMillan and colleagues set out five provocations for ethical HCI research. They cover informed consent, the researcher-participant power differential, the presentation of data, review boards and corporate-facilitated projects, and they ground an alternative in the sensitivities of those studied (Brown et al., 2016). The decisions this module trains you to make about what to measure, whom to observe and when to stop are ethics decisions. A board reviewing a finished protocol reviews a design that fixed most of them already.

Contested question: can fairness be engineered, or does a metric relocate the choice?

The first position treats fairness as an engineering problem. Definitions exist, they can be organised into pre-processing, in-processing and post-processing methods, and a team can select and apply one (Caton and Haas, 2023). What remains on this account is standardisation and tooling.

The second holds that the choice is ineliminably normative. Kleinberg and colleagues prove that the central criteria cannot be satisfied together. Corbett-Davies and colleagues show that enforcing popular definitions can produce Pareto dominated policies that harm the groups they were meant to protect (Kleinberg et al., 2016; Corbett-Davies et al., 2018). Bennett and Keyes add the disability case, where the framing can reinforce medical gatekeepers or benefit already-privileged disabled people, while harming those rendered outliers in multiple ways (Bennett and Keyes, 2020).

The third answers with procedure. Friedman, Kahn and Borning set out value sensitive design, a tripartite method of conceptual, empirical and technical investigation. It treats human values as requirements to be investigated with the people affected rather than assumed by the designer (Friedman et al., 2008).

What would settle the disagreement is an outcome measurement rather than a further argument. The first position implies that a fixed definition, honestly applied, improves the group it addresses. The second implies that the same improvement brings a measurable loss elsewhere. A study that preregisters the definition, documents who chose it and reports subgroup outcomes on both sides of the constraint would adjudicate. Absent such studies, each side holds principle rather than measurement.

The evidence is uneven in kind. The impossibility result is a proof within stated assumptions. The harm from enforcing a definition rests on worked examples rather than an intervention, and the disability critique is a conceptual argument from two cases. Only one side reports a quantity a designer can measure and move.

The strongest objection is that the critique may prevent work that would reduce measured harm. A designer who follows the unit's caution can point to no number that improved, while a team that reports subgroup error has one.

What to do while the dispute is open

While the dispute is open, preregister the fairness definition and record who chose it. Report subgroup outcomes on both sides of the constraint, and publish the provenance of the metric.

Where the argument is heading

The direction of travel is from fairness towards accountability for the whole measurement chain. Bennett and Keyes name the destination in asking for justice rather than fairness (Bennett and Keyes, 2020). Hutchinson and colleagues supply infrastructure for it, documenting the decisions behind a dataset at each stage of construction (Hutchinson et al., 2021). The question the module closes on is D'Ignazio and Klein's, who held the power to decide what counts (D'Ignazio and Klein, 2020).

The sources this unit rests on

• Bennett, C. L. and Keyes, O. (2020). What is the point of fairness? ACM SIGACCESS Accessibility and Computing. doi:10.1145/3386296.3386301 From two computer-vision case studies of technologies intended to help disabled people, they argue that common fairness metrics are insufficient for disability and that fairness is a question about relations between people rather than a property of a model. It is the unit's strongest statement that a metric relocates the choice.
• Brown, B., Weilenmann, A., McMillan, D., et al. (2016). Five Provocations for Ethical HCI Research. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. doi:10.1145/2858036.2858313 They present five provocations for ethical HCI research, on informed consent, the researcher-participant power differential, the presentation of data, the role of review boards and corporate-facilitated projects, and ground an alternative in everyday practice and judgement. It is the evidence that ethics is a design question in HCI.
• Buolamwini, J. and Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of the 1st Conference on Fairness, Accountability and Transparency (PMLR 81). https://proceedings.mlr.press/v81/buolamwini18a.html They found two facial analysis benchmarks to be 79.6 and 86.2 per cent lighter-skinned, then evaluated three commercial gender classifiers on a balanced dataset and reported errors up to 34.7 per cent for darker-skinned women against a maximum of 0.8 per cent for lighter-skinned men. It is the unit's worked case: an aggregate conceals a subgroup failure.
• Caton, S. J. and Haas, C. S. (2023). Fairness in Machine Learning: A Survey. ACM Computing Surveys. doi:10.1145/3616865 They survey fairness in machine learning, organising the literature into pre-processing, in-processing and post-processing methods and 11 method areas, and close with five open dilemmas. It supplies the engineering position in the contested question and the evidence that the field has not converged.
• Corbett-Davies, S., Gaebler, J., Nilforoshan, H., et al. (2018). The Measure and Mismeasure of Fairness. arXiv (Cornell University). doi:10.48550/arXiv.1808.00023 They classify formal fairness definitions into those constraining disparities and those constraining the influence of protected characteristics, and show both families typically produce strongly Pareto dominated policies, so enforcing a definition can harm the groups it was meant to protect. It is the unit's argument that the choice of measure is a policy choice.
• D'Ignazio, C. and Klein, L. F. (2020). Data Feminism. The MIT Press. doi:10.7551/mitpress/11805.001.0001 They argue that data science is a form of power and set out principles for examining it, including asking who benefits and who is excluded from a dataset. It turns the module's argument into a question about who holds the power to decide what counts.
• Friedman, B., Kahn, P. H. and Borning, A. H. (2008). Value Sensitive Design and Information Systems. The Handbook of Information and Computer Ethics. doi:10.1002/9780470281819.ch4 They set out value sensitive design, a tripartite method of conceptual, empirical and technical investigation that treats human values as requirements to be investigated, not assumed. It is the procedural answer to whose values a measurement encodes.
• Friedman, B. and Nissenbaum, H. F. (1996). Bias in computer systems. ACM Transactions on Information Systems. doi:10.1145/230538.230561 From actual cases they distinguish preexisting, technical and emergent bias, and conclude that freedom from bias should be judged alongside reliability, accuracy and efficiency. It gives the unit its vocabulary for naming what kind of harm a measurement choice does.
• Hutchinson, B., Smart, A., Hanna, A., et al. (2021). Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3442188.3445918 They propose a framework for dataset development transparency drawn from the software development lifecycle, in which each stage yields documents that make the decisions behind a dataset visible. It converts the fairness question into the question of which decisions a reader can inspect.
• Kleinberg, J. M., Mullainathan, S. and Raghavan, M. (2016). Inherent Trade-Offs in the Fair Determination of Risk Scores. arXiv (Cornell University). doi:10.48550/arXiv.1609.05807 They formalise three fairness conditions for risk scores and prove that no method satisfies the three simultaneously except in highly constrained special cases. It is the impossibility result that makes measure it fairly an insufficient prescription.
• Mack, K., McDonnell, E., Jain, D., et al. (2021). What Do We Mean by "Accessibility Research"? Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3411764.3445412 They coded 506 accessibility papers at CHI and ASSETS from 2010 to 2019, and 836 over 26 years, finding that over 43 per cent of the recent papers concerned blind and low-vision people and a median group of 13 participants for studies with disabilities and older adults. It quantifies who the field measures, the unit's exclusion argument made concrete.
• Vitak, J., Shilton, K. and Ashktorab, Z. (2016). Beyond the Belmont Principles. Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing. doi:10.1145/2818048.2820078 From a survey of 263 researchers using online data they document the practices on which the community converges and its disagreements, and show the disagreements are not explained by discipline, method or workplace. It is the claim that inherited ethics do not settle the discipline's data questions.
• Olteanu, A., Castillo, C., Diaz, F., et al. (2019). Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries. Frontiers in Big Data. doi:10.3389/fdata.2019.00013 They catalogue the biases and methodological pitfalls in research using social data, distinguishing population, behavioural, content and linking biases and organising them in a framework for identifying them.