Risk tiering turns the ethical question into a classification question, and accountability needs an institution rather than a document

Year 3 / Module 22 Ethics, privacy, law and responsible design / Unit 9 / Risk tiering turns the ethical question into a classification question, and accountability needs an institution rather than a document

Risk tiering turns the ethical question into a classification question, and accountability needs an institution rather than a document

HCI3106 · Unit 9 of 10

The AI Act attaches its obligations to the risk tier rather than to the principle, so the contested work moves into the definition of the system, and the proposals for accountability that survive the evidence are the ones that place a function inside the organisation rather than producing another document.

Risk tiering turns the ethical question into a classification question, and accountability needs an institution rather than a document

The unit before this one ends with an audit that reports a value choice rather than a verdict. This ninth of ten units argues that the statute does not ask whether a system is ethical. Regulation (EU) 2024/1689, the Artificial Intelligence Act, asks which tier the system falls into and attaches the duties to the tier. When the obligation hangs on the classification, the contested work moves into the definition of the system. The accountability proposals that survive the evidence put a function inside the organisation rather than produce another document. Risk tiering is a definitional instrument, and a document is not accountability. Accountability needs someone who can answer, a forum with standing to ask, and a power that can act.

The principles era ended in a statute that sorts systems

Hickok (2020) reviews what the decade of principles achieved as artificial intelligence spread and concern grew about how systems collect, use and share data, and states the lessons for the regulation that followed. The lesson the Act enacts is narrow: agreement on a value is not agreement on a rule, since a principle everyone accepts can be discharged by incompatible instruments.

Regulation (EU) 2024/1689 prohibits a closed list of practices and attaches the heaviest duties to systems it classifies as high risk. Those duties include risk management, data governance, documentation, logging and human oversight. The Act adds separate obligations for general-purpose models. Application is staged, and the staging has been revised: the prohibitions applied from 2 February 2025 and the general-purpose obligations from 2 August 2025, while the AI Omnibus, in force on 27 July 2026, moved the high-risk rules to 2 December 2027 and the rules for systems embedded in products to 2 August 2028, because the harmonised standards the Act depends on had not been delivered (European Commission 2026).

The obligation attaches to the classification, not to the principle

Novelli, Casolari, Rotolo and colleagues (2023) state the structure plainly: the Act defines four risk categories, unacceptable, high, limited and minimal, and the obligation is triggered by the category a system occupies. A code of conduct says what to value; the Act says which box the system is in, and the box decides the duties.

Veale and Zuiderveen Borgesius (2021) find that different rules for different risk levels make sense, while other provisions carry surprising legal implications and some may be largely ineffective.

The contested work moves into the definition of the system

Ebers, Hoch, Rosenkranz and colleagues (2021) work through the 2021 proposal and find problems at each stage: its scope of application, the prohibited uses, the high-risk provisions and the regulatory gaps.

Amazon built a recruiting engine from 2014 to score applicants for technical posts from one to five stars, trained on a decade of resumes the company had received, most of them from men. By 2015 it was not rating candidates in a gender-neutral way. It penalised resumes carrying the word "women's" and downgraded graduates of two all-women's colleges; Amazon edited those terms out, could not guarantee the model would not find other routes, and disbanded the team by early 2017 (Dastin 2018). Classify that system as high risk and it draws documentation, logging and human oversight; as limited risk, most of that falls away. Nothing about the system changes between the readings, only the obligations, so the classification is where a position has to be defended.

The tiers miss the systems the draft did not anticipate

Hacker, Engel and Mauer (2023) take a class of system the draft did not have in view, the large generative model, and find that regulation was built around conventional models. They separate developers, deployers, users and recipients of output along the value chain and argue for layered obligations matched to those actors, because a tier defined by field of application does not capture a model whose application is chosen after training.

Novelli, Casolari, Rotolo and colleagues (2023) make the structural version of the complaint: because the categories depend statically on broad fields of application, risk magnitude can be estimated wrongly and the Act may not be enforced effectively, a problem sharpest for general-purpose systems.

Risk tiering is one model, not the only one

Bradford (2023) contrasts three approaches to regulating the digital economy, the American market-driven, the Chinese state-driven and the European rights-driven, each resting on a different theory of markets, the state and rights.

Roberts, Cowls, Morley and colleagues (2020) analyse the policy debates shaping China's strategy. Smuha (2021) frames the setting as a shift from a race to artificial intelligence to a race to regulate it, in which jurisdictions compete on the terms.

Answerability needs authority, a forum that asks and a power that acts

Novelli, Taddeo and Floridi (2023) define accountability in artificial intelligence as answerability. They identify three conditions of its possibility: authority recognition, interrogation, and limitation of power. They add an architecture of seven features read through four goals. Interrogation needs a forum with standing to ask, and limitation of power needs a body able to act. Both assume a claimant who can reach the forum, which is a distributional claim: the systems the tiers catch, welfare, credit and hiring, are used on the person with low literacy, no interpreter or no reliable connection. A document can support reporting, and cannot supply the forum or the power.

Contested question: can a document deliver accountability, or does it need an institution?

Selbst (2021) holds that a document can earn its place. He sets two goals for an algorithmic impact assessment: to make firms consider and mitigate social impacts before development, and to create documentation supporting future policy learning. The first is hard because firms hold the information and the discretion. Institutional logics such as liability avoidance and the profit motive let them undermine the requirement. The second is why he persists: documentation can be useful without full compliance, and an assessment regime can feed a broader cultural shift towards accountability.

Moss (2021) treats the assessment as an object to be judged. He maps the challenges of constructing algorithmic impact assessments and sets out a framework for evaluating their effectiveness. An assessment can be evaluated only if someone holds a standard against which to judge it, and holding a standard is a function rather than a form.

The counter-position is that the document is necessary but not sufficient, because its limits are institutional. Novelli, Taddeo and Floridi (2023) supply the criteria: without authority recognition and a limit on power, answerability is a report no one is obliged to answer. Veale, Van Kleek and Binns (2018) supply field evidence. They interviewed twenty-seven public sector machine learning practitioners across five OECD countries. They found a disconnect between the practitioners' organisational realities and the account of usable, transparent and discrimination-aware machine learning that research offered, a gap likely to undermine practical initiatives.

What would settle it is a prospective comparison, not a better argument. Two organisations on the same high-stakes task would run, one with a standing accountability function holding authority, a forum and a limit on power, the other with documentation alone. Both would be measured on the harms the assessment is meant to prevent. Until then, a document can satisfy the reporting goal and cannot by itself satisfy the conditions Novelli, Taddeo and Floridi set.

The two positions rest on different kinds of evidence. Neither Selbst (2021) nor Moss (2021) measures an outcome. The institutional position is a definition in Novelli, Taddeo and Floridi (2023) plus one interview study of twenty-seven practitioners in Veale, Van Kleek and Binns (2018). No study compares a document with a function on the same task. The strongest objection to this unit's thesis is that third-party certification already delivers answerability in other regulated fields, so a document plus an external auditor may do the institutional work without an in-house function. While the dispute is open, a team should publish the assessment and give a named person the authority to stop the release.

Where the argument is heading

Two movements are named in the recent work. The first is from field to scenario: Novelli, Casolari, Rotolo and colleagues (2023) would assess specific scenarios through an integrated risk model, because categories tied to fields estimate magnitude badly. The second is from system to value chain: Hacker, Engel and Mauer (2023) match duties to the actors along a generative model's value chain, so the unit of regulation becomes the relation between developer, deployer and user rather than a single product. Behind both sit Bradford's (2023) comparative frame and Tabassi's (2023) voluntary framework wherever the statute does not reach. The tenth unit asks what a duty owes a disabled person.

The sources this unit rests on

Each note summarises what the cited document did and found; none is a quotation.

European Parliament and Council (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act). https://eur-lex.europa.eu/eli/reg/2024/1689/oj Prohibits a closed list of practices and attaches the heaviest duties, including risk management, data governance, documentation, logging and human oversight, to systems classed as high risk, with separate obligations for general-purpose models and staged application.
European Commission (2026). AI Omnibus enters into force. https://digital-strategy.ec.europa.eu/en/news/ai-omnibus-enters-force Records that the AI Omnibus proposed on 19 November 2025 entered into force on 27 July 2026, extending the application of the high-risk rules to 2 December 2027 for Annex III systems and 2 August 2028 for systems embedded in products after delays in the harmonised standards, and adding a ban on systems generating non-consensual intimate content.
Moss (2021). Assembling accountability: algorithmic impact assessment for the public interest. Data & Society. https://datasociety.net/library/assembling-accountability/ Maps the challenges of constructing algorithmic impact assessments and sets out a framework for evaluating the effectiveness of current and proposed regimes. The assessment becomes an object to be judged.
Selbst, A. D. (2021). An institutional view of algorithmic impact assessments. Harvard Journal of Law & Technology. https://jolt.law.harvard.edu/assets/articlePDFs/v35/Selbst-An-Institutional-View-of-Algorithmic-Impact-Assessments.pdf Sets two goals, early mitigation of social impacts and documentation for policy learning, and argues that institutional logics such as liability avoidance and the profit motive make the first hard, while the second can succeed without full compliance.
Veale, M., Van Kleek, M. and Binns, R. (2018). Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making. CHI. doi:10.1145/3173574.3174014 Interviewed twenty-seven public sector machine learning practitioners across five OECD countries and found a disconnect between their organisational realities and the account of usable, transparent and discrimination-aware machine learning that research offered. The gap is institutional rather than informational.
Roberts, H., Cowls, J., Morley, J. et al. (2020). The Chinese approach to artificial intelligence. AI & Society. doi:10.1007/s00146-020-00992-2 Analyse the policy debates shaping China's strategy, dated from the 2017 State Council plan aiming at global leadership by 2030 and an industry worth a trillion yuan, about 150 billion dollars. Supplies the second model the unit tests risk tiering against.
Hickok, M. (2020). Lessons learned from AI ethics principles for future actions. AI and Ethics. doi:10.1007/s43681-020-00008-1 Reviews what the principles era achieved as artificial intelligence spread and concern grew about how systems collect, use and share data. Its lesson for later regulation is that agreement on a value is not agreement on a rule.
Veale, M. and Zuiderveen Borgesius, F. (2021). Demystifying the Draft EU Artificial Intelligence Act. Computer Law Review International. doi:10.9785/cri-2021-220402 Find that different rules for different risk levels make sense, but that some provisions carry surprising legal implications while others may be largely ineffective. The tier scheme is contested in operation, not settled by enactment.
Smuha, N. A. (2021). From a race to AI to a race to AI regulation. Law, Innovation and Technology. doi:10.1080/17579961.2021.1898300 Argues that competition to capture the benefits of artificial intelligence has become a competition to regulate it, as its risks produced calls for systems that are legal, ethical and robust. The Act is one jurisdiction's entry in that contest.
Ebers, M., Hoch, V. R. S., Rosenkranz, F. et al. (2021). A Critical Assessment of the European Commission's Proposal for an AI Act (RAILS). J. doi:10.3390/j4040043 Work through the proposal in sequence, covering the unclear preemptive effect, the scope of application, the prohibited uses, the high-risk provisions and the regulatory gaps. Scope is what a practitioner has to settle first.
Hacker, P., Engel, A. and Mauer, M. (2023). Regulating ChatGPT and other Large Generative AI Models. ACM FAccT. doi:10.1145/3593013.3594067 Take the large generative model, a class the draft did not have in view, separate developers, deployers, users and recipients of output along the value chain, and argue for layered obligations matched to those actors. A field-based tier fits such a model poorly.
Novelli, C., Casolari, F., Rotolo, A. et al. (2023). Taking AI risks seriously. AI & Society. doi:10.1007/s00146-023-01723-z State that the four risk categories depend statically on broad fields of application, so risk magnitude may be wrongly estimated, and propose assessing specific scenarios through a model integrating the Act with the Intergovernmental Panel on Climate Change risk approach. The tier stays; the assessed unit changes.
Bradford, A. (2023). Digital Empires: The Global Battle to Regulate Technology. Oxford University Press. doi:10.1093/oso/9780197649268.001.0001 Contrasts three approaches to regulating the digital economy, American market-driven, Chinese state-driven and European rights-driven, each resting on a different theory of markets, the state and rights. Risk tiering is one model among several.
Tabassi, E. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST. doi:10.6028/nist.ai.100-1 Sets out a voluntary framework for organisations that design, develop, deploy or use artificial intelligence systems, intended to be rights-preserving, non-sector specific and use-case agnostic. It governs a UK team's risk process where the Act arrives only through market access.
Novelli, C., Taddeo, M. and Floridi, L. (2023). Accountability in artificial intelligence. AI & Society. doi:10.1007/s00146-023-01635-y Define accountability as answerability, identify three conditions of its possibility, authority recognition, interrogation and limitation of power, and an architecture of seven features read through four goals, compliance, report, oversight and enforcement. Two conditions are institutional.
Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, 10 October 2018. https://www.reuters.com/article/world/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08J/ Reports that Amazon's machine learning recruiting tool, built from 2014, penalised resumes carrying the word women's and downgraded graduates of two all-women's colleges because it had been trained on a decade of resumes mostly submitted by men, and that the team was disbanded by early 2017. It replaces the unit's constructed example with a real screening system.

Check your understanding & apply it · 4 questions, 1 essay

1A UK team building credit scoring argues that Regulation (EU) 2024/1689 does not apply because the team is in the UK, and that a completed impact assessment documents the risks so nothing further is needed. Which diagnosis does the unit support?

2A team changes nothing about a screening model of the kind Dastin (2018) reports, but moves it from limited risk to high risk. What does the unit predict follows?

3Two lenders run the same high-stakes task: one has a standing accountability function holding the three conditions Novelli, Taddeo and Floridi (2023) set out, and the other files documentation alone. What does the unit say such a comparison would do?

4A general-purpose model is trained without a fixed field of application, and its use is chosen later. On the reading of Hacker, Engel and Mauer (2023) and Novelli, Casolari, Rotolo and colleagues (2023), what does that fact predict about how the tier scheme fits the model?

5Essay A UK software team builds a tool that scores the creditworthiness of applicants for a small lender. The lender also uses the tool to rank existing customers for a retention offer. The vendor is established in the European Union and the tool is sold across the single market. The team's manager argues that the project is outside the Artificial Intelligence Act because the team is in the UK, and that no further work is needed because a completed data protection impact assessment already documents the risks. Assess the manager's position using the unit's argument. State the unit's thesis, explain how the Act attaches its obligations and why the classification of the system is the contested work, say whether the market-access point supports the manager, and explain what accountability would require beyond the document the team has produced. Name the sources you rely on at each step, and state what evidence would settle the disagreement.

The unit's thesis is that the Act attaches its obligations to the risk tier rather than to the principle, so the contested work moves into the definition of the system, and the accountability proposals that survive the evidence place a function inside the organisation rather than producing another document. The manager's first claim fails on the structure of the instrument. Regulation (EU) 2024/1689 prohibits a closed list of practices and attaches the heaviest duties, risk management, data governance, technical documentation, logging, human oversight and standards for accuracy and robustness, to systems it classifies as high risk, and credit scoring is named among the high-risk domains, so the question is which tier this system occupies rather than whether the team is in the UK. That is the definitional contest: Ebers and colleagues (2021) raise the scope of application and the high-risk provisions as points carrying problems, and Novelli, Casolari, Rotolo and colleagues (2023) show that the categories rest on broad fields of application, so the risk magnitude for a tool used both for lending and for retention offers may be estimated wrongly. Veale and Zuiderveen Borgesius (2021) add that the tier scheme is not self-executing, since some provisions are unclear and others may be ineffective. The market-access point does not help the manager; it is the reason the Act reaches the team at all, as Smuha (2021) frames in the competition between regulatory models and Bradford (2023) locates in the European rights-driven model, and where the Act arrives only through market access the voluntary NIST framework of Tabassi (2023) is the instrument that actually governs the team's risk process. The second claim fails because a data protection impact assessment is a document, and accountability is not a document. Novelli, Taddeo and Floridi (2023) define accountability as answerability with three conditions, authority recognition, interrogation and limitation of power, and two of those are institutional: a forum with standing to ask and a body able to act on the answer. Veale, Van Kleek and Binns (2018), interviewing twenty-seven public sector practitioners across five OECD countries, found a disconnect between practitioners' organisational and institutional realities and what research offered, which is the gap a document does not close. Against this, Selbst (2021) argues that documentation can still work if it is legible to the firm and feeds a cultural shift, so the assessment is not worthless, but even he concedes that institutional logics such as liability avoidance and the profit motive let firms undermine the mitigation goal. What would settle the disagreement is a prospective comparison of two lenders performing the same task, one with a standing accountability function holding authority, a forum and a limit on power, the other with documentation alone, measured on the harms the assessment is meant to prevent; the field has not run it, so the defensible position is that the document can support reporting but cannot supply the forum or the power.