Manipulation is defined by the mechanism of influence, not by the outcome or the user's reported feeling

A design that changes what a person does is doing work that language sorts into three kinds. Persuasion offers reasons the person can accept and act on. Manipulation works around or against those reasons, so the person's own deliberation is not what produced the change. Coercion removes the choice. This sixth of ten units argues that the difference is a claim about the mechanism of influence, not about the outcome that follows. Nor is it about how the person later says they felt. The tradition that made influence a design goal assumed the change was the one the user wanted. Personalisation dissolved the setting in which that assumption held.

The persuasive technology programme assumed the change was wanted

Fogg (1998) introduced captology at CHI, defining a persuasive computer as an interactive technology that attempts to change attitudes or behaviour in some way.

Berdichevsky and Neuenschwander (1999) separate technologies that influenced people incidentally from persuasive technologies created to change attitudes and behaviours. Their golden rule holds that creators should never seek to persuade anyone of something they would not consent to being persuaded of.

A default, an order of options, a required field: each prescribes something, so influence sits in the artefact rather than arriving with it (Thaler and Sunstein 2008).

The definition turns on two words, covertly and vulnerabilities

Muller (2026) keeps the terms apart in the reference account of the field. The entry records that choice has a control condition and an epistemic condition, and that manipulation can use either.

Susser, Roessler and Nissenbaum (2019) give the criterion its clearest form:

Online manipulation is the use of information technology to covertly influence another person's decision-making, by targeting and exploiting their decision-making vulnerabilities.

Covertly: the influence is not announced as influence. Vulnerabilities: it works on a weakness rather than on the person's reasons. A vulnerability need not be a deficit: fatigue, a second language or a slow connection is enough, so the criterion describes an ordinary condition of use rather than a rare one, and personalisation is what makes its reach uneven. The paper argues the deeper harm is to autonomy rather than economic interest.

The line runs through the mechanism

Klenk (2020) defines manipulation by what the means of influence do. He argues that some software agents, such as social media newsfeed curators, manipulate users because they do not intend their means of influence to reveal the user's reasons.

Sunstein (2015) arrives at a similar test from law and policy. A statement or action is manipulative, he writes, if it does not sufficiently engage or appeal to the person's capacity for reflective and deliberative choice. His first objection is that this affronts autonomy and dignity. His second is that a manipulated choice may promote the manipulator's welfare rather than the chooser's.

A statute has now written its own test, and it is the outcome test rather than the mechanism test. Article 5(1)(a) of the Artificial Intelligence Act prohibits manipulative or deceptive techniques with the objective, or the effect, of materially distorting behaviour by impairing an informed decision, where that causes or is likely to cause significant harm (Boine 2025). The prohibition turns on the effect and on the harm, which is the position this unit's mechanism test sets aside, and Boine (2025) argues it is under-inclusive precisely there.

Two accounts written five years apart, one in the philosophy of technology and one in law, converge on the same question: did the influence pass through the person's reasons, or around them?

The humane nudge already carries the tension the mechanism test names

Thaler and Sunstein (2008) argue that no arrangement of a choice setting is neutral, so the setting should steer people towards options that serve them. They call the approach libertarian paternalism, and they treat the preserved option to opt out as what separates it from coercion.

The account already carries the tension this unit is about: if the test is the outcome, a nudge towards a good end is acceptable and one towards a bad end is not, and the mechanism does no work. Sunstein (2015) strengthens that reading by treating manipulation as a matter of degree.

Personalisation is what made the assumption untenable

Yeung (2016) describes algorithmic decision-guidance techniques that shape the informational context in which decisions are made. She argues that, unlike the static nudge of placing the salad in front of the lasagne, these hypernudges are networked, continuously updated and pervasive. Their legitimacy is not resolved through individual notice and consent, which is exactly the assumption the persuasive tradition rested on.

Cambridge Analytica is the case that shows the mechanism at scale. Its researchers obtained Facebook profiles through a personality application in 2014, scored the participants and their friends on five personality traits, matched those scores to voter records, and targeted political advertising in the United Kingdom's 2016 referendum on European Union membership (Bakir 2020; ICO 2018). Bakir's document analysis of inquiries, regulatory findings and journalism concludes that the practice was a form of psychological operations, because the targeting worked on psychological dispositions rather than through reasons the person could accept. The case does not measure persuasion, so it establishes the mechanism rather than the effect, which is the distinction this unit draws.

Lorenz-Spreen and colleagues (2020) make the empirical case. The online ecosystem has been designed to capture attention rather than support deliberate cognition and autonomous choice. Overload, finely tuned personalisation and distorted cues open the way to manipulation.

Contested question: is manipulation a mechanism, an intention, or an outcome?

The literature agrees on neither the criterion nor the bearer of the wrong. Susser, Roessler and Nissenbaum (2019) point to covert exploitation of vulnerabilities, Klenk (2020) to means that do not reveal the user's reasons, and Sunstein (2015) to influence that does not engage reflective choice.

The intention position is Wilkinson's (2012). Governments that nudge as Thaler and Sunstein intend do not manipulate, he argues, because they lack the intention to. He holds that consent can make manipulation consistent with autonomy. If either claim holds, mechanism alone is not sufficient, since two systems with the same mechanism could be persuasion in one case and manipulation in the other. The golden rule of Berdichevsky and Neuenschwander (1999) points the same way.

The outcome position justifies the steer by its service to the chooser. Thaler and Sunstein (2008) do this directly. Frischmann and Selinger (2018) add a structural version, in which an engineered environment makes predictable, programmable people and the harm is the removal of the occasions for judgement.

What would settle it is an account that separates the mechanism from the wrongness of using it, and evidence that can tell the two apart. Wilkinson (2012) shows no such account exists, since we lack a clear and complete description of what perverting the decision-making process is. The study that would settle it would hold the mechanism constant, vary intention and outcome independently, and test which one people's judgements track. Reported feeling is the weakest test, because Sunstein's criterion (2015) is whether the influence engages reflective choice, not whether it is resented.

The three positions do not rest on equal evidence. The mechanism position is a set of definitions from Susser, Roessler and Nissenbaum (2019), Klenk (2020) and Sunstein (2015), with no experiment behind them, and the other two rest on an argument in Wilkinson (2012) and a design tradition in Thaler and Sunstein (2008). The dispute is conceptual at present, and the counter-case section supplies the objection to this unit's thesis. While the dispute is open, a designer should describe in the design record what the feature does to the user's reasons and make its own purpose visible, since visibility is the one feature all three accounts treat as improving the case.

The counter-case: a mechanism test over-convicts

The strongest objection is that the mechanism test convicts too much, in two ways. Sunstein (2015) shows a mechanism-identified manipulation can be justified. When the manipulator is informed and interested in the chooser's welfare, the identification cannot be the whole judgement. Wilkinson (2012) shows the nudge Thaler and Sunstein defend shares its mechanism with cases they would condemn. The reply is that the objection concerns how much a manipulation matters, not whether it is one. Collapsing the two is what the persuasive tradition did when it assumed the change was wanted.

Where the argument is heading

The constructive direction is to design for the condition that manipulation attacks, which is autonomy, as Calvo, Peters, Vold and Ryan (2020) do in making it a design requirement. A mechanism test can be operationalised: if manipulation is influence that does not reveal the person's reasons, a design can be assessed by whether it does. The seventh unit follows the mechanism into the workplace.

The sources this unit rests on

Each note is this unit's summary of what the cited document did and found, not a quotation from it.

• Fogg, B. (1998). Persuasive computers. CHI '98. doi:10.1145/274644.274677 The paper that founded captology, the study of computers that change what people think and do. It defines a persuasive computer as an interactive technology that attempts to change attitudes or behaviour in some way and sets out a tool, medium, social actor triad. The change is one the designer intends.
• Berdichevsky, D. and Neuenschwander, E. (1999). Toward an ethics of persuasive technology. Communications of the ACM. doi:10.1145/301353.301410 The first sustained ethics of the field. It separates technologies that influenced people incidentally from those created to change attitudes and behaviours, and proposes principles on outcomes, motivations, responsibility and privacy, including a golden rule about what a creator would consent to being persuaded of, a test of the designer rather than the mechanism.
• Boine, C. (2025). The AI Act Manipulation Gap. Emory International Law Review 39(2), 417. https://scholarlycommons.law.emory.edu/eilr/vol39/iss2/1/ Reads Article 5(1)(a), which prohibits manipulative or deceptive techniques with the objective or the effect of materially distorting behaviour. It argues the prohibition is under-inclusive because it centres subliminal techniques and conscious purpose.
• Muller, V. C. (2026). Ethics of artificial intelligence and robotics. Metaphysics Research Lab, Stanford University (first published 2020, substantive revision 27 March 2026). https://plato.stanford.edu/entries/ethics-ai/ The reference account of the field, with a section on human autonomy and manipulation. It records that choice has a control condition and an epistemic condition, that manipulation can use either, and that the epistemic condition joins what manipulation can attack.
• Susser, D., Roessler, B. and Nissenbaum, H. (2019). Technology, autonomy, and manipulation. Internet Policy Review. doi:10.14763/2019.2.1410 A definitional article written to make online manipulation legible to policymakers. It defines online manipulation as covert influence on another person's decision-making that targets and exploits their decision-making vulnerabilities, and argues the deeper harm is to autonomy rather than economic interest.
• Klenk, M. (2020). Digital Well-Being and Manipulation Online. In Burr and Floridi (eds), Ethics of Digital Well-Being. doi:10.1007/978-3-030-50585-1_4 A chapter asking whether software agents manipulate users. It argues that newsfeed curators and similar agents manipulate because they do not intend their means of influence to reveal the user's reasons. The criterion becomes a property of the system.
• Sunstein, C. R. (2015). Fifty Shades of Manipulation. doi:10.2139/ssrn.2565892 A legal and policy essay on why manipulation is rarely policed. It defines a manipulative statement or action as one that does not sufficiently engage or appeal to the person's capacity for reflective and deliberative choice, and treats the norm against manipulation as a welfarist heuristic that can err.
• Thaler, R. H. and Sunstein, C. R. (2008). Nudge: Improving Decisions About Health, Wealth, and Happiness. Yale University Press. https://yalebooks.yale.edu/book/9780300122237/nudge/ The book that made choice architecture a policy instrument. It argues no arrangement of a choice setting is neutral, so the setting should steer people towards options that serve them, and treats the preserved option to opt out as what separates it from coercion.
• Yeung, K. (2016). Hypernudge: Big Data as a mode of regulation by design. Information, Communication and Society. doi:10.1080/1369118x.2016.1186713 An article that reads big data through regulatory governance scholarship. It describes algorithmic decision-guidance techniques that shape the informational context of choice, and argues these hypernudges are networked, continuously updated and pervasive, unlike the static nudge.
• Lorenz-Spreen, P., Lewandowsky, S., Sunstein, C. R. and Hertwig, R. (2020). How behavioural sciences can promote truth, autonomy and democratic discourse online. Nature Human Behaviour. doi:10.1038/s41562-020-0889-7 An article that diagnoses the online environment and proposes repairs. It argues the ecosystem has been designed to capture attention rather than support deliberate cognition and autonomous choice, and that overload, personalisation and distorted cues open the way to manipulation.
• Wilkinson, T. M. (2012). Nudging and Manipulation. Political Studies. doi:10.1111/j.1467-9248.2012.00974.x A philosophical article asking when a nudge would be manipulative. It argues that governments which nudge as Thaler and Sunstein intend do not manipulate because they lack the intention to, that consent can make manipulation consistent with autonomy, and that no account fully describes the perversion of the decision-making process.
• Frischmann, B. and Selinger, E. (2018). Re-Engineering Humanity. Cambridge University Press. doi:10.1017/9781316544846 A book about human judgement in engineered environments. It argues that designing programmable worlds goes with engineering predictable, programmable people, tracing the link across trackers, contracts, platforms and companion devices. Its harm, the removal of the occasions for judgement, is a structural outcome.
• Calvo, R. A., Peters, D., Vold, K. and Ryan, R. M. (2020). Supporting Human Autonomy in AI Systems: A Framework for Ethical Enquiry. In Burr and Floridi (eds), Ethics of Digital Well-Being. doi:10.1007/978-3-030-50585-1_2 A chapter that turns autonomy from a constraint into a design requirement. It argues responsible AI requires designing for human rather than machine autonomy, drawing on psychological evidence that autonomy supports motivation, growth and well-being.
• Information Commissioner's Office (2018). Investigation into the use of data analytics in political campaigns: investigation update. https://ico.org.uk/media2/migrated/2259371/investigation-into-data-analytics-for-political-purposes-update.pdf The regulator's interim account of its inquiry into data analytics in political campaigns. It records the link between Cambridge Analytica, SCL Elections and Aggregate IQ over data obtained from Facebook, and the notice of intent it served on Facebook.
• Bakir, V. (2020). Psychological operations in digital political campaigns. Frontiers in Communication. doi:10.3389/fcomm.2020.00067 Analyses inquiries, regulatory findings and journalism on Cambridge Analytica's activities in the 2016 referendum campaign, and concludes that its psychographic profiling and targeting was a form of psychological operations. It records the data collected through a Facebook application, the personality scoring, and the matching to voter records.