Name the adversary before the mechanism: threat modelling and secure design

A security mechanism is bought against something, and that something is usually left unstated. The claim argued here is that a design which cannot name its adversary and its assets cannot size its protections. The order of the work therefore matters: the adversary and the asset come before the countermeasure. The unit that follows asks who fails when the control meets a person.

A protection has no size until the adversary has one

Anderson (2020) organises security engineering around four questions: who the opponent is, what they want, what they can do, and what they will spend. He treats the threat model as prior to the choice of any mechanism. A control appropriate to a vandal is absurd against a state, and a control appropriate to a state is unaffordable against a vandal.

Saltzer and Schroeder (1975) reached a compatible conclusion a generation earlier, naming eight design principles for protection mechanisms: economy of mechanism, fail-safe defaults, complete mediation, open design, separation of privilege, least privilege, least common mechanism, and psychological acceptability. They insist the principles hold independently of any mechanism, and their examples run from bank safe-deposit boxes and military need-to-know rules to descriptor-based hardware.

Myagmar, Lee and Yurcik (2005) put the failure in requirement-engineering terms. Security requirements are usually written before any adversary is named, so a claim of security cannot be justified because its assumptions are unstated. Their remedy is to derive requirements from a structured threat model of the assets, the attacker's goals and the entry points.

The method is applied three times, to a software-defined radio, a network traffic monitor and a cluster monitor. In the cluster case the asset list and the adversary yield threats a mechanism-first design would not reach, among them a process masquerading as a legitimate node, though the work shows a method rather than its yield.

The model must name the assets as well as the attacks

Naming the adversary is only half the work. The other half is naming what the adversary wants, and the asset list is a design commitment rather than a given. Yoder and Barcalow (1997) collect seven architectural patterns for application security, among them single access point, check point and roles, each confining a security decision to one place in the code. The structure a pattern protects is the set of assets the design has decided to care about.

Lampson (1973) supplies the corrective. His confinement problem is stopping a program that has legitimately read information from passing it to a party that should not receive it. He shows a channel can be built from almost any shared resource, including timing. Absolute confinement is impractical, so the goal has to be limiting the bandwidth of the leak. A boundary a threat model draws is leaky by construction, and the honest question is how much leak to accept.

Three files that record only whether they are open let two programs pass a stream of bits, one opening to signal a value and the other closing to acknowledge.

The attack surface can be measured, and the adversary priced

Manadhata and Wing (2011) turn the sizing judgement into a measurement. They formalise a system's attack surface as its entry points, the resources reachable through them and the damage those resources permit, and give a metric computed from an implementation, validated through empirical studies on systems in C and Java.

Schneier (1999) contributes the attacker's side of the same arithmetic. His attack trees represent the ways an attacker can reach a goal as a tree of subgoals annotated with cost, so defences can be compared against the cheapest path rather than the most likely attack. Naming the adversary therefore includes naming the price at which the adversary stops.

Two cases that show what a named adversary buys

Spafford (1989) reconstructs the internet worm of 2 November 1988 from independent reverse-compilations of its code: it exploited flaws in BSD-derived UNIX utility programs to break in and copy itself, spreading to thousands of machines and disrupting connectivity for days. It travelled along trust relationships the designers had treated as given, the relations a threat model would have named as assets.

Langner (2011) fixes the other end of the adversary range. Analysing Stuxnet, he reports that the malware was not industrial espionage. It did not steal, manipulate or erase information; its goal was to damage a physical process. A confidentiality model of that system would have sized its protections against the wrong loss.

The formal answer has a proven limit

The literature contains a negative result that any claim of completeness must meet. Harrison, Ruzzo and Ullman (1976) model protection systems and show that the safety problem, whether a subject can acquire a particular right to an object, is decidable only under restricted assumptions. In general, under surprisingly weak assumptions, it cannot be decided, and the proof reduces the problem to the halting problem. Complete mediation, which Saltzer and Schroeder (1975) require, cannot establish that the list of subjects and objects is complete, only that every access is checked.

Contested question: does a threat model change the design or only describe it?

The literature agrees that naming the adversary is good practice. It disagrees, in three positions, about whether the canonical methods do anything measurable.

The first position treats threat modelling as a method that improves requirements and therefore systems. Myagmar and colleagues (2005) hold this directly; Anderson (2020) treats the threat model as the prior step from which mechanisms are chosen. On this account the method is causal, and its benefit should appear in the systems built with it.

The second position holds that the field cannot yet support that claim. Xiong and Lagerstrom (2019) conducted what they describe as the first systematic literature review of threat modelling, assessing 176 articles and selecting 54 for analysis. They report that the domain lacks common ground, that most threat modelling remains manual and that validation is rarely assured. Their finding is not that the methods fail, but that the evidence they succeed has not been assembled. Katsikeas and colleagues (2024) answer part of it. They ran an empirical falsification experiment on a domain-specific threat modelling language for the generic IT domain, and found that its simulation-generated assessments matched or beat human experts on average, and that both beat random guessers.

The third position is institutional rather than empirical. Souppaya, Scarfone and Dodson (2022) set out the Secure Software Development Framework as a core set of high-level practices that can be added to any development lifecycle. The practices include defining security requirements and testing against them. They present it as a shared vocabulary. Naming the adversary has become a procurement condition, so the practice is spreading faster than the second position's evidence is accumulating.

The evidence that would settle the dispute is a comparative study that runs named threat-modelling methods on the same systems with the same teams. It would count the vulnerabilities found, those fixed before release and those exploited later. It would also have to report the attacks the model did not anticipate, since a method that is never wrong has probably not been tested. The one empirical evaluation the unit now carries validates a single language on one domain rather than comparing methods on live systems, which is why the disagreement survives.

The three positions rest on different kinds of evidence. The method-improves-design position rests on an applied method and a reference text (Myagmar et al., 2005; Anderson, 2020); the no-evidence-yet position on a systematic literature review and the one falsification experiment the unit carries (Xiong and Lagerstrom, 2019; Katsikeas et al., 2024); the institutional position on a framework rather than a trial (Souppaya et al., 2022). The designer writes the adversary, the asset list and the user's own budget of effort and equipment into the requirements, and treats the model as a bound on the search. The strongest objection is that this unit orders the adversary before the mechanism on argument alone, since the review it cites finds no assembled evidence that the order yields better systems.

Where the argument is heading

The direction of travel is towards the parts of the system a designer does not own. Ohm and colleagues (2020) analyse 174 malicious packages distributed through npm, PyPI and RubyGems between November 2015 and November 2019 as real attacks on open-source supply chains. A dependency manager installs hundreds of packages without the designer naming any of them, so the assets a threat model must list now include code the team never chose. Manadhata and Wing (2011) point towards measurement repeated across versions rather than a judgement made once, and Xiong and Lagerstrom (2019) towards validation as the field's unfinished work, which Katsikeas and colleagues (2024) and the SSDF have each begun to answer.

The sources this unit rests on

• Anderson, R. (2020). Security engineering: a guide to building dependable distributed systems. Wiley. doi:10.1002/9781119644682 Argues that security must be understood through incentives and adversaries as much as through cryptography, organising the subject around who the opponent is, what they want and what they will spend, with the threat model prior to any mechanism. It is the reference statement of the unit's ordering claim.
• Saltzer, J. H. and Schroeder, M. D. (1975). The protection of information in computer systems. Proceedings of the IEEE. doi:10.1109/PROC.1975.9939 Names eight design principles for protection mechanisms, among them economy of mechanism, fail-safe defaults, complete mediation, separation of privilege, least privilege and psychological acceptability, with examples running from bank safe-deposit boxes and military need-to-know rules to descriptor-based hardware. It argues that the principles hold independently of any mechanism, so secure design is a set of commitments made before one is chosen.
• Myagmar, S., Lee, A. J. and Yurcik, W. (2005). Threat modelling as a basis for security requirements. Symposium on Requirements Engineering for Information Security. https://people.cs.pitt.edu/~adamlee/pubs/2005/sreis-05.pdf Argues that security requirements are usually written before any adversary is named, so a security claim cannot be justified because its assumptions are unstated. Derives requirements from a structured threat model of assets, attacker goals and entry points, applied to three systems. It restates the unit's thesis as a requirement-engineering failure.
• Yoder, J. and Barcalow, J. (1997). Architectural patterns for enabling application security. Pattern Languages of Programs. https://hillside.net/plop/plop97/Proceedings/yoder.pdf Catalogues seven architectural patterns for application security, including single access point, check point and roles, each confining a security decision to one place in the code. It shows the principles as reusable structure, and shows that the structure presupposes an asset list.
• Lampson, B. W. (1973). A note on the confinement problem. Communications of the ACM. doi:10.1145/362375.362389 Defines the confinement problem as stopping a program that has legitimately read information from passing it to a party that should not receive it, and shows that a channel can be built from almost any shared resource, including timing. Concludes that absolute confinement is impractical and that the goal must be bounding the leak. It establishes that the boundary a threat model draws leaks by default.
• Manadhata, P. K. and Wing, J. M. (2011). An attack surface metric. IEEE Transactions on Software Engineering. doi:10.1109/tse.2010.60 Formalises a system's attack surface as its entry points, the resources reachable through them and the damage they permit, and gives a metric computed from an implementation. Validates it through three exploratory empirical studies on systems in C and Java, turning the sizing judgement into a quantity comparable between versions.
• Schneier, B. (1999). Attack trees. Dr Dobb's Journal. https://www.schneier.com/academic/archives/1999/12/attack_trees.html Represents the ways an attacker can reach a goal as a tree of subgoals annotated with cost, so defences are compared against the cheapest path rather than the most likely attack. The diagram makes the attacker's economics explicit.
• Spafford, E. H. (1989). The internet worm program: an analysis. ACM SIGCOMM Computer Communication Review. doi:10.1145/66093.66095 Reconstructs the worm of 2 November 1988 from independent reverse-compilations: it exploited flaws in BSD-derived UNIX utility programs to break in and copy itself, spreading to thousands of machines and disrupting connectivity for days. It is a case of a threat model failing at the level of trust assumptions rather than code defects.
• Langner, R. (2011). Stuxnet: dissecting a cyberwarfare weapon. IEEE Security and Privacy. doi:10.1109/msp.2011.67 Analyses Stuxnet and reports that it was not industrial espionage, since it did not steal, manipulate or erase information; its goal was to damage a physical process. It fixes the top of the unit's adversary range, where the opponent is patient, resourced and specific.
• Harrison, M. A., Ruzzo, W. L. and Ullman, J. D. (1976). Protection in operating systems. Communications of the ACM. doi:10.1145/360303.360333 Models protection systems and shows that the safety problem, whether a subject can acquire a particular right to an object, is decidable only in restricted cases and in general cannot be decided, proving it by reduction to the halting problem. It is the negative result that a claim of completeness must confront.
• Xiong, W. and Lagerstrom, R. (2019). Threat modelling: a systematic literature review. Computers and Security. doi:10.1016/j.cose.2019.03.010 Reviews threat modelling through systematic queries of four databases, assessing 176 articles and selecting 54 for analysis across three clusters. Reports that the domain lacks common ground and that most threat modelling remains manual with limited assurance of validation.
• Katsikeas, S., Rencelj Ling, E., Johnsson, P. and Ekstedt, M. (2024). Computers and Security. doi:10.1016/j.cose.2024.103743 Ran an empirical falsification experiment on a domain-specific threat modelling language for the generic IT domain, finding its simulation-generated assessments matched or beat human experts on average while both beat random guessers. The first empirical validation study of its kind here.
• Souppaya, M., Scarfone, K. and Dodson, D. (2022). Secure software development framework (SSDF) version 1.1. NIST Special Publication 800-218. doi:10.6028/NIST.SP.800-218 Sets out a core set of high-level secure development practices that can be added to any lifecycle, including defining security requirements and testing against them, as a common vocabulary for producers and acquirers. It shows that naming the adversary has become a procurement condition.
• Ohm, M., Plate, H., Sykosch, A. and Meier, M. (2020). Backstabber's knife collection: a review of open source software supply chain attacks. Lecture Notes in Computer Science. doi:10.1007/978-3-030-52683-2_2 Analyses a dataset of 174 malicious packages used in real-world attacks on open-source supply chains, distributed through npm, PyPI and RubyGems between November 2015 and November 2019. It extends the adversary model to the dependencies a design never chose.