Tests make change safe; the evidence on writing them first is contested

A program that cannot be checked cannot be changed safely: a test re-asks yesterday's question without a person present, and keeps asking it after the code underneath has moved. Around the order in which those tests are written, the literature disagrees. This unit argues that the evidence supports tests throughout, in small steps. The claim that writing the test first causes quality does not survive once step size, discipline and context are controlled. The strongest counter-argument, taken up below, is that the industrial case studies are the only evidence from real work and they favour test-first, while the experiments that find no effect are short laboratory tasks. Unit 9 records those changes, and the decisions behind them, for the people who were not there.

Tests are a question that can be asked again

A test is a claim about a program that a machine can check without a person present. Beck made that claim the centre of a working method: write a failing test, write the least code that makes it pass, then clean up. He presents that cycle as design rather than only verification (Beck, 2002). Whether the cycle runs in that order is a question about method; whether the tests exist decides whether a change is safe. A suite grows, and Yoo and Harman surveyed the techniques that keep it affordable: minimisation, selection and prioritisation of test cases, and the trade-off between early detection and running cost (Yoo and Harman, 2012). A test of a member discount asserts that a price of 100 with the member flag returns 90, and it fails the moment the function returns 80. The assertion records a decision that survives the programmer who made it.

What continuous testing buys, and what it does not

Saff and Ernst tested a stronger version: tests that run automatically as code is edited. In a controlled experiment, students using continuous testing were three times more likely to complete the task before the deadline, and a continuous compilation group twice as likely, with a significant effect on completion and none on time worked (Saff and Ernst, 2004). That is evidence for tests present throughout, and it concerns frequency, not order. The bounds matter: students, assignment-sized tasks and completion rather than quality.

Where the strong claim came from

The claim that writing the test first causes quality has an industrial origin. George and Williams ran experiments with 24 professional pair programmers, test-first against a waterfall-like process, on a small Java program: the pairs passed 18 per cent more black-box test cases, at 16 per cent more development time, and the authors blamed granularity for the gain (George and Williams, 2003). Their observation that the waterfall group often failed to write the required tests afterwards is the more durable finding (George and Williams, 2003). Nagappan and colleagues then measured four products at Microsoft and IBM that had adopted the practice: pre-release defect density was 40 to 90 per cent lower than in comparable projects, at 15 to 35 per cent more initial time. The teams chose the practice themselves, so the gain cannot be separated from their discipline (Nagappan et al., 2008).

When step size is held constant, order stops carrying the effect

The experimental work attacks that separation. Fucci and colleagues analysed 82 data points from 39 professionals: sequencing had no important influence, while granularity and uniformity were positively associated with quality and productivity (Fucci et al., 2017). Their conclusion is the thesis of this unit: the benefits attributed to test-driven development may come from fine-grained, steady steps rather than from the test-first dynamic. Tosun and colleagues ran an industry experiment with 24 professionals on two simple tasks and one realistic brownfield application: no statistical difference in quality, a productivity gain for test-first on the simple task, and a drop on the complex one (Tosun et al., 2017).

A test can be wrong about the world

If tests make change safe, an unreliable test matters. Inozemtseva and Holmes studied suite coverage, the fraction of code executed, as a proxy for fault detection. Once suite size was controlled, coverage was not strongly correlated with effectiveness (Inozemtseva and Holmes, 2014). Luo and colleagues studied 201 commits that likely fix flaky tests across 51 open-source projects and classified the causes. Nondeterministic outcomes undermine the assumption that an unmodified test always passes or always fails for the same code (Luo et al., 2014). One cause is order-dependence: a test can pass alone and fail when another has run first, because the two share state the suite assumed was fresh. Nothing in the code under test had changed, and the suite had quietly become a second program with its own faults. A test that waits on an asynchronous update also fails on a slow device or a slow connection, so two machines do not run the same suite. Claessen and Hughes move the test from example to property. In QuickCheck a property is an executable predicate checked against random inputs, and their case studies found faults on real programs (Claessen and Hughes, 2000). A property states a rule rather than a case: reverse a list twice and the original returns, for every list.

The interface is the hardest object to test

The object this module builds is not a pure function. Banerjee and colleagues mapped GUI-testing research, reducing 230 articles from 1991 to 2011 to a final pool of 136. A GUI test case is a sequence of input events on widgets (Banerjee et al., 2013). Events over time carry a far larger state space than a function's arguments, so the cheap determinism of a unit test does not transfer. The mapping records that evaluation of the field's techniques is uneven, so the rule that a change is safe only where tests exist must be qualified by how hard those tests are to write here (Banerjee et al., 2013).

Contested question: does the order of writing the tests cause the quality?

The literature disagrees, and about which evidence counts. The practice camp takes the industrial case studies as the closest thing to evidence from real work. George and Williams report higher black-box pass rates for test-first pairs (George and Williams, 2003), and Nagappan and colleagues report defect density down by between 40 and 90 per cent (Nagappan et al., 2008). The meta-analysis finds a real quality gain alongside its productivity cost (Rafique and Misic, 2013). The strength of that evidence varies: the industrial teams chose the practice themselves, and the first study's 24 pairs worked on one small program against a waterfall-like control.

The process camp holds that those results do not isolate the cause. Its strongest evidence is Fucci and colleagues, who analysed 82 data points from 39 professionals: sequencing had no important influence, while granularity and uniformity were positively associated with quality and productivity (Fucci et al., 2017). Tosun and colleagues then found the task dominating, with no quality difference and a productivity drop for test-first on the complex task (Tosun et al., 2017). The family of twelve found novices slightly better on iterative test-last (Santos et al., 2021). These are experiments with small samples and short tasks, so they bound the effect where developers are watched.

What would settle it is an experiment that holds the code, task, developer skill and step size constant and varies only the sequence, measuring quality and productivity across sites. The family Santos and colleagues report is the nearest attempt, and its finding that the task matters more than the order is why no single study has closed the question. Adoption is the other half of the answer: Causevic and colleagues identified seven factors limiting industrial take-up, including increased development time and weak adherence to the protocol (Causevic et al., 2011). Until the sequence is isolated, write tests throughout in small uniform steps and keep the suite fast enough to run on every edit. The practice should be judged on defects per release, not on the label attached to the process.

Where the argument is heading

Five lines point the way. The family-of-experiments method, twelve coordinated studies aggregated by meta-analysis, is the field's answer to an effect that depends on context (Santos et al., 2021). Property-based testing, introduced by Claessen and Hughes, states what a program must never do rather than what it returns for one input (Claessen and Hughes, 2000). Flaky tests became an object of study once Luo and colleagues classified their causes across 51 projects, because a suite that cannot be trusted cannot make a change safe (Luo et al., 2014). GUI testing, mapped by Banerjee and colleagues, is the frontier closest to this module, where event sequences and a large state space keep testing harder than the code behind it (Banerjee et al., 2013). The newest line runs through generative code. When a model writes the code, the tests are the specification it is handed, and Mathews and Nagappan (2024) found that giving tests alongside the problem statement raised success on two standard code-generation benchmarks. The author changes; the rule does not. What makes a change safe is a test that states the requirement, whenever and by whoever it was written.

The sources this unit rests on

• Beck, K. (2002). Test-Driven Development: By Example. Boston: Addison-Wesley. https://web.archive.org/web/2023/https://www.oreilly.com/library/view/test-driven-development/0321146530/ Defines the cycle of a failing test, passing code and refactoring, and presents the practice as design rather than only verification. The canonical statement of the method this unit tests.
• George, B. and Williams, L. (2003). An initial investigation of test driven development in industry. SAC 03. doi:10.1145/952532.952753 Ran experiments with 24 professional pair programmers, test-first against a waterfall-like process: 18 per cent more black-box tests passed at 16 per cent more time. The claim that seeded the dispute, on one small program.
• Nagappan, N., Maximilien, E. M., Bhat, T. and Williams, L. (2008). Realizing quality improvement through test driven development: results and experiences of four industrial teams. Empirical Software Engineering 13(3), 289-302. doi:10.1007/s10664-008-9062-z Case studies of four teams at Microsoft and IBM that adopted the practice: pre-release defect density fell 40 to 90 per cent against comparable projects, at a reported 15 to 35 per cent more initial time. Strongest pro-test-first evidence, though self-selected.
• Rafique, Y. and Misic, V. B. (2013). The effects of test-driven development on external quality and productivity: a meta-analysis. IEEE Transactions on Software Engineering 39(6), 835-856. doi:10.1109/TSE.2012.28 Meta-analysis of 27 studies. A small positive effect on external quality and little to no effect on productivity overall, both larger in industrial studies and rising with task size.
• Fucci, D., Erdogmus, H., Turhan, B., Oivo, M. and Juristo, N. (2017). A dissection of the test-driven development process: does it really matter to test-first or to test-last? IEEE Transactions on Software Engineering 43(7), 597-614. doi:10.1109/TSE.2016.2616877 Analysed 82 data points from 39 professionals, one per task. Sequencing had no important influence, while granularity and uniformity were positively associated with both outcomes. Step size, not order, drives quality.
• Tosun, A., Dieste, O., Fucci, D., Vegas, S., Turhan, B., Erdogmus, H., Santos, A. et al. (2017). An industry experiment on the effects of test-driven development on external quality and productivity. Empirical Software Engineering 22(6), 2763-2805. doi:10.1007/s10664-016-9490-0 Experiment with 24 professionals on two simple tasks and one brownfield application: no quality difference, a productivity gain for test-first on the simple task and a drop on the complex one. Task complexity confounds the effect.
• Santos, A., Vegas, S., Dieste, O., Uyaguari, F., Tosun, A., Fucci, D., Turhan, B. et al. (2021). A family of experiments on test-driven development. Empirical Software Engineering 26. doi:10.1007/s10664-020-09895-8 Twelve coordinated experiments in academia and industry, aggregated by meta-analysis: novices reached slightly higher quality with iterative test-last, and the task largely determined quality. Reads against the test-first claim.
• Saff, D. and Ernst, M. D. (2004). An experimental evaluation of continuous testing during development. ISSTA 04. doi:10.1145/1007512.1007523 Controlled experiment on tests that run automatically as code is edited: participants were three times more likely to finish before the deadline, with no effect on time worked. Tests throughout help by being run often.
• Yoo, S. and Harman, M. (2012). Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability 22(2). doi:10.1002/stvr.430 Survey of minimisation, selection and prioritisation of regression tests, and the trade-offs between early detection and running cost. The reason a growing suite is not simply run whole.
• Inozemtseva, L. and Holmes, R. (2014). Coverage is not strongly correlated with test suite effectiveness. ICSE 14. doi:10.1145/2568225.2568271 Studied the correlation between suite coverage and fault detection across real and synthetic programs, controlling for suite size. Once size was controlled, coverage was not strongly correlated with effectiveness. Contests coverage as a measure of safety.
• Luo, Q., Hariri, F., Eloussi, L. and Marinov, D. (2014). An empirical analysis of flaky tests. FSE 14. doi:10.1145/2635868.2635920 Studied 201 commits that likely fix flaky tests across 51 open-source projects, and classified the root causes. Nondeterministic outcomes undermine the assumption regression testing rests on, so a test can lie.
• Claessen, K. and Hughes, J. (2000). QuickCheck: a lightweight tool for random testing of Haskell programs. ICFP 00. doi:10.1145/351240.351266 Present QuickCheck, in which a property is an executable predicate checked against random inputs. Case studies found faults on real programs, random testing suiting functional code because properties can be stated finely. Catches a class of defect example tests miss.
• Banerjee, I., Nguyen, B., Garousi, V. and Memon, A. (2013). Graphical user interface (GUI) testing: systematic mapping and repository. Information and Software Technology 55(10). doi:10.1016/j.infsof.2013.03.004 Mapped GUI-testing research from 230 articles to a final pool of 136, cataloguing models and techniques. A GUI test case is a sequence of user input events on widgets, so the interface is the hardest object to test here.
• Mathews, N. S. and Nagappan, M. (2024). Test-driven development and LLM-based code generation. ASE 24. doi:10.1145/3691620.3695527 Gave GPT-4 and Llama 3 tests alongside the problem statement on standard code-generation benchmarks and found consistently higher success than from the statement alone. Shows the test acting as a specification where a model writes the code.
• Causevic, A., Sundmark, D. and Punnekkat, S. (2011). Factors limiting industrial adoption of test driven development: a systematic review. ICST 11. doi:10.1109/ICST.2011.19 Systematic review of why industry does not adopt test-driven development. Seven limiting factors, including increased development time and weak adherence to the protocol. Adoption, not only effect, conditions the practice.