Testing and the contested case for test-first

The sixth unit is the second of those four, and asks how a team knows a change is safe. Three levels of test, one suite, and a promise that the code works. No suite proves a program correct. What a suite buys is information about what the next change will break. The evidence that writing the test first improves quality or speed is small, and it is partly negative. That evidence depends on how a team runs the process, not on the order of two actions. Test-first survives as a discipline for design and feedback, not as a measured quality or productivity win.

Testing is a portfolio, and every technique misses something

Basili and Selby (1987) put three techniques side by side. Thirty-two professional programmers and 42 advanced students worked on four unit-sized programs. They applied code reading by stepwise abstraction, functional testing by equivalence partitioning and boundary value analysis, and structural testing at 100 per cent statement coverage. Among the professionals, code reading detected more faults than either automated technique. Code reading also found more interface faults, while functional testing found more control faults. The faults observed, the rate and the effort depended on the software tested.

No technique dominates. Two techniques that both claim to check a program find different failures. A suite is a portfolio, and its value lies in which failures it can surface.

What test-first claims, and why the claim is about design

Beck (2002) states two rules. Write new code only if you first have a failing automated test, and eliminate duplication. The order they imply is red, green, refactor, which he calls the TDD mantra. The tests are the teeth of a ratchet that holds each working step while the next begins. Beck derives design consequences from them. He argues that designs must consist of many highly cohesive, loosely coupled components, just to make testing easy, and that design proceeds organically with running code as feedback. Fucci and others (2017) give the process its neutral description: a unit test for the desired functionality, then the production code, then refactoring.

A money type adds two currencies, so that five dollars plus ten francs equals ten dollars at a rate of two to one. Beck builds it one test at a time from that demand (Beck, 2002).

The measured case is small, and it depends on the setting

Rafique and Misic (2013) pooled 27 studies. They found a small positive effect of test-driven development on external quality, and little to no discernible effect on productivity. Their subgroup analysis found the quality gain and the productivity drop to be much larger in industrial studies than in academic ones. It also found a statistically significant positive correlation between task size and the size of the quality gain.

George and Williams (2003) supply a primary study with the same shape. Twenty-four professional pair programmers built a small Java program, one group test-first and one waterfall-like. The test-first pairs passed 18 per cent more functional black box test cases, and they took 16 per cent more time. The control programmers often failed to write the required automated tests after finishing.

The experiment that dissolves the mechanism

Fucci and others (2017) attack the causal claim directly. They collected 82 data points from 39 professionals, each recording the process used on a specific task. They modelled sequencing, granularity, uniformity and refactoring effort. Whether the test or the production code came first had no important influence. More refactoring effort went with worse outcomes on both measures. The claimed benefits of test-driven development may not be due to its distinctive test-first dynamic at all, but to the fact that such processes encourage something else. One study cannot settle a mechanism, and this one follows professionals on tasks rather than teams.

The number teams quote is not a measure of the suite

Inozemtseva and Holmes (2014) generated 31,000 test suites for five Java systems of up to 724,000 lines of source code. They measured statement, decision and modified condition coverage, and they used mutation testing to evaluate what each suite could detect. With the number of test cases controlled for, the correlation between coverage and effectiveness was low to moderate. Stronger forms of coverage gave no greater insight than weaker ones. Coverage stays useful for finding under-tested code, but a definition of done that names a percentage names an instrument that predicts the property weakly.

A suite that fails unreliably is a broken instrument

Gruber and others (2021) sampled 22,352 open-source Python projects and analysed 876,186 test cases. They found 7,571 flaky tests. Flakiness was about as prevalent in Python as in Java, but it had different causes. Order dependency accounted for 59 per cent of the flaky tests, test infrastructure for 28 per cent, and network and randomness APIs for most of the rest. A developer on an intermittent connection cannot reproduce the network-dependent kind. Parry and others (2021) survey 76 papers on the problem and state the consequence for this argument. Flaky tests threaten the validity of any method that assumes a test's outcome depends only on the code it covers. A survey they report found 59 per cent of developers dealing with flaky tests monthly, weekly or daily. A red suite is information, and information needs a deterministic instrument.

The first extensive study of the problem examined 201 commits that fixed flaky tests across 51 open-source projects and put order dependence among the most common causes (Luo and others, 2014).

The tests are code too, and they decay

van Deursen and others (2001) observed that in extreme programming the ideal ratio of test code to production code approaches one to one. They also observed that refactoring tests differs from refactoring production code: a distinct set of bad smells appears, and improving tests needs additional test-specific refactorings. Bavota and others (2015) supplied the measurement that vocabulary lacked, in two studies. The first found test smells widely diffused, with 86 per cent of the JUnit tests examined carrying at least one. The second found a strong negative impact on program comprehension and maintenance, with comprehension 30 per cent better in their absence. The design discipline Beck derived from testability extends to the tests.

Contested question: does the order of writing cause anything?

Three positions disagree about what test-first does. The first holds that the order carries a real if modest effect on quality. Rafique and Misic (2013) are its measured statement. Madeyski (2010) is its strongest single programme, reporting three experiments that used acceptance tests passed and design complexity as indicators. The second holds that the order is not the cause. Fucci and others (2017) find sequencing without important influence, while granularity and uniformity carry the effects. George and Williams (2003) had already conjectured that their result came from the granularity of the cycle rather than the sequence. The third does not argue about outcomes at all. Beck (2002) offers the rules as a way to manage fear and to force a design, and a productivity measurement cannot refute a design claim.

One study would settle it. In that study the order of writing is the manipulated variable, and everything else stays constant: the same granularity, refactoring effort, test effort and task size, with whole teams followed for months. Nothing here does that. Garousi and Mantyla (2016) examined 101 secondary studies and named test-driven development among the topics still short of a secondary study.

What to do while the dispute is open

The evidence is small on every side. A meta-analysis of 27 studies finds a modest quality effect and no productivity effect (Rafique and Misic, 2013). A dissection of 82 data points finds the sequence immaterial, with granularity and uniformity carrying the improvements (Fucci and others, 2017). The strongest programme for the order comes from one laboratory tradition (Madeyski, 2010). The strongest objection to this unit's thesis is that it measures the wrong thing, because Beck (2002) offers test-first as a way to hold attention and force a structure, and a study of quality cannot see the structure. While the dispute is open, a designer should hold the cycle small and uniform, and treat the order of the two keystrokes as taste rather than a target.

Where the argument is heading

The direction is from labels towards measured properties of the process. A team can record and model granularity, uniformity and refactoring effort (Fucci and others, 2017). A team can keep coverage as a diagnostic for under-tested code, not as a target (Inozemtseva and Holmes, 2014). A team can detect flakiness and classify it as an ordinary property of a suite (Gruber and others, 2021; Parry and others, 2021). The map of what is unexamined is now explicit (Garousi and Mantyla, 2016). Name the discipline it is buying, say which property of the process it is trying to hold, and stop treating the order of two keystrokes as the question.

The sources this unit rests on

• Basili, V. R. and Selby, R. W. (1987). Comparing the Effectiveness of Software Testing Strategies. IEEE Transactions on Software Engineering SE-13(12), 1278-1296. doi:10.1109/TSE.1987.232881 Thirty-two professionals and 42 students applied code reading, functional testing and 100 per cent statement coverage to four programs. Code reading found the most faults and each technique found different fault classes, so a suite is a portfolio rather than a proof.
• Beck, K. (2002). Test Driven Development: By Example. Addison-Wesley, Boston, MA. https://www.informit.com/store/test-driven-development-by-example-9780321146533 The book states the two rules and the order red, green, refactor. Beck argues the implications include designing organically and building highly cohesive, loosely coupled components just to make testing easy, so it is cited for the doctrine, not an outcome measurement.
• George, B. and Williams, L. (2003). An Initial Investigation of Test Driven Development in Industry. In Proceedings of the 2003 ACM Symposium on Applied Computing (SAC 2003). ACM, New York, 1135-1139. doi:10.1145/952532.952753 Twenty-four professional pair programmers built a small Java program, one group test-first and one waterfall-like. The test-first pairs passed 18 per cent more functional black box test cases at the cost of 16 per cent more time.
• Madeyski, L. (2010). Test-Driven Development: An Empirical Evaluation of Agile Practice. Springer, Berlin. doi:10.1007/978-3-642-04288-1 The book reports three experiments that used the number of acceptance tests passed and design complexity metrics as evaluation indicators, and argues for the benefits of agile practice. It is the strongest single programme for test-first, though its evidence comes from one laboratory tradition.
• van Deursen, A., Moonen, L., van den Bergh, A. and Kok, G. (2001). Refactoring Test Code. CWI Report SEN-R0119, Centrum Wiskunde en Informatica, Amsterdam. https://ir.cwi.nl/pub/4324 The report notes that in extreme programming the test-to-production ratio approaches one to one. It argues that refactoring tests differs from refactoring production code, with a distinct set of bad smells and additional test-specific refactorings, and is the origin of the test smell vocabulary.
• Rafique, Y. and Misic, V. B. (2013). The Effects of Test-Driven Development on External Quality and Productivity: A Meta-Analysis. IEEE Transactions on Software Engineering 39(6), 835-856. doi:10.1109/TSE.2012.28 A meta-analysis of 27 studies found a small positive effect on external quality and little to no discernible effect on productivity. Both effects were larger in industrial studies and the quality gain correlated significantly with task size, which is the unit's central measurement.
• Inozemtseva, L. and Holmes, R. (2014). Coverage Is Not Strongly Correlated with Test Suite Effectiveness. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, 435-445. doi:10.1145/2568225.2568271 The study generated 31,000 test suites for five Java systems of up to 724,000 lines and used mutation testing to measure effectiveness. With suite size controlled, coverage correlated only low to moderately with effectiveness, so the metric teams quote is not the property they want.
• Bavota, G., Qusef, A., Oliveto, R., De Lucia, A. and Binkley, D. (2015). Are test smells really harmful? An empirical study. Empirical Software Engineering 20(4), 1052-1094. doi:10.1007/s10664-014-9313-0 Two studies found high diffusion of test smells, with 86 per cent of JUnit tests carrying at least one. The second found a strong negative impact on comprehension and maintenance, with comprehension 30 per cent better without them, which is the measured harm the vocabulary lacked.
• Garousi, V. and Mantyla, M. V. (2016). A systematic literature review of literature reviews in software testing. Information and Software Technology 80, 195-216. doi:10.1016/j.infsof.2016.09.002 A tertiary study mapped 101 secondary studies published between 1994 and 2015. It found important areas with none, including test management, human factors in testing, exploratory testing and test stopping criteria, and names test-driven development as still short of one.
• Fucci, D., Erdogmus, H., Turhan, B., Oivo, M. and Juristo, N. (2017). A Dissection of the Test-Driven Development Process: Does It Really Matter to Test-First or to Test-Last? IEEE Transactions on Software Engineering 43(7), 597-614. doi:10.1109/TSE.2016.2616877 From 82 data points on 39 professionals the authors modelled sequencing, granularity, uniformity and refactoring effort. Granularity and uniformity carried the improvements and sequencing had no important influence, which is the strongest challenge to the test-first claim.
• Gruber, M., Lukasczyk, S., Kroiss, F. and Fraser, G. (2021). An Empirical Study of Flaky Tests in Python. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST 2021). IEEE, Piscataway, NJ, 148-158. doi:10.1109/ICST49551.2021.00026 The study analysed 876,186 test cases from 22,352 Python projects and found 7,571 flaky tests. Order dependency caused 59 per cent, test infrastructure 28 per cent and network or randomness APIs most of the rest, so flakiness is a property of practice rather than of one language.
• Parry, O., Kapfhammer, G. M., Hilton, M. and McMinn, P. (2021). A Survey of Flaky Tests. ACM Transactions on Software Engineering and Methodology 31(1), Article 17, 1-74. doi:10.1145/3476105 The survey covers 76 papers on the causes, costs, detection and repair of flaky tests. A developer survey it reports found 59 per cent dealing with flaky tests monthly, weekly or daily, and the authors frame flakiness as a threat to any method that assumes a test outcome depends only on the code it covers.
• Luo, Q., Hariri, F., Eloussi, L. and Marinov, D. (2014). An Empirical Analysis of Flaky Tests. In FSE 2014. New York: ACM, 643-653. doi:10.1145/2635868.2635920 The first extensive study of flaky tests examined 201 commits that fixed them across 51 open-source projects, with order dependence among the most common root causes.