Debugging is comprehension, and the tool is not the skill

A failure appears where a person can see it, and a fault sits where the programmer wrote it. A total is wrong because a discount was applied twice. A panel is blank because a response was refused. This unit argues that closing the gap is a comprehension problem. The programmer must build a correct model of what the program did. The gap between the debugger's reputation and its measured use shows that having a tool does not supply the skill to interrogate it. A debugger shows the state you told it to show. It does not tell you what to ask. The strongest counter-argument, taken up below, is that the record measures today's tools rather than the skill. A debugger built around line breakpoints will be used for line breakpoints, and a why-question interface cut debugging time by nearly a factor of eight. Unit 8 turns from finding a fault to making the fix safe to keep.

Comprehension is the field the skill belongs to

The study of how programmers understand code began with models rather than tools. Von Mayrhauser and Vans compared six comprehension models (von Mayrhauser and Vans, 1995). Among them were a top-down model, in which prior expectations guide the reading, a bottom-up model built from the smallest statements upwards, and an integrated model that moves between the two as the task demands. Storey reviewed thirty years of that literature, together with the tools built on it, and judged that theory and tool have matured unevenly (Storey, 2006). Debugging inherits this theory, so a claim that a tool fixes debugging is a claim about comprehension, and can be tested.

What the debugger is used for, and what the record shows

Beller and colleagues gave the question its largest measurement. They surveyed 176 developers, instrumented 458 who installed a purpose-built plug-in for Eclipse and IntelliJ, and interviewed participants to explain where the two disagreed (Beller et al., 2018). The debugger ran in 9 per cent of the 723 observed sessions, and took 13.7 per cent of active in-IDE development time. Yet 81.3 per cent of the surveyed developers said they used it, and 71.6 per cent reported print statements. Two thirds of the instrumented users never started it, against their own account, and when it was used it was line breakpoints (Beller et al., 2018).

The authors draw an educational conclusion rather than a tooling one: "These results call to strengthen hands-on debugging experience in computer science curricula" (Beller et al., 2018).

Perscheid and colleagues reached the same pattern by another method, combining a literature review, a field study of eight professional developers in four companies, and a survey (Perscheid et al., 2017). Few knew all the features of their tool. Back-in-time debuggers and automatic fault localisation were largely unknown, and the faults they called hard were the ones that made them rebuild their model of the program (Perscheid et al., 2017). The evidence has bounds. The developers opted in as plug-in users, and self-report overstates practice. The samples are professional developers, so the record describes people reading English-language tool output with a team to ask, not a self-taught learner meeting the same fault alone.

The tool that answers the question the programmer is asking

Ko and Myers built the Whyline so that a programmer can ask why a particular output occurred, or why it did not. It returns the runtime events relevant to that question, rather than a dump of state (Ko and Myers, 2004). A worked case makes the difference concrete. A shape on a canvas stays the wrong colour after an edit, and a plain debugger shows the variables at a breakpoint the programmer chose, which may sit far from the cause. The Whyline lets the programmer select the shape and ask why it did not become blue. It answers with the run-time events that set that attribute, in order, so the failing assignment is read out of the history rather than guessed. The interface chooses how to answer, but not which question to ask. On identical scenarios it cut debugging time by nearly a factor of eight and increased completed tasks by 40 per cent (Ko and Myers, 2004). Ko, Myers and Aung show why the question matters. Studying beginners learning a visual environment, they identified six barriers, among them understanding as distinct from use (Ko, Myers and Aung, 2004).

Automating the search still leaves a reading task

The strongest competing account says that the search for the faulty statement is the expensive part, and that automating it is the fix. That account has results. Jones and Harrold compared their Tarantula technique with four other localisation techniques on the same subjects. Tarantula ranks statements by how strongly their execution tracks failing rather than passing tests, and it was consistently more effective and comparable in efficiency to the cheapest (Jones and Harrold, 2005). Jones, Harrold and Stasko mapped each statement's part in passing and failing tests as a colour. They found that the technique can help a user locate faults in the subject studied (Jones, Harrold and Stasko, 2002). Liblit and colleagues sample predicates across many runs to isolate several undiagnosed faults at once, and found previously unknown crashing bugs in widely used systems (Liblit et al., 2005).

Each result narrows where to look and returns something still to be read. A ranking says that a statement is suspicious, not what the program was doing when it reached that statement. Parnin and Orso made that gap their subject. Localisation rests on assumptions about behaviour, they note, such as that seeing the faulty statement in isolation is enough to understand and repair the bug. A root cause usually requires navigating dependencies and rerunning the program on different inputs (Parnin and Orso, 2011). They gave developers a tool and two tasks, one with it and one without, and found those assumptions did not hold.

The skill is teachable, and the hardest faults are conceptual

The comprehension account makes two predictions. Debugging skill is learnt separately from program knowledge, and the hardest faults break an assumption rather than merely hide. Both predictions have support. Ahmadzadeh, Elliman and Higgins analysed novices' compiler errors and debugging activity. Many who understood programming well had not acquired the skill to debug effectively, they found, and they argue for teaching it explicitly (Ahmadzadeh, Elliman and Higgins, 2005). Eisenstadt asked practitioners for their worst bugs, and analysed 59 accounts by why the bug was difficult, how it was found and its root cause (Eisenstadt, 1997). Zeller turns the pattern into a method: reproduce, form and test hypotheses, reduce the failing input until what remains is minimal, and follow cause and effect through the run (Zeller, 2009).

Contested question: does the tool carry the skill, or only the search?

The literature disagrees about what the evidence shows, and about what the data are evidence of. The search camp holds that finding the faulty statement is the expensive part, so more tool coverage is the fix. Its evidence is comparative. Tarantula beat four rival localisation techniques on the same subjects (Jones and Harrold, 2005), and a colour map of test outcomes helped users locate faults in the subject studied (Jones, Harrold and Stasko, 2002). Statistical isolation found previously unknown crashing bugs in widely used systems (Liblit et al., 2005), and delta debugging reduces a failing input mechanically (Zeller, 2009). Those are real gains, and each was measured against another technique or against no tool, so none measures what a person does with the output.

The comprehension camp holds that a tool answers one question while the difficulty lies in forming the others. Its strongest single result is the Whyline, which cut debugging time by nearly a factor of eight on identical scenarios. That gain came from answering a why-question rather than from displaying more state (Ko and Myers, 2004). The strongest evidence about practice points the same way. Beller and colleagues measured the debugger in 9 per cent of 723 sessions, while 81.3 per cent of surveyed developers said they used it (Beller et al., 2018). The bounds matter on both sides: the Whyline study is old, runs in a laboratory on small programs, and the instrumented developers opted in, so their behaviour may not represent a team under deadline.

A third strand explains why the camps read one set of data differently. Parnin and Orso gave real developers an automated tool and two tasks, and the assumptions those techniques make about behaviour did not hold (Parnin and Orso, 2011). That is a negative result about one tool and one pair of tasks, so it constrains the search camp without settling the comprehension camp. What would settle it is an experiment nobody here has run. Hold the fault, the task and the time constant, and vary only the interface. Compare ranked suspect statements, a why-question interface and a plain debugger, and measure time to a correct statement of the root cause, not to the faulty line. Until then a teacher should require a written hypothesis before any tool is opened, and treat the debugger as one instrument for testing it.

Where the argument is heading

Three named lines of work point the same way. Interrogative debugging was built once and was not absorbed into mainstream tools, though the Whyline beat a plain debugger by nearly a factor of eight (Ko and Myers, 2004). Beller and colleagues, by contrast, find the debugger they measured used for line breakpoints and little else (Beller et al., 2018). Automated isolation has moved from ranking statements to sampling behaviour, and returns circumstances rather than a verdict (Liblit et al., 2005). Parnin and Orso ask that such techniques be evaluated with humans on realistic tasks (Parnin and Orso, 2011).

The sources this unit rests on

• Beller, M., Spruit, N., Spinellis, D. and Zaidman, A. (2018). On the dichotomy of debugging behaviour among programmers. ICSE 18, 572-583. doi:10.1145/3180155.3180175 Surveyed 176 developers and instrumented 458 in Eclipse and IntelliJ. The debugger ran in 9 per cent of 723 sessions and took 13.7 per cent of active IDE time, while 81.3 per cent reported using it. The gap between reputation and use.
• Perscheid, M., Siegmund, B., Taeumel, M. and Hirschfeld, R. (2017). Studying the advancement in debugging practice of professional software developers. Software Quality Journal 25(1), 83-110. doi:10.1007/s11219-015-9294-2 Combined a literature review, a field study of eight developers in four companies and a survey. Advanced tools were largely unknown, and the faults called hard were those that required rebuilding a model of the program. A second confirmation that capability and practice are separate.
• Ko, A. J. and Myers, B. A. (2004). Designing the whyline: a debugging interface for asking questions about program behaviour. CHI 04. doi:10.1145/985692.985712 Built an interface in which a programmer asks why an output occurred and receives the relevant runtime events. It cut debugging time by nearly a factor of eight and raised completed tasks by 40 per cent. Shows the interface, not the volume of state, carries the comprehension step.
• Ko, A. J., Myers, B. A. and Aung, H. H. (2004). Six learning barriers in end-user programming systems. VL/HCC 04. doi:10.1109/VLHCC.2004.47 Studied beginners learning a visual environment and identified design, selection, coordination, use, understanding and information as distinct barriers. Understanding is separate from use, so an environment can be operable and still unpredictable. Explains why a debugger can show a learner unreadable values.
• Jones, J. A. and Harrold, M. J. (2005). Empirical evaluation of the tarantula automatic fault-localization technique. ASE 05. doi:10.1145/1101908.1101949 Compared Tarantula, which ranks statements by the outcomes of failing and passing tests, with four other techniques on the same subjects. Tarantula was consistently more effective and comparable in efficiency to the cheapest. The strongest evidence that automating the search works.
• Jones, J. A., Harrold, M. J. and Stasko, J. (2002). Visualization of test information to assist fault localization. ICSE 02. doi:10.1145/581339.581397 Mapped each statement's participation in passing and failing test outcomes as a colour and evaluated the prototype in empirical studies. The authors conclude it can help a user locate faults in the subject studied. Shows visual support helps the search.
• Liblit, B., Naik, M., Zheng, A. X., Aiken, A. and Jordan, M. I. (2005). Scalable statistical bug isolation. PLDI 05. doi:10.1145/1065010.1065014 Samples predicates across many runs to isolate faults in programs holding several undiagnosed bugs, separating their effects and estimating each failure mode's frequency. Case studies included previously unknown crashing bugs in widely used systems. Returns circumstances, not a cause.
• Parnin, C. and Orso, A. (2011). Are automated debugging techniques actually helping programmers? ISSTA 11. doi:10.1145/2001420.2001445 Gave developers an automated tool and two tasks, one with it and one without, and examined the assumptions these techniques make about behaviour. The assumptions did not hold in practice. The clearest statement that ranking statements is not the whole task.
• Ahmadzadeh, M., Elliman, D. and Higgins, C. (2005). An analysis of patterns of debugging among novice computer science students. ITiCSE 05. doi:10.1145/1067445.1067472 Analysed students' compiler errors and debugging activity. Many who understood programming well had not acquired the skill to debug effectively, and the authors argue for teaching debugging explicitly. Shows the skill is separable from program knowledge.
• Eisenstadt, M. (1997). My hairiest bug war stories. Communications of the ACM 40(4), 30-37. doi:10.1145/248448.248456 Solicited debugging anecdotes from practitioners and analysed 59 accounts by why the bug was difficult, how it was found and its root cause. Practitioners' own accounts of the hardest faults.
• Zeller, A. (2009). Why Programs Fail: A Guide to Systematic Debugging. 2nd edition. Morgan Kaufmann. https://www.elsevier.com/books/why-programs-fail/zeller/978-0-12-374515-6 Presents debugging as a systematic discipline: reproducing failures, testing hypotheses, reducing a failing input to a minimal case and tracking cause and effect. A method text rather than an experiment; supplies the teachable procedure the thesis needs.
• von Mayrhauser, A. and Vans, A. M. (1995). Program comprehension during software maintenance and evolution. IEEE Computer 28(8). doi:10.1109/2.402076 Compared six comprehension models, including top-down, bottom-up and an integrated model, and raised open questions about the scalability of results from small programs. Establishes that reading code is theory-driven model building.
• Storey, M.-A. (2006). Theories, tools and research methods in program comprehension. Software Quality Journal 14(3), 187-208. doi:10.1007/s11219-006-9216-4 Reviewed thirty years of comprehension theories and the tools built on them, and reflected on how each was evaluated. Tool and theory have matured unevenly, which sets the research agenda the closing paragraph follows.