Code review: what it is actually for

Year 2 / Module 16 Software engineering for interactive products / Unit 8 / Code review: what it is actually for

HCI2108 · Unit 8 of 10

The defect-finding yield of review is small and mostly local, and its larger product is shared understanding of a change, so a review process judged only on defects found will be dropped in the first busy quarter.

Code review: what it is actually for

The eighth unit closes those four, and turns to the person who reads a change. Two claims carry this unit, and the literature supports them unequally. The first is that review finds fewer defects than its reputation suggests, and mostly local ones: a wrong condition, a missing check, a name that misleads. The second is that its larger product is a shared understanding of the change. A process judged only on defects found will look like a poor return and will be dropped in a busy quarter, though it does work nothing else does.

Inspection supplied the claim the practice still trades on

Fagan (1976) is the origin of peer review as a defined process rather than a meeting. He set out design and code inspections as a sequence of operations, each with exit criteria. The objectives stay separate, so a team attends to one at a time, and the team classifies errors by type and ranks them by frequency.

Fagan (1986) states the claim the practice still trades on: inspection "usually detects more defects in the product", at lower cost than machine testing.

The paper also reports quality improving as costs fell, and adds a hedged sentence that anticipates the second half of this argument: there is "some evidence" that developers who inspect their own product create fewer later defects.

The modern practice is a different process

What most teams call review is not inspection, and the difference was documented before the tooling arrived. Rigby, German and Storey (2008) reconstructed the Apache server's two review techniques from email archives and version control, finding early and frequent reviews of small, independent and complete contributions, run asynchronously by a small group of self-selected experts.

Rigby and Bird (2013) tested whether that shape is local to open source, comparing Google-led, Microsoft and AMD internal projects against a Lucent inspection and six open-source projects. Many characteristics had converged on similar values across different settings and incentive systems, which the authors treat as general principles, and they introduce a measure of how much knowledge is shared during a review.

Sadowski and others (2018) supply the commercial case at scale: twelve interviews, a survey with 44 respondents and the review logs for nine million reviewed changes at Google. Being descriptive, it cannot attribute outcomes to the practice.

A review comment is mostly an improvement, not a defect

Bacchelli and Bird (2013) observed 17 developers across 16 Microsoft product teams, classified 570 comments, and surveyed 165 managers and 873 programmers. Finding defects was the most common stated motivation. Defect comments were only the fourth most frequent of nine categories, at 78 comments or 14 per cent. Code improvements led with 165 comments or 29 per cent, covering better practices, deleted code and readability.

Reviews "are less about defects than expected and instead provide additional benefits such as knowledge transfer, increased team awareness, and creation of alternative solutions to problems." (Bacchelli and Bird, 2013)

Code and change understanding is the key aspect of reviewing, and current tools do not meet most of the mechanisms developers use to reach it. Knowledge transfer appears as only 12 explicit comments, because it leaves little trace to count.

Review tools present changed files in alphabetical order of path. In a study of real review sessions a little over half began with the file the tool put first, and more than a third of moves between files followed that order, although the reviewers described it as poor for understanding (Baum, Schneider and Bacchelli, 2017). That study is a mixed-methods account from interviews, a survey and review logs, and it does not show that another order finds more defects.

Mantyla and Lassenius (2009) sharpen this by classifying defects rather than counting them. From nine industrial reviews in C and C++ and 23 student reviews in Java they recorded 388 and 371 defects. Three quarters did not affect the visible functionality of the software; the largest group, which they call evolvability defects, make code harder to understand and modify.

The outcome evidence, and what it cannot separate

Against that, the strongest outcome evidence points at the process rather than the comment. McIntosh and others (2016) studied the Qt, VTK and ITK projects, using post-release defects as a proxy for long-term quality. Review coverage, participation and reviewer expertise each share a significant link with quality.

Both findings can hold at once. A defect yield of 14 per cent describes individual comments; a link between coverage and post-release defects describes a process across thousands of changes. Evolvability defects left in place make later changes riskier, so a small comment count can sit behind a real difference in shipped defects. The study is observational, so nobody can read the mechanism off the comments.

Who reviews decides what review is

If review is a team practice rather than a code filter, who reviews is part of its mechanism. Thongtanunam and others (2015) measured the cost of assigning badly. Reviews with a code-reviewer assignment problem took 12 days longer to approve a change. Bosu and Carver (2014) split developers in eight open-source projects into core and periphery. Core developers got earlier feedback, faster completion and more acceptances, while peripheral developers could wait 2 to 19 times longer, between 12 and 96 hours.

Baysal and others (2016) widened the same finding, reverse engineering patch submission for WebKit and Google Blink. Patch size, priority and component mattered, and the organisation behind the contributor and the developer's review load and experience mattered too, so non-technical factors significantly affect response time and outcome. A review metric measures the social system as much as the code. A contributor outside the core team, or writing in a second language, meets it with less standing.

Contested question: is review a quality gate or a knowledge practice?

The contested question is not whether review finds defects. It is which outcome justifies the time. Fagan (1976, 1986) and McIntosh and others (2016) treat review as a quality gate justified at the level of the process: coverage, participation and expertise each share a significant link with post-release defects. A small share of defect comments then shows the gate is cheap rather than unimportant, and a defect removed before release costs less than one removed after it.

Bacchelli and Bird (2013) and Mantyla and Lassenius (2009) treat review as a knowledge practice. Reviewers mostly produce a better shape for the code and a shared understanding of it, and defect comments are a small and local minority. On this reading a defect count is the wrong instrument. The process would survive the automation of a quality gate, and it would not survive the loss of the understanding. Those who value understanding must argue from interviews, from 12 explicit comments, and from the knowledge-sharing measure of Rigby and Bird (2013).

One design would settle it, and neither side has produced it. Classify the comments in a large review archive and link each category to defects found later in the same files, controlling for change size and reviewer expertise, to show which comment categories predict fewer post-release defects. The harder test turns review on or off in one organisation and follows defect outcomes and a validated measure of understanding over time.

What to do while the dispute is open

The evidence is asymmetric. The quality-gate case reaches outcome data, linking review coverage, participation and expertise to post-release defects (McIntosh and others, 2016). The knowledge case rests on interviews, 12 explicit comments and one sharing measure, because understanding leaves little trace to count (Bacchelli and Bird, 2013; Rigby and Bird, 2013). The strongest objection to this unit's thesis is that the understanding it credits may be a by-product of a process justified on defects alone. While the dispute is open, spend the scarce attention on the shape of the change, not on what automation can check.

Where the argument is heading

The live direction is automation, and it presses on the frontier between the two halves. Tufano and others (2021) trained two deep learning architectures on real review activity, one proposing a revised change from the transformations reviewers request, one generating review comments. That aims at the local comment, the 29 per cent of improvements and the 14 per cent of defects. Cihan and others (2025) studied three projects in one company that adopted a large language model review tool. Across 4,335 pull requests, 1,568 received an automated review and 73.8 per cent of the automated comments were resolved, while the average pull request took longer to close. Practitioners reported a minor quality gain alongside faulty reviews and irrelevant comments. The tool absorbed the local comment and charged the queue for it. The understanding is what remains, and its nearest instrument is still the measure of Rigby and Bird (2013). Keep the change small, write a description that lets a reviewer reconstruct the decision, and measure what only people contribute.

The sources this unit rests on

Fagan, M. E. (1976). Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal 15(3), 182-211. doi:10.1147/sj.153.0182 The paper defines design and code inspections as a sequence of operations, each with exit criteria, and instructs teams to classify errors by type and rank their frequency; it is the origin of peer review as a defined process rather than a meeting.
Fagan, M. E. (1986). Advances in Software Inspections. IEEE Transactions on Software Engineering 12(7), 744-751. doi:10.1109/TSE.1986.6312976 The follow-up generalises inspection to further stages and states that it usually detects more defects, at lower cost, than machine testing. It adds hedged evidence that developers who inspect their own product create fewer later defects.
Bacchelli, A. and Bird, C. (2013). Expectations, Outcomes, and Challenges of Modern Code Review. In ICSE 2013. IEEE, 712-721. doi:10.1109/ICSE.2013.6606617 Observation of 17 developers across 16 Microsoft teams, 570 classified comments and surveys of 165 managers and 873 programmers; defect comments were the fourth of nine categories at 14 per cent, while code improvements led at 29 per cent.
Rigby, P. C., German, D. M. and Storey, M.-A. (2008). Open Source Software Peer Review Practices: A Case Study of the Apache Server. In ICSE 2008. ACM, 541-550. doi:10.1145/1368088.1368162 A case study of the Apache server measuring review frequency, participation, contribution size and time to review from email archives and version control. Apache reviews are early and frequent, of small, independent and complete contributions, and run asynchronously.
Rigby, P. C. and Bird, C. (2013). Convergent Contemporary Software Peer Review Practices. In FSE 2013. ACM, 202-212. doi:10.1145/2491411.2491444 Review measures from Google-led, Microsoft and AMD internal projects, contrasted with a Lucent inspection and six open-source projects, showed characteristics converging on similar values across settings, which the authors read as general principles; they also introduce a measure of knowledge shared during review.
Sadowski, C., Soderberg, E., Church, L., Sipko, M. and Bacchelli, A. (2018). Modern Code Review: A Case Study at Google. In ICSE-SEIP 2018. ACM, 181-190. doi:10.1145/3183519.3183525 An exploratory case study at Google using 12 interviews, a survey with 44 respondents and logs for nine million reviewed changes. It establishes the process at industrial scale and, being descriptive, cannot attribute outcomes to the practice.
McIntosh, S., Kamei, Y., Adams, B. and Hassan, A. E. (2016). An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering 21(5), 2146-2189. doi:10.1007/s10664-015-9381-9 A case study of the Qt, VTK and ITK projects relating post-release defects, a proxy for long-term quality, to review coverage, participation and reviewer expertise, each of which shares a significant link with quality.
Mantyla, M. V. and Lassenius, C. (2009). What Types of Defects Are Really Discovered in Code Reviews? IEEE Transactions on Software Engineering 35(3), 430-448. doi:10.1109/TSE.2008.71 Defects were classified in nine industrial C and C++ reviews and 23 student Java reviews, recording 388 and 371 defects; three quarters did not affect visible functionality, and the largest group, evolvability defects, made code harder to understand and modify.
Thongtanunam, P., Tantithamthavorn, C., Kula, R. G., Yoshida, N., Iida, H. and Matsumoto, K. (2015). Who Should Review My Code? A File Location-Based Code-Reviewer Recommendation Approach for Modern Code Review. In SANER 2015. IEEE, 141-150. doi:10.1109/SANER.2015.7081824 The study measured what happens when the right reviewer is hard to find: reviews with a code-reviewer assignment problem take 12 days longer to approve a change.
Bosu, A. and Carver, J. C. (2014). Impact of Developer Reputation on Code Review Outcomes in OSS Projects. In ESEM 2014. ACM, Article 33. doi:10.1145/2652524.2652544 Social network analysis of review data from eight open-source projects divided developers into core and periphery groups using six centrality measures; core developers got earlier feedback and faster completion, while peripheral developers waited 2 to 19 times longer, between 12 and 96 hours.
Baysal, O., Kononenko, O., Holmes, R. and Godfrey, M. W. (2016). Investigating technical and non-technical factors influencing modern code review. Empirical Software Engineering 21(3), 932-959. doi:10.1007/s10664-015-9366-8 The authors reverse engineered patch submission for WebKit and Google Blink, confirming that patch size, priority and component matter and adding the organisation behind the contributor and the developer's review load; non-technical factors significantly affect response time and outcome.
Baum, T., Schneider, K. and Bacchelli, A. (2017). On the Optimal Order of Reading Source Code Changes for Review. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME 2017). Piscataway, NJ: IEEE, 329-340. doi:10.1109/ICSME.2017.28 A mixed-methods study of how reviewers read a change, from interviews, a survey and review logs; reviewers largely followed the tool's file order while reporting it as poor for understanding.
Tufano, R., Pascarella, L., Tufano, M., Poshyvanyk, D. and Bavota, G. (2021). Towards Automating Code Review Activities. In ICSE 2021. IEEE, 163-174. doi:10.1109/ICSE43902.2021.00027 Two deep learning architectures were trained on real review activity, one proposing a revised change from the transformations reviewers request, one generating review comments; review became a target for automation.
Cihan, U., Haratian, V., Icoz, A., Gul, M. K., Devran, O., Bayendur, E. F., Ucar, B. M. and Tuzun, E. (2025). Automated Code Review in Practice. In Proceedings of the 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP 2025). Piscataway, NJ: IEEE, 425-436. doi:10.1109/ICSE-SEIP66354.2025.00043 Across 4,335 pull requests, 1,568 received an automated review and 73.8 per cent of the automated comments were resolved, while the average closure time rose from five hours fifty-two minutes to eight hours twenty. Automation deployed and priced.

Check your understanding & apply it · 4 questions, 1 essay

1An engineering manager proposes to cut code review because defect comments are only about 14 per cent of review comments. Using the unit's account of what review yields, which diagnosis is supported?

2A review records mostly defects that do not affect visible functionality and leaves them in place. Using the unit's classification, what does the unit predict?

3A team wants evidence that review earns the attention it costs. Judging by the unit, which outcome should it treat as the one that justifies the time?

4A reviewer argues that the understanding the unit credits to review may be a by-product of a process justified on defects alone. Which diagnosis does the unit concede?

5Essay A team's engineering manager proposes to cut code review on the grounds that review finds few defects, that the test suite and static analysis catch bugs better, and that reviewers are a bottleneck. Write the note you would give them. Say what review actually yields, with numbers, and what those numbers are counts of; state what the outcome evidence supports and what it cannot separate; name the social factors that decide how long review takes and who gets heard; give one falsifier that would change your recommendation; and close with the outcome this unit cannot measure with the instruments it names. Name the source behind each step.

Start with what the yield actually is, because the manager's premise is half right. Bacchelli and Bird (2013) observed 17 developers across 16 Microsoft product teams, classified 570 comments in 200 threads and surveyed 165 managers and 873 programmers; defect comments were only the fourth most frequent of nine categories, at 78 comments or 14 per cent, while code improvements led with 165 comments, 29 per cent. That is a count of comments, not of escaped defects, and the same study says defect comments mainly cover small logical issues while knowledge transfer, team awareness and alternative solutions appear as additional outcomes, with understanding the key aspect of reviewing and only 12 comments explicitly about knowledge transfer. Second, concede how weak the counter-evidence is. Mantyla and Lassenius (2009) classified 388 and 371 defects from nine industrial C and C++ reviews and 23 student Java reviews and found that three quarters did not affect visible functionality at all; their largest group, evolvability defects, made code harder to understand and modify, which is precisely what no test detects. So the proposal would remove the only mechanism in the toolchain aimed at that class of problem. Third, put the outcome evidence in front of them with its limit named. McIntosh and others (2016) studied Qt, VTK and ITK and found review coverage, participation and reviewer expertise each share a significant link with post-release defects, and the design is observational, so it cannot separate the practice from the teams that sustain it; more importantly, it measures the process, so cutting coverage moves exactly the variable the study links to quality. Fourth, name the social mechanism rather than treating review as a queue. Thongtanunam and others (2015) found reviews with a code-reviewer assignment problem take 12 days longer to approve a change, and propose RevFinder, which recommends reviewers from similar reviewed file paths, so most of the reported bottleneck may be assignment rather than review. Bosu and Carver (2014) add that reputation decides as much as content: across eight open-source projects, core developers got earlier feedback and more acceptances while peripheral developers waited 2 to 19 times longer, between 12 and 96 hours, and Baysal and others (2016) found in WebKit and Google Blink that patch size, priority, the organisation behind the contributor and the reviewer's load and experience all significantly affect response time and outcome. Fifth, offer a falsifier and an honest limit. I would change the recommendation if comment-level data showed that code-improvement comments, not defect comments, predict the same post-release defects that a stronger test suite already catches, or if mandatory review could be turned off in one organisation and post-release defects did not rise while a validated measure of understanding held steady. What no instrument in this unit measures is the shared understanding itself: Rigby and Bird (2013) introduce a knowledge-sharing measure and they are the exception, and the automation work of Tufano and others (2021) targets the local comment, which is the part of review that is least in dispute.