Code review is a team obligation, with limits worth naming

A change that nobody else has read is a change the team has accepted by default. In code review, someone other than the author examines a change before it enters the shared code. That interrupts the default, and the rule this unit defends is deliberately thin: no change reaches the mainline without a second reader. This is the sixth of ten units, and it follows the repository with the check the build cannot make. The evidence for review is real but narrower than its advocates claim: its documented value lies as much in spreading knowledge of the code as in finding defects, its limits include contexts a five-person team does not share, and the reviewer's own variability sets a ceiling on what any process can guarantee.

Review began as a heavy staged process and converged on a thin one

Fagan (1976) set out design and code inspections as a staged sequence of five operations, each with its own exit criteria and defined roles: an overview communicates the design, preparation educates the reader, the inspection finds errors, rework fixes them, and a follow-up confirms the fixes, with error types ranked so the team looks hardest for the kinds it makes most. He reported substantial gains in quality and productivity, with error rates falling steadily across successive projects. The honest limit is that this was the method's own author reporting results from his employer's projects, so the process and its measurement were not independent.

Rigby and Bird (2013) examined Android and Chromium OS at Google, Bing, Office and MS SQL at Microsoft and work at AMD. They contrasted these with a traditional inspection at Lucent and with open-source review on six projects. The parameters had converged: review happens early, before the code is committed, quickly and frequently, with about two reviewers. Sadowski and colleagues (2018) describe the same shape at Google, where review is near universal and a change reaches the mainline once a reviewer approves it. Their case study rests on twelve interviews, a survey of forty-four respondents and review logs for nine million changes. Convergence shows that a lightweight rule can hold; the scale of the Google case shows that it need not transfer.

The defect evidence sets the ceiling on what a second reader can guarantee

The strongest quantitative answer comes from individual reviews rather than projects. Kemerer and Paulk (2009) analysed two sets of personal software process data, of 371 and 246 programs. They found review rate a significant factor in defect removal effectiveness, after controlling for developer ability and initial quality. The recommended rate of 200 lines of code per hour or less identified nearly two-thirds of design defects and more than half of code defects.

The data also support the observation that review quality declines when the review rate exceeds the recommended maximum of 200 LOC/hour. (Kemerer and Paulk, 2009)

At project scale the evidence is correlational but consistent. McIntosh and colleagues (2014) studied the Qt, VTK and ITK projects. They found that review coverage and review participation each shared a significant link with quality. Low values were estimated to produce up to two and up to five additional post-release defects. The same team found reviewer expertise mattered too (McIntosh et al., 2015). Thongtanunam and colleagues (2016) examined 196,712 reviews across Android, Qt and OpenStack. Participation is patterned: a patch's past exposure to review predicts whether it attracts reviewers, and new features are likelier to wait for first feedback. A rule requiring a second reader is also a rule about where attention is spent.

Porter and colleagues (1998) found, in a controlled study of inspection, that team size and the number and sequencing of sessions did not significantly change defect detection. The reviewers, authors and code units explained much more of the variation than process structure did. Different readers find different problems; a process guarantees the reading, not the finding.

The durable return is knowledge, and the channel also carries judgements

The defect justification is weaker than the practice assumes. Bacchelli and Bird (2013) observed seventeen developers across sixteen Microsoft teams, classified 570 review comments and surveyed 165 managers and 873 programmers. Defect-related comments were the fourth most frequent of nine categories, about fourteen per cent, and concerned mainly small logical issues. Review also delivered knowledge transfer, team awareness and alternative solutions. Rigby and Bird (2013) put a number on that. Taking part in review increased the number of distinct files a developer knew about by between 66 and 150 per cent, depending on the project.

Review is also social. Bosu and colleagues (2017) surveyed open-source developers and developers at Microsoft. They found that they spend roughly ten to fifteen per cent of their time on review, and value it for knowledge sharing, community building and quality as well as defects. The quality of what an author submits shapes the impression reviewers form of a teammate. The channel that spreads knowledge also carries judgements about people.

Three limits keep the rule thin

The first limit is context. Baum and colleagues (2016) interviewed professionals from nineteen companies. They found wide variation, with a large fraction performing almost no review and process details traced to context and the effects a team wanted.

The second limit is the performance mode. Review is also where impressions form: usefulness turns on linguistic features such as comprehensibility and politeness (Turzo and Bosu, 2023), so an author in a second language is judged on more than the change. An author under time pressure may seek a fast approval rather than a careful reading, and a reviewer may give one. The third limit is the reader: because reviewers differ more than processes do (Porter et al., 1998), a heavier process cannot buy detection it has not earned.

Contested question: a quality gate or a knowledge pump, and how thin can the rule be?

One position holds that review is a quality gate whose value is measured in defects prevented. Its evidence is Fagan (1976), the review-rate effect on defect removal (Kemerer and Paulk, 2009), and the link between coverage, participation and reviewer expertise and post-release defects (McIntosh et al., 2014, 2015). On that view the rule should specify coverage and a rate, and a thin rule is a licence to skim.

The other holds that the defect yield is smaller than the justification assumes, and that the durable returns are in knowledge and coordination. Its evidence is Bacchelli and Bird (2013) for the low share of defect comments, Rigby and Bird (2013) for the measured spread of files known, and Bosu and colleagues (2017) for the non-defect benefits. On that view a thin rule is enough and a heavier one spends the team's scarcest resource on the weakest of review's outputs.

What would settle it is a controlled comparison holding team, task and deadline constant and varying only the review rule: whether a change needs a second reader, whether a rate or coverage target is set, and whether reading is separate from conversation. Defect escape and the spread of knowledge are then measured against criteria fixed in advance. Nobody has run it at student-team scale. The evidence is controlled inspection data (Fagan, 1976; Kemerer and Paulk, 2009), correlational project studies (McIntosh et al., 2014, 2015; Thongtanunam et al., 2016), and one firm's logs beside interviews and surveys (Sadowski et al., 2018; Baum et al., 2016; Bosu et al., 2017). The controlled data favour the gate, the naturalistic data the pump. While the dispute is open, keep the rule thin and require the second reader to say in writing what was checked and what was found. The strongest objection is that a rule can be satisfied by an approval, and no case shows a required reader catching what a voluntary one missed.

Where the argument is heading

The direction of travel is from the human process towards the tooling around it. Wessel and colleagues (2022) used a regression discontinuity design on 1,194 GitHub projects and interviewed twelve practitioners. They found that adopting a review bot increased monthly merged pull requests, decreased non-merged ones and reduced communication among developers: automation changed the social process, not only the effort. Turzo and Bosu (2023) asked what makes a review worth having, classifying 2,500 comments from the OpenDev Nova project and finding usefulness tied to technical content and to linguistic features such as comprehensibility and politeness.

The newest turn is the model as a second reader. Adalsteinsson and colleagues (2025) combined a field study with a field experiment on two prototypes of an LLM-assisted review tool at one company. Reviewers valued an AI-led review most on large or unfamiliar changes, and named false positives and trust as the standing worries. That does not show a model catching what a person misses. It moves the reviewer's judgement to when to trust the model.

For a student team the reading is stable. Keep the thin rule, and expect to be judged on whether a second reader caught something the author could not see.

The sources this unit rests on

• Fagan, M. E. (1976). Design and code inspections to reduce errors in program development. IBM Systems Journal, 15(3), 182-211. doi:10.1147/sj.153.0182 Sets out formal inspection as staged operations with defined exit criteria and roles, reporting substantial gains in quality and productivity with falling error rates. It is the origin of the heavy process modern review reacts against.
• Porter, A., Siy, H., Mockus, A. and Votta, L. (1998). Understanding the sources of variation in software inspections. ACM Transactions on Software Engineering and Methodology, 7(1). doi:10.1145/268411.268421 Models a controlled inspection experiment and finds team size and session sequencing did not significantly change defect detection, while the reviewers and code units explained more variation than process structure. Different readers find different problems.
• Rigby, P. C. and Bird, C. (2013). Convergent contemporary software peer review practices. ESEC/FSE 2013. doi:10.1145/2491411.2491444 Examines two Google-led projects, three Microsoft projects and AMD work against a traditional inspection and six open-source projects, finding review converged on lightweight, early, frequent practice with about two reviewers. Its knowledge-sharing measure shows review increases the files a developer knows by 66 to 150 per cent.
• Bacchelli, A. and Bird, C. (2013). Expectations, outcomes, and challenges of modern code review. ICSE 2013, 712-721. doi:10.1109/ICSE.2013.6606617 Observes seventeen developers across sixteen Microsoft teams, classifies 570 review comments and surveys 165 managers and 873 programmers. Defect comments were the fourth most frequent category, about fourteen per cent, while review also delivered knowledge transfer.
• Baum, T., Liskin, O., Niklas, K. and Schneider, K. (2016). Factors influencing code review processes in industry. FSE 2016. doi:10.1145/2950290.2950323 Interviews professionals from nineteen companies and finds a large fraction performing almost no review, wide variation in process detail, and culture and social factors central to adoption. Process variations trace to context and to the effects a team wants.
• Bosu, A., Carver, J. C., Bird, C., Orbeck, J. and Chockley, C. (2017). Process aspects and social dynamics of contemporary code review. IEEE Transactions on Software Engineering, 43(1), 56-75. doi:10.1109/TSE.2016.2576451 Surveys open-source and Microsoft developers and finds they spend roughly ten to fifteen per cent of their time on review and value it for knowledge sharing and quality as well as defects. Submission quality also shapes reviewers' impressions of teammates.
• Kemerer, C. F. and Paulk, M. C. (2009). The impact of design and code reviews on software quality: an empirical study based on PSP data. IEEE Transactions on Software Engineering, 35(4), 534-550. doi:10.1109/TSE.2009.27 Analyses two personal software process data sets, of 371 and 246 programs, and finds review rate a significant factor in defect removal effectiveness after ability and initial quality were controlled. The recommended rate of 200 lines of code per hour found nearly two-thirds of design defects.
• McIntosh, S., Kamei, Y., Adams, B. and Hassan, A. E. (2014). The impact of code review coverage and code review participation on software quality. MSR 2014, 192-201. doi:10.1145/2597073.2597076 Studies the Qt, VTK and ITK projects and finds review coverage and participation each share a significant link with quality, with low values estimated to produce up to two and up to five additional post-release defects per component.
• McIntosh, S., Kamei, Y., Adams, B. and Hassan, A. E. (2015). An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering, 21, 2146-2189. doi:10.1007/s10664-015-9381-9 Extends the earlier study on the same projects, finding coverage, participation and reviewer expertise each significantly linked to fewer post-release defects. It tests the review-quality link rather than assuming it.
• Thongtanunam, P., McIntosh, S., Hassan, A. E. and Iida, H. (2016). Review participation in modern code review. Empirical Software Engineering, 22, 768-817. doi:10.1007/s10664-016-9452-6 Examines 196,712 reviews across Android, Qt and OpenStack and finds past participation predicts which patches attract reviewers, and that new features are likelier to wait for first feedback. Review attention is patterned, not evenly spread.
• Sadowski, C., Soderberg, E., Church, L., Sipko, M. and Bacchelli, A. (2018). Modern code review: a case study at Google. ICSE-SEIP 2018. doi:10.1145/3183519.3183525 Investigates review at Google through twelve interviews, a survey of forty-four respondents and review logs for nine million changes, finding review near universal and lightweight, with a change merged once a reviewer approves it.
• Turzo, A. K. and Bosu, A. (2023). What makes a code review useful to OpenDev developers? An empirical investigation. Empirical Software Engineering, 29, article 6. doi:10.1007/s10664-023-10411-x Classifies 2,500 comments from the OpenDev Nova project and models developers' judgements, finding usefulness depends on technical content and on linguistic features such as comprehensibility and politeness, while frequent reviews between the same two people lower it.
• Wessel, M., Serebrenik, A., Wiese, I., Steinmacher, I. and Gerosa, M. A. (2022). Quality gatekeepers: investigating the effects of code review bots on pull request activities. Empirical Software Engineering, 27, article 108. doi:10.1007/s10664-022-10130-9 Uses a regression discontinuity design on 1,194 GitHub projects and twelve practitioner interviews to measure what changes after a review bot is adopted. Bot adoption increased merged pull requests and reduced communication among developers.
• Adalsteinsson, F. S., Magnusson, B. B., Milicevic, M., Davidsson, A. N. and Cheng, C.-H. (2025). Rethinking code review workflows with LLM assistance: an empirical study. ESEM 2025. doi:10.48550/arXiv.2505.16339 Combines a field study of review practice with a field experiment on two LLM-assisted review prototypes at one company. Developers preferred an AI-led review, most on large or unfamiliar changes. It adds the model as a second reader; its limit is one company and self-reported preference.