Integration, delivery and release: deployment is not the same as release

The seventh unit is the third of those four, and follows the change to the user. The plan's thesis has two halves, and the literature supports them unequally. The first is that separating deployment from release turns a user-facing decision into an engineering one. Once the mainline is release-ready, when a feature becomes visible is a configuration question rather than a rebuild. The second is that the separation is not free. It manufactures a configuration surface, the pipeline and its flags, that acquires faults and accumulates debt. No study here shows that surface to be pruned.

Integration is a contract, and the build enforces it

Fowler (2006) states the practice in a sentence. Everyone merges their changes into a codebase with their colleagues' changes at least daily, and an automated build that includes tests verifies each integration. The build must be self-testing. The team must fix a broken build immediately, because a red build says the shared claim about the product is false.

The largest empirical account is Hilton and others (2016). They analysed 34,544 open-source GitHub projects, 1,529,291 builds from the most commonly used continuous integration system, and 442 developers. They found evidence that continuous integration helps projects release more often, and that adoption by the most popular projects keeps growing. Hilton and others (2017) show that it is not a single decision. From interviews across industries and scales, triangulated by two surveys, they found three trade-offs. The first is between speed and certainty. The second is between access to the build and its security. The third is between configuration options and ease of use.

Vasilescu and others (2015) used historical data on process metrics and outcomes across GitHub projects. They found that continuous integration improves team productivity, since teams integrate more outside contributions, without observable diminishment in code quality.

Deployment and release come apart

Humble and Farley (2010) supply the distinction and the machinery that makes it usable. They describe the deployment pipeline, an automated path from check-in to a release-ready build that gives feedback on every change. Deployment installs a version into an environment, while release exposes that version, or a feature inside it, to users. The user whose screen reader, older handset or metered connection cannot follow the release sits outside that separation. Fowler (2006) draws the consequence. With a release-ready mainline the decision to release into production becomes purely a business decision, a matter of minutes running an automated pipeline.

Facebook's web front-end is the largest published case of perpetual deployment. The team worked in a perpetual development mode, deploying new work once it was ready. It relied on peer review, extensive automated testing and a culture of personal responsibility rather than on a release date (Feitelson, Frachtenberg and Beck, 2013).

Continuous deployment runs at a different scale, and its evidence is two firms

Savor and others (2016) define continuous deployment as many small incremental updates leading to a continuous stream of tens, hundreds or even thousands of deployments per day. Their case studies sit at Facebook and OANDA. It is a practice report describing two firms rather than setting a target. Chen (2015) is the corrective. It reports two years of continuous delivery at one organisation. The benefits sit in release frequency and predictability, and the real challenges sit in testing, culture and configuration management.

What the pipeline buys, and how thin the evidence is

Two syntheses describe the state of the field and neither flatters it. Shahin, Babar and Zhu (2017) reviewed papers on continuous integration, delivery and deployment published between 2004 and June 2016. They selected 69 and catalogued 30 approaches and tools plus the factors for adoption. Most were of the validation (34.7 per cent) and evaluation (36.2 per cent) research types, which are small-scale designs. Rodriguez and others (2017) mapped deployment across 50 primary studies published between 2001 and 2014 and concluded the topic is still in its infancy.

The flag surface is a second product, with its own faults

Fowler (2006) explains why flags accumulate. A team integrates before a feature is ready to be seen, so the unfinished code lies latent in the released executable, and a flag keeps it active in testing and disabled in production. Remove the flag logic promptly once the feature is fully released. Rahman and others (2016) measured what happens next: across 39 Google Chrome releases spanning five years, and from Google's toggle maintenance spreadsheet, toggles reconciled rapid releases with long-term development but introduced technical debt and additional maintenance.

The pipeline itself carries faults, and its workload depends on what it builds

Zampetti and others (2020) widen the account to the pipeline itself. From 13 expert interviews and more than 2,300 Stack Overflow posts they compiled a catalogue of 79 continuous integration bad smells in 7 categories. They also surveyed 26 developers on how important those smells are. Some results confirm existing knowledge, such as poor use of branches. Some are new, such as the abuse of shell scripts. At least one contradicts earlier advice, since avoiding nightly builds turns out to be poor practice. The machinery that makes deployment routine is a source of faults, and the faults are catalogued rather than imagined.

Bernardo, Da Costa, Medeiros and Kulesza (2024) compared 93 machine learning projects with 92 others on GitHub. Machine learning projects took longer to build, their medium-sized projects carried less test coverage, and their small and medium-sized projects more often showed build durations rising over time. The configuration surface that makes deployment routine therefore carries a different workload, and a different fault profile, for a model than for an application.

Contested question: does the separation reduce risk or relocate it?

The disagreement is not about whether deployment and release can be separated, which is settled. It is about what the separation buys. One position, held by Humble and Farley (2010), Fowler (2006), Savor and others (2016) and Chen (2015), is that the separation is an engineering win. Deployment becomes routine, release becomes a business decision, and cadence rises without a matching rise in risk. That evidence is strong on capability, since it shows the machinery working at scale. It is weak on comparison, since none of it measures a team against an alternative practice. A second position holds that the separation relocates risk rather than removing it. Rodriguez and others (2017) find the evidence base in its infancy. Shahin and others (2017) find it dominated by validation and evaluation studies. Zampetti and others (2020) catalogue 79 bad practices inside the pipeline, and Rahman and others (2016) document the debt that toggles bring. On this reading the separation converts one risk, an unready feature reaching users, into another, a configuration surface nobody owns.

The two positions also disagree about mechanism. Vasilescu and others (2015) associate continuous integration with productivity and no observable quality loss, which reads as cause; comparing adopters with non-adopters cannot separate the practice from the disposition of the teams that adopt it. One design would settle the question, and neither side has produced it: take the adoption dates of many projects as an interrupted time series with matched controls, measure release frequency and pipeline faults before and after adoption, add a longitudinal measure of flag removal, and pair the bad-smell catalogue with defect data.

What to do while the dispute is open

The evidence is strong on capability and weak on comparison. The machinery works at scale (Hilton and others, 2016), and none of the accounts measures a team against an alternative practice. The strongest objection to this unit's thesis is that the separation relocates risk rather than removing it, converting an unready feature into a configuration surface that nobody owns (Zampetti and others, 2020; Rahman and others, 2016). While the dispute is open, a designer should treat the flags and the pipeline as a product with an owner and a defect count, and schedule each flag's removal before the next release rather than after it.

Where the argument is heading

The direction is from advocating the pipeline to measuring it. Rodriguez and others (2017) make the infancy of the evidence the finding rather than the excuse. Shahin and others (2017) name the research-type gap that produces it. Hilton and others (2017) turn adoption into trade-offs that can be studied one at a time. Zampetti and others (2020) close their catalogue with an agenda that includes automated detectors for bad smells, and teaching continuous integration culture rather than tools. Rahman and others (2016) leave the toggle maintenance question open.

The sources this unit rests on

• Fowler, M. (2006). Continuous Integration. martinfowler.com. https://martinfowler.com/articles/continuousIntegration.html The article states the practice of integrating at least daily against a mainline verified by a self-testing automated build. It argues that a release-ready mainline makes release purely a business decision, and it instructs teams to remove flag logic promptly. It is the source for both halves of the thesis; the live page carries a 2024 rewrite that retains the practice statement used here.
• Humble, J. and Farley, D. (2010). Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley, Boston, MA. https://www.informit.com/store/continuous-delivery-reliable-software-releases-through-9780321670250 The book introduces the deployment pipeline as an automated path from check-in to a release-ready build. It separates deployment, which installs a version into an environment, from release, which exposes it to users, and is the origin of the distinction the thesis depends on.
• Hilton, M., Tunnell, T., Huang, K., Marinov, D. and Dig, D. (2016). Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016). ACM, New York, 426-437. doi:10.1145/2970276.2970358 The study analysed 34,544 GitHub projects, 1,529,291 builds and 442 developers. It found evidence that continuous integration helps projects release more often and that adoption keeps growing.
• Hilton, M., Nelson, N., Tunnell, T., Marinov, D. and Dig, D. (2017). Trade-offs in Continuous Integration: Assurance, Security, and Flexibility. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (FSE 2017). ACM, New York, 197-207. doi:10.1145/3106237.3106270 Interviews across industries and scales, triangulated by two surveys, identified trade-offs between speed and certainty, access and security, and configuration options and ease of use, reframing adoption as a set of bargains rather than a switch.
• Vasilescu, B., Yu, Y., Wang, H., Devanbu, P. and Filkov, V. (2015). Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (FSE 2015). ACM, New York, 805-816. doi:10.1145/2786805.2786850 Large historical data across GitHub projects showed that continuous integration improves team productivity, allowing more outside contributions, without observable diminishment in code quality; the design is observational, so it cannot separate the practice from the teams that adopt it.
• Savor, T., Douglas, M., Gentili, M., Williams, L., Beck, K. and Stumm, M. (2016). Continuous Deployment at Facebook and OANDA. In Proceedings of the 38th International Conference on Software Engineering Companion (ICSE-C 2016). ACM, New York, 21-30. doi:10.1145/2889160.2889223 The authors define continuous deployment as many small incremental updates leading to tens, hundreds or even thousands of deployments per day, with case studies at Facebook and OANDA, a practice report rather than a comparison.
• Chen, L. (2015). Continuous Delivery: Huge Benefits, but Challenges Too. IEEE Software 32(2), 50-54. doi:10.1109/MS.2015.27 A case study of one organisation's two years implementing continuous delivery records substantial benefits in release frequency and predictability alongside real challenges in testing, culture and configuration management, a corrective to a gains-only reading of the adoption accounts.
• Shahin, M., Babar, M. A. and Zhu, L. (2017). Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices. IEEE Access 5, 3909-3943. doi:10.1109/ACCESS.2017.2685629 The review selected 69 peer-reviewed papers published between 2004 and June 2016 and catalogued 30 approaches and tools plus the critical factors for adoption; the majority were of the validation (34.7 per cent) and evaluation (36.2 per cent) research types, which shows how small-scale the evidence is.
• Rodriguez, P., Haghighatkhah, A., Lwakatare, L. E., Teppola, S., Suomalainen, T., Eskeli, J. et al. (2017). Continuous deployment of software intensive products and services: A systematic mapping study. Journal of Systems and Software 123, 263-291. doi:10.1016/j.jss.2015.12.015 The mapping covers 50 primary studies published between 2001 and 2014 and extracts ten recurrent themes of benefits and challenges; its conclusion that the topic is still in its infancy is the claim the unit uses to hold the practice reports at arm's length.
• Feitelson, D. G., Frachtenberg, E. and Beck, K. L. (2013). Development and Deployment at Facebook. IEEE Internet Computing 17(4), 8-17. doi:10.1109/MIC.2013.25 An account by participants of Facebook's web front-end, describing a perpetual development mode in which new work is deployed once ready, supported by peer review, extensive automated testing and a culture of personal responsibility; it reports the practice in operation rather than its effect against an alternative.
• Rahman, A., Querel, L., Rigby, P. C. and Adams, B. (2016). Feature Toggles: Practitioner Practices and a Case Study. In Proceedings of the 13th International Conference on Mining Software Repositories (MSR 2016). ACM, New York, 201-211. doi:10.1145/2901739.2901745 The study analysed toggle usage across 39 Google Chrome releases spanning five years, mined the spreadsheet developers used to maintain toggles and validated the findings with four developers. Toggles reconcile rapid release with long-term development but introduce technical debt and additional maintenance.
• Zampetti, F., Vassallo, C., Panichella, S., Canfora, G. and Gall, H. C. (2020). An empirical characterization of bad practices in continuous integration. Empirical Software Engineering 25(2), 1095-1135. doi:10.1007/s10664-019-09785-8 From 13 expert interviews and more than 2,300 Stack Overflow posts the authors compiled a catalogue of 79 continuous integration bad smells in 7 categories and surveyed 26 developers on their importance. The catalogue includes faults the pipeline itself introduces.
• Bernardo, J. H., Da Costa, D. A., Medeiros, S. Q. and Kulesza, U. (2024). How do Machine Learning Projects use Continuous Integration Practices? An Empirical Study on GitHub Actions. In Proceedings of the 21st International Conference on Mining Software Repositories (MSR 2024). New York: ACM, 665-676. doi:10.1145/3643991.3644915 Comparing 93 machine learning projects with 92 other projects on GitHub, the study found machine learning projects took longer to build, their medium-sized projects carried less test coverage, and their smaller projects more often showed build durations rising over time. It adds a new fault profile to the pipeline.