Pre-preface: We have about 50 heavily linked builds that make up our “core” environment. We’re trying to cut "major" release times from ~3 months to ~3 weeks. Check out the prior blog post for more background information.
In a continuous integration/continuous deployment shop, if you decouple the deployments away from the builds you will improve tester productivity, decrease test environment outages throughout the day, find defects earlier, speed up the builds, you are encouraged to invest and rewarded by investing in automated tests and ultimately position yourself for continuous production deployments ala Push Engineering. (scary, but woo!) That’s a lot of claims to make, so let’s get on with it.
Preface - this isn’t a step away from the Continuous Integration or Continuous Deployment methodology, it’s a refinement to keep it from becoming Continuous Interruption or Continuous Regression.
I’m willing to bet that if you build on every commit, 95% of your test environment stakeholders (development, QA, product owners, etc) that are using the environment in question aren’t looking for that specific fix/feature/revision at that moment. They want it eventually – usually after they are done testing the story they picked up after they sent the first story back to development with a defect. While they want that fix/feature/etc, they want a Quality change; they want the commit to build successfully so the build isn’t dead on arrival, they want it to pass the unit/system test suite (hopefully you’re at 100% coverage) so there is a high confidence you aren’t regressing existing critical functionality, and they want their test environment to get that revision at an expected time – either scheduled or on demand. The last thing testers want is to have their valuable time spent setting up a test wasted when the environment is reset on them for a defect fix or change they could not care less about.
A moderately sized software development unit – say, two teams adding up to 10 developers with Quality Assurance, Business Analysts, Project Managers, Database Admins and Developer Operations people – can generate a lot of environmental churn on a daily basis. Your busiest environment will probably be your lowest environment; I frequently see 10 developers able to commit 20-30 revisions every day to the dev branches, and then those revisions ( once they pass whatever gateway test/merge qualifications necessary) being selectively merged to the next more-mature environment – generally it divides by 2 or 3 each environment. So your 30 revisions in Dev turns to 10 in the next environment up, then 5, then 2, then maybe you’re up to Prod and it’s hopefully just one build ;) If you clump these deployment cycles into “a” number – without knowing your business or processes, there is no magic number – but let’s assume for a moment we go to four deployments a day in non-production – 9AM-Noon-3PM-6PM, and only when there are changes to push – you’re now limited to 4 deployment interruptions in your lowest environment, to one or two in all other non-prod environments, and it’s on an expected timetable. From ~50 continuous interruption deployments a day down to ~8 – that’s a slight improvement.
If you have frequent “hot” builds that need to be slammed in as soon as possible, your people environment is probably lacking in software development process maturity. People need to learn the old fashioned way – experience – that there is never going to be a silver bullet fix that solves everything in one build & deploy . You can alleviate the need for “War-room ASAP” builds by working in as close to 100% test coverage percentage as you can afford, level-set the environment stakeholders by explaining how many people/groups are being impacted with a rebuild, communicate the end to end outage length, and put visibility on the rebuild & deployment, each and every one. When the world is watching, that super-important fix can wait 45 minutes until the scheduled, automated deploy happens. And if you’re in a dynamic place where you have multiple groups working on multiple stories simultaneously, it’s extremely rare for a critical group of people to be blocked & waiting for a deployment. They may say they are, but in reality they most likely have several other items ready to test, or items marked resolved that they need to sign off on. Don’t get me wrong – you will need to allow for manual deployment pushes – just make sure the button is pushable by the right stakeholders and there is clear visibility on the deployment occurring, why it’s occurring and who pushed the button/authorized it. Listen to the users too – don’t let a pushy manager terrorize everyone by smashing their builds in. You need to listen and push back if need be.
Defects & defect cost
If your deployments are coupled with your CI builds, even with stepped down commits as you go up in environments, that’s a lot of interruptions in each environment. The closer the environment is to Production makes the outages more expensive in the short-term, due to the perceived instability of the higher environments by senior leadership and external clients and the interruptions of client UAT, sales demos, etc. These outages/interruptions tend to chase the legitimate testers away, and because the lower environments (closest to Development) are being rebuilt most often, you’re training your most valuable testers to stay out and stay away from the environments you actually need them in the most – the sooner you find a defect, the cheaper it is to fix it. When the people who you are making these changes for finally tests a few weeks before release or in an environment right before Production, having the “you didn’t understand our requirements” card get played is somewhere on a level of fun between playing “Sorry” with your significant other (they show no mercy!) and having the Draw-Four card played on you in Uno, except in a large release hundreds of thousands of dollars in time & effort are at stake. That is the long-term cost of low-environment outage and over time will greatly outweigh the short-term cost of your close-to-production environments recycling during the day. Too often we let the “rough” development environments be wild-wild-west, not realizing it’s your most valuable environment to test in.
Speeding up the build
If you have any tests baked into the build, you know they can take just a few seconds to several minutes, depending where they sit between pre-canned, self-contained nunit tests or if you are setting up data and comitting transactions. When you have high percentage levels of test coverage your tests can often exceed the compile times of the build itself; there’s nothing wrong with that. This is where you can successfully argue for more build server horsepower. The time spent on deployments is often out of your hands; you are constrained by network speeds, SAN or local disk speeds, and hard-timed delays waiting for services to stop and restart. The deployment tasks in your build quickly add up; even if deployment tasks are only 10% of your build time, it’s increasing your environment outage by 10%. If you asked QA “Hey, would it be worth trimming 10% off a complete environment rebuild?” they would jump on it. Time is money; a 10% decrease in time is literally dollars – often it’s many dollars added up over dozens of builds a day – being saved. That’s something senior leadership can definitely understand and get behind.
Not slamming the build hot into the environment provides incentive to build up the automated test coverage percentage. Nobody wants a DOA build, and if they are now getting them 4 times a day max (let’s say) they want them to count. Assuming a 9AM/Noon/3PM/6PM deployment cycle, developers at 10AM might have an easier time making sure their change is covered in unit tests when they know the build won’t be deployed until Noon. I have seen project managers and directors literally standing behind a developer waiting for them to commit because “everyone is blocked!” or “The client is waiting to test!”. That’s not a conducive work environment, nor one anyone would want to work in. Taking the deployments out of the build also gives you the ability to put in longer, slower, deeper automated tests and post-build reports, like test coverage reports, in each and every build. It sounds counterintuitive to speed up the build by removing the deploy stage and then slow it down with more automated tests and post-compile reports, but those tests and reports will increase quality, which will decrease the amount of time spent in the whole “code it-throw it over the wall-have it thrown back-fix and throw it over the wall again” cycle.
Continuous Production Deployments
That’s where I am trying to get to, and that’s why I’m removing deployments from our build cycle. I need two major results out of this – our automated tests need to inc rease in quality & importance, and the decoupled deployment stage - broken out into a separate stage – can be perfected in the lower environments and then used in Production. The only reason I haven’t strung together our piecemeal deployment scripts from each build and run them against Production already is we don’t have good automated tests running (so we can’t automatically trust the build), and we don’t always have good visibility on what needs to be released due to how much shared code we use. Yanking deployments out and relying on and improving our test coverage and perfecting our deployment stage gives me half of this; my next post will be on how I’m going to solve the release build of materials question.