Friday, December 21, 2012

Duplicate Notification Emails In Jira

Once the dust settles from a new Jira install and you have users active in it on a daily basis, the little things start to get noticed. A frequent complaint is how Jira seems to spam users with email notifications. If you google around for "why am I getting duplicate jira notifications" you find alot of results, such as:
https://answers.atlassian.com/questions/69131/duplicate-notification-emails-from-jira
https://confluence.atlassian.com/display/JIRAKB/Notifications+Troubleshooting
https://answers.atlassian.com/questions/44064/how-to-prevent-duplicated-outgoing-mail-on-notification to name a few of the best search returns.

Atlassian even put a wizard into Jira for admins to figure out why a specific user got a duplicate email on a specific issue and action. However, the wizard doesn't work for notifications that aren't about a specific action on a specific issue, for instance when someone uses the "Contact Administrators" form. Luckfully, duplicate emails are almost overwhelmingly not caused by bugs in Jira, but the way your user directory is setup and the way your notification schemes are setup.  Doubly luckfully, here's the two rules that will help troubleshoot most duplicate email issues in my experience.

1) Jira, when it sends emails, only checks that the username is unique, not the email address.
2) Certain events get forwarded to everyone with the Jira Administrators global permission.

A messy user directory will make what looks like duplicate emails because of rule #1; if you have an internal directory with jsmith@company.com as a username and you then connect Jira to active directory and importes domain\jsmith as a user, Jira will treat jsmith@company.com and jsmith as two separate users (because they are!) even though they have the same smtp email address. If they are both in the same group that has a project role that gets notifications, it will appear that Jira is sending out duplicates when it's actually not.

For rule number two, let’s take the ContactAdministrators form for instance; it’s one of those system events that emails everyone with the Jira Administrator’s global permission.  If you check your Global Permissions, you can see which user groups have the Jira Administrators permission. Then check the email addresses for everyone in every group with the Jira Administrator global permission; you will almost certainly find either duplicate email addresses (remember, Jira checks usernames for duplicity when it sends emails - not email addresses!) or unique email addresses that have the same destination (for instance, jsmith@company.com and john.smith@company.com may both be valid and go to the same person, depending how your email scheme was setup), or unique email addresses that result in a distribution group; perhaps you're jsmith@company.com and you're a member of jira-admins@company.com. Jira will email both and not feel bad in the slightest.

Long term, you need to:
1) Be aware of Jira's behavior so you can be more tolerant of what looks like duplicate emails/spam.
2) Be very wary of internal directory accounts using the smtp email address of a distribution group. Jira will not parse the contents of a distribution group - you literally can't over SMTP.

-Kelly Schoenhofen

Thursday, December 13, 2012

Linux App + ports below 1024 + not root = permission denied


Issue: you set your application on a Linux server (say Jira, Fisheye, Stash, Bamboo, etc) to port 80 and/or 443, and you get "permission denied" right away during startup. 

This has been a growing issue for the last couple years - it appears that sometime around the Linux 3.0 kernel release, they made the decision to deny port binding/usage below 1024 to applications not running as root. Example: a Tomcat based application can't use port 80 with running as root. I get some of their reasoning - it's harder to run a rogue website on port 80 on a box you've slightly compromised and steal people's data/identity/etc, if you don't have access to port 80 unless you're a super-privileged user. But it sure makes things difficult when your user base wants to type in http://application.company.com internally and it doesn't come up. 

Keeping an application on its default port (let's say port 1234 for simplicity's sake) has alot of advantages, for instance:

  • Future upgrades are a breeze, because you don't have to "fix" configuration files the upgrade overwrites. In fact, some upgrades detect your changes and if the migration of your changes fail, the upgrade can get very messy. (i.e. new features that need to be part of your server.xml file that you modified to use port 80 aren't written during the upgrade because it's too freaked out by your small change). 
  • Documentation & QA from the vendor is all based around a clean, out of the box install. When you start messing with ports, you start playing with fire. 
  • Staying with default ports means you can run your application as a non-root user; it's a really good idea in the first place, and as more corporations embrace Suse/Ubuntu/etc as viable application servers, security standards are being set & enforced, and not having access to root/sudo is becoming more common. 
  • Port proxying and reverse-proxies aren't always a panacea, because they kill gzip compression of the web stream and while they do their best, they are an imperfect software solution; you are going to take an increasingly performance hit the more web traffic you have to deconstruct, reconstruct and forward, and the whole deconstruct/reconstruct process isn't always transparent to the client or server ends; weird behavior or bugs can result. And if you call for support, the first thing they will have you do is bypass your proxy and reproduce the issue. And if the issue is from the proxying, they won't support you. 

But sometimes you have users (or management) that really want to type in http://app.company.com and have it work. 

What I've done in the past is use apache2 to setup a simple port redirection (not proxying) so users can type in http://app.company.com and it redirects them seamlessly to http://app.company.com:1234. 

Sample setup using a clean Ubuntu 12.04 LTS server, with an existing application listening for http requests on port 1234 and https requests on 1443:

1) Install apache2 (as in sudo apt-get install apache2)
2) Enable ssl - sudo a2enmod ssl
3) Edit the now-existing /etc/apache2/httpd.conf - 


NameVirtualHost *:80

<VirtualHost *:80>
   ServerName app.company.com
   DocumentRoot /home/appuser
   Redirect permanent / http://app.company.com:1234
</VirtualHost>

<VirtualHost *:443>
   ServerName app.company.com
   DocumentRoot /home/appuser
   Redirect permanent / https://app.company.com:1443
   SSLEngine on
   SSLCertificateFile /home/appuser/app.crt
   SSLCertificateKeyFile /home/appuser/app.key
</VirtualHost>


This works fairly well - it leaves the original ports intact for direct calls (i.e Jira needing to integrate with Stash or Fisheye - it will not work via a redirect), and users will automatically use the new port when they bookmark pages.  
---
However, let's say you need to run on port 80 & 443. Authbind is probably your best bet, I've never had success with setcap. I'll use Jira under Tomcat as an example, but you can adapt this to anything. 

The following steps assumes a nice clean Ubuntu 12.04 LTS server, a dedicated jira user and an existing install of Jira 5.2.

1) If you have anything using port 80/443 already, you'll have to remove it. In my prior example I used apache2 to do some redirects for user browsers; you might as well uninstall apache2 completely now. (i.e. sudo apt-get --purge remove apache2.2-common)

2) Install authbind (i.e. sudo apt-get install authbind)

3) Bind your ports to your dedicated Jira user with touch, chown and chmod. Here's port 80:

sudo touch /etc/authbind/byport/80
sudo chown jira /etc/authbind/byport/80
sudo chmod 755 /etc/authbind/byport/80

Repeat as needed for other ports like 443.

4) Disable ipv6 in Jira/Tomcat - this was not clear to me for quite while; I glossed over that authbind does not currently support IPv6, and I didn't realize ipv6 was the primary protocol in Tomcat when it's enabled on your nic. (i.e. add CATALINA_OPTS="-Djava.net.preferIPv4Stack=true" in the setenv.sh in your jira/bin folder). You cannot skip this step! Use ifconfig to see if Ipv6 is enabled; if it is, you have to disable it in Tomcat.

5) Don't forget to change your /jira/conf/server.xml to port 80 and/or port 443 instead of 8080 & 8443 respectively. To use Jira/Tomcat's basic http to https redirection, make sure your 80 block contains redirectPort="443" in it. If your users use http against anything but the exact url your SSL cert is for, they will get an ssl warning about a certificate mismatch. 

6) Reconfigure Jira to have authbind launch it with the --deep option so child spawns are covered. 

The simplest spot to set this is at your /etc/init.d/jira script, but if someone stops and starts Jira manually in the /jira/bin folder, then you will get permission denied on ports below 1024 and Jira won't be accessible. I got a little messier and edited the start-jira.sh script, so the /etc/init.d/jira script is covered (because it execs start-jira.sh) and other people who jump on your server and restart Jira don't have to know to start it with authbind --deep. Change your start-jira.sh to the following (my additions on line 4 & 6 - specifically the "authbind --deep" portion): 

if [ -z "$JIRA_USER" ] || [ $(id -un) == "$JIRA_USER" ]; then
    echo executing as current user and authbind
    if [ "$PRGRUNMODE" == "true" ] ; then
        exec authbind --deep $PRGDIR/catalina.sh run $@
    else
        exec authbind --deep $PRGDIR/startup.sh $@
    fi

Bonus 7) If you want to get extra fancy, so users get redirected from whatever http address they try your Jira instance with (say http://jira when it should be http://jira.company.com) to the correct, FQDN using https (so no ssl cert mismatch warnings), add the following rule to the /jira/WEB-INF/urlrewrite.xml instead of redirectPort="443" in the server.xml:

<rule>
   <name>Http Check</name>
   <condition type="scheme" operator="equal">^http$</condition>
   <from>^/(.*)</from>
   <to type="permanent-redirect" last="true">https://jira.company.com/$1</to>
</rule>

and then remove the redirectPort="443" from your server.xml. 

For brevity, I glossed over alot of bits that hopefully are trivial. Hopefully this helps other people get Jira (or any other product) working on ports below 1024, and not run with sudo/root. 

-Kelly Schoenhofen

Monday, November 19, 2012

Stash - Case Sensitive Collation Detection at the Database Level vs Server Level


Using MS-SQL 2008 & Stash 1.31 and the database migration wizard (after a clean install using the embedded hsql db), a new DB for stash set to SQL_Latin1_General_CP1_CI_AS for collation (CI = case insensitive) made the database migration wizard spit out the following error when I clicked "Test":

Stash requires a case sensitive database, but the target database is case insensitive. For information about configuring databases for use with Stash, please refer to the Stash documentation.

I set the database to SQL_Latin1_General_CP1_CS_AS, yet Stash's DB migration wizard continues to insist it's still case insensitive. I restarted Stash, it still says the DB is case insensitive. I took the DB offline and then back online, and Stash still says it's case insensitive.

I put some dummy chars in the name of the database, and Stash's DB migration wizard still says the "the target database is case insensitive". So I'm thinking it's not querying the database, it's querying the server default. And since this is a multi-tenant DB server, the owners prefer it to be SQL_Latin1_General_CP1_CI_AS by default. Regardless, this server also hosts a Jira database, which has to be set case insensitive.

Your products should work together a little better, Atlassian.

I can't go forward with this tool until I have it using a real database. After a couple hours of mucking around, I'm answering my own question, but it's not an optimal answer.

Stash apparently stores its external database configuration in a stash-config.properties file, in the root of the stash_home folder. When you install Stash using HSQL, it doesn't create the file. The absence of the file in fact prompts Stash to use its embedded HSQL DB.

Atlassian documentation (https://confluence.atlassian.com/display/STASH/Stash+config+properties) gives an example for a Postgres file - I looked at a Fisheye config file that uses MS-SQL and cobbled together one for Stash that also uses Windows authentication, like thus:

jdbc.url=jdbc:jtds:sqlserver://servername:1433;databaseName=database;domain=company.com;
jdbc.user=jirauser
jdbc.password=password
jdbc.driver=net.sourceforge.jtds.jdbc.Driver


I saved the file, restarted Stash and the web UI was back on the initial setup questions, this time populating a MS-SQL DB instead of HSQL.

So I'm "working", but there still seems to be a bug here. The migration wizard appears to be querying the server's collation, not the database's collation. Blindly crafting a config file to bypass faulty logic != acceptable workaround.

I posted this question & eventual answer on the Atlassian Answers forum, and Atlassian support down-voted my workaround; in at least v1.31, Stash's case sensitivity check is bypassed once the stash-config.properties file is present & valid, so I'm playing with fire here ;)

I'll mark it as answered. Someone else can create a defect ;)

-Kelly

Thursday, November 1, 2012

Jira Email Listener Failure?


Scenario: Either out of the blue or after an upgrade to either your Jira server or Jira itself, emails to Jira - to append comments onto an existing issue for instance - are seemingly ignored by Jira with no obvious errors.


In Jira, go to Administration (assuming you're an admin), Incoming Mail, "Edit" the Jira incoming mail listener you have setup and click "Test". If you get the following exception the rest of this article is for you:
SunCertPathBuilderException: unable to find valid certification path to requested target

Solution: The exception SunCertPathBuilderException: unable to find valid certification path to requested target means you're probably using SECURE_IMAP as your mail protocol. SECURE_IMAP makes a secure connection to your email server and needs the x509 (essentially SSL) cert the email server was secured with to be trusted by the account/process running Jira. Most companies run Jira internally and secure their (internal) mail server with a self-signed cert, so the cert has to be manually trusted for it to work (unlike a VeriSign cert, for instance).

The error message (SunCertPathBuilderException: unable to find valid certification path to requested target) simply means that the Jira process can't find a trusted public cert in a keystore with the same name as your email server.

So - either:
A) The email server name changed or
B) For some reason the cert is no longer in "the" keystore.

Solution to A) - the email server name changed or they changed which email server you can use -
1) Either install openssl on the server running Jira or on your local workstation, and use it to retrieve the public cert from your mail server:

openssl s_client -connect [email.servername.com]:imaps

It should open up a connection and you should see the server's public certificate in plaintext right in front of you - grab everything from Begin to End like thus:

-----BEGIN CERTIFICATE-----
MIIFjjCCBHagAwIBAgIKVggUuwAAAACkATANBgkqhkiG9w0BAQUFADBIMRMwEQYK
[snip]
3xUBlGXakDBkbL1euN3AR2yAZcXi2LqFrAz0mWKvR3icew==
-----END CERTIFICATE-----

and save it to a text file, call it "imap_cert.cer". 

2) Import it!
keytool -importcert -keystore /new_jre_location/lib/security/cacerts -file imap_cert.cer  -alias [email.servername.com]

Close to real world example:
jira@jira_server:/opt/atlassian/jira/jre/bin$ keytool -importcert -keystore /opt/atlassian/jira/jre/lib/security/cacerts -file ~/imap_cert.cer -alias mail.company.com

When you're prompted to trust it, type "yes" and enter. 
Restart Jira for the change to take effect. 


Solution to B)
In my case, the cert failure to be found was because I cleaned up an install of Jira and put it on the recommended straight-and-narrow path from Atlassian, which includes using the JRE bundled with Jira (at least for version 5.1.x for Linux) vs a Java runtime you installed manually or as a package.
It turns out that by default Jira uses the "cacerts" keystore in its Java JRE home, not the user keystore; so changing the JRE location will break the imap cert installation path (which is what happened to me).

Two obvious options:
1) Re-grab the x509 cert from the imap mail server (see Solution A)
2) If you can't use openssl and you have the JRE you were using before still, you can export the public cert from the old keystore and import it to the new keystore. That's what I did, and it worked out very well.

Three steps:
1) Get the name of your cert - this doublechecks you actually have the prior cert and it's named correctly.
keytool -list -keystore /old_jre_location/lib/security/cacerts
This dumps the summary contents of your old keystore; get the exact name of the cert alias so you can export it - normally this is the dns name of your email server (what I refer to as )
2) Export it!
keytool -exportcert -keystore /old_jre_location/lib/security/cacerts -file imap_cert.cer -alias [email.servername.com] 
3) Import it!
keytool -importcert -keystore /new_jre_location/lib/security/cacerts -file imap_cert.cer  -alias [email.servername.com]
By default, the keystore password in the JRE bundled with Jira is "changeit".
Make sure you trust it when prompted and it takes a restart of Jira for this to take effect.

Good luck!

-Kelly Schoenhofen

Friday, October 12, 2012

git - copying one repo into another with history, preserving the folder structure

I'm new to git and I'm learning to be wary of it (like a sharp knife); on one hand the more I dive into it, the more I see how powerful it can be. On the other hand I am starting to understand why the designers of all the big-name commercial source control systems hid all of this flexibility from you; in the right hands (and mindset) git is insanely cool. In the wrong hands it's frustrating, difficult and immensely destructive (to your code and your time). It's like letting a toddler play with a lightsaber sometimes.

With that preface out of the way - a developer had a local repo he had created and worked in for some time, and he needed it pushed up to the corporate git server (where it never existed).
We're currently using gitosis for management (but we're transitioning to Stash - no worries)(unless I should be worried?) so I couldn't figure out how I could just clone it to our server's gitosis folder structure.

The more I read, the more it seemed like my best path was to merge the one repo into a repo I create on the corporate server; but root to root - not into separated folders; 99.9% of the results from google were predicated that you're trying to merge one repo into another at a sub-level, i.e. ProjectA + ProjectB = ProjectA with a ProjectB subfolder in it (or vice versa, or even Project A + Project B = Project C with A & B subfolders).  I needed to merge his repo at the root level into another repo at the root level, if possible.

In the end, I had a couple technical difficulties that I haven't quite conquered -
1) I kept getting odd errors (or no error but it didn't actually do anything) during and after the merge; it was suggested on several similar solutions that merging into an empty repo is problematic. "They say" to always add a dummy file to an empty repo before trying to merge to it. Every log file/screenshot that I found of someone actually successfully merging to a brand new empty repo showed them adding a dummy file to the beginning, before their merge. I had so many issues I ended up doing the same thing.
2) At the end, a git push would hang at a high % (say 75% or 100%) and never finish. I ended up suspecting the Kaspersky Endpoint Antivirus was blocking the connection my PC was trying to make to the git server. I worked around this by repeating all of these steps verboten while putty'd to the git server, logged in as myself in my home dir. I also opened up a helpdesk ticket to add git.exe to the exclusion list for KAV, but that's another story. I'll try to circle back and re-try this if I ever get KAV under control.

Here’s (essentially) my steps. I think there was about 100 ways I could have done this, but this is what I ended up doing:


1) Create the empty repo on the git server itself and add it to gitosis’s administration
2) In my home directory, pull down the empty repo.
git clone git://server/newrepo.git
3) Create dummy.txt with contents, add, commit & push it back to origin
ps -ef > dummy.txt
git add .
git commit -m "adding dummy file prior to merge"
git push origin master
4) Add the developer's repo as a remote
git remote add devs_initials mylogin@devs.pc.com:devsrepo
5) Fetch his repo
git fetch devs_initials
6) Merge the master branch into the new repo and push it to origin again.
git merge devs_initials/master
git push
7) Remove the dummy.txt (its purpose is done)  and commit & push to origin
git rm dummy.txt
git commit -m "Removing dummy.txt, as it's no longer needed."
git push
8) Profit.

Sigh.

Feel free to comment with what a smarter, more experienced git user would have done.

Monday, October 8, 2012

New Job - What Do You Master First?

Sorry for the lack of updates for the last 12 months; I finished up at a place I was at for almost six years, spent 7 months at a different place, then changed yet again. Today marks exactly two weeks since I started at my new place.

So - what do you master first at a new job? What do you make a top priority in scoping out?
What is the first and foremost necessary skill needed to survive a new workplace?

If you have a CM/DevOps bent, you might say "the build system" or "the source control toolset". If you're a developer maybe "the software development lifecycle" or "the nearest Five Guys or In & Out".  Finally, if you're in QA/SA, you might say "the defect system" or "the test case management tool".

As I mentioned earlier, today marks my 2 week anniversary since starting here, and I've mastered the most important and foremost necessary survival skill -

Upon opening the door & taking a single (silent) step inside, I can immediately tell which stall doors are closed and locked (and occupied) versus empty stalls that have swung shut on their own, on the three closest restrooms to my desk.
The results of said single glance will determine if I continue into the restroom or depart as silently as I first arrived.

---

I'll try to be more dutiful with updates for the rest of this year. Thanks everyone!

-Kelly

Monday, November 7, 2011

Build & Deploy Decoupling (and get some tests running in between!)


Pre-preface: We have about 50 heavily linked builds that make up our “core” environment. We’re trying to cut "major" release times from ~3 months to ~3 weeks. Check out the prior blog post for more background information.
---

In a continuous integration/continuous deployment shop, if you decouple the deployments away from the builds you will improve tester productivity, decrease test environment outages throughout the day, find defects earlier, speed up the builds, you are encouraged to invest and rewarded by investing in automated tests and ultimately position yourself for continuous production deployments ala Push Engineering. (scary, but woo!) That’s a lot of claims to make, so let’s get on with it.

Preface - this isn’t a step away from the Continuous Integration or Continuous Deployment methodology, it’s a refinement to keep it from becoming Continuous Interruption or Continuous Regression. 

Productive testers

I’m willing to bet that if you build on every commit, 95% of your test environment stakeholders (development, QA, product owners, etc) that are using the environment in question aren’t looking for that specific fix/feature/revision at that moment. They want it eventually – usually after they are done testing the story they picked up after they sent the first story back to development with a defect.  While they want that fix/feature/etc, they want a Quality change; they want the commit to build successfully so the build isn’t dead on arrival, they want it to pass the unit/system test suite (hopefully you’re at 100% coverage) so there is a high confidence you aren’t regressing existing critical functionality, and they want their test environment to get that revision at an expected time – either scheduled or on demand.  The last thing testers want is to have their valuable time spent setting up a test wasted when the environment is reset on them for a defect fix or change they could not care less about.

Outages

A moderately sized software development unit  – say, two teams adding up to 10 developers with Quality Assurance, Business Analysts, Project Managers, Database Admins and Developer Operations people – can generate a lot of environmental churn on a daily basis. Your busiest environment will probably be your lowest environment;  I frequently see 10 developers able to commit 20-30 revisions every day to the dev branches,  and then those revisions ( once they pass whatever gateway test/merge qualifications necessary)  being selectively merged to the next more-mature environment – generally it divides by 2 or 3 each environment. So your 30 revisions in Dev turns to 10 in the next environment up, then 5, then 2, then maybe you’re up to Prod and it’s hopefully just one build ;) If you clump these deployment cycles into “a” number – without knowing your business or processes, there is no magic number – but let’s assume for a moment we go to four deployments a day in non-production – 9AM-Noon-3PM-6PM, and only when there are changes to push – you’re now limited to 4 deployment interruptions in your lowest environment, to one or two in all other non-prod environments, and it’s on an expected timetable.  From ~50 continuous interruption deployments a day down to ~8 – that’s a slight improvement.
If you have frequent “hot” builds that need to be slammed in as soon as possible, your people environment is probably lacking in software development process maturity. People need to learn the old fashioned way – experience – that there is never going to be a silver bullet fix that solves everything in one build & deploy . You can alleviate the need for “War-room ASAP” builds by working in as close to 100% test coverage percentage as you can afford, level-set the environment stakeholders by explaining how many people/groups are being impacted with a rebuild, communicate the end to end outage length, and put visibility on the rebuild & deployment, each and every one. When the world is watching, that super-important fix can wait 45 minutes until the scheduled, automated deploy happens.  And if you’re in a dynamic place where you have multiple groups working on multiple stories simultaneously, it’s extremely rare for a critical group of people to be blocked & waiting for a deployment.  They may say they are, but in reality they most likely have several other items ready to test, or items marked resolved that they need to sign off on. Don’t get me wrong – you will need to allow for manual deployment pushes – just make sure the button is pushable by the right stakeholders and there is clear visibility on the deployment occurring, why it’s occurring and who pushed the button/authorized it. Listen to the users too – don’t let a pushy manager terrorize everyone by smashing their builds in. You need to listen and push back if need be.

Defects & defect cost

If your deployments are coupled with your CI builds, even with stepped down commits as you go up in environments, that’s a lot of interruptions in each environment. The closer the environment is to Production makes the outages more expensive in the short-term, due to the perceived instability of the higher environments by senior leadership and external clients and the interruptions of client UAT, sales demos, etc.  These outages/interruptions tend to chase the legitimate testers away, and because the lower environments (closest to Development) are being rebuilt most often, you’re training your most valuable testers to stay out and stay away from the environments you actually need them in the most – the sooner you find a defect, the cheaper it is to fix it. When the people who you are making these changes for finally tests a few weeks before release or in an environment  right before Production, having the “you didn’t understand our requirements” card get played is somewhere on a level of fun between playing “Sorry” with your significant other (they show no mercy!) and having the Draw-Four card played on you in Uno, except in a large release  hundreds of thousands of dollars in time & effort are at stake.  That is the long-term cost of low-environment outage and over time will greatly outweigh the short-term cost of your close-to-production environments recycling during the day. Too often we let the “rough” development environments be wild-wild-west, not realizing it’s your most valuable environment to test in.

Speeding up the build

If you have any tests baked into the build, you know they can take just a few seconds to several minutes, depending where they sit between pre-canned, self-contained nunit tests or if you are setting up data and comitting transactions. When you have high percentage levels of test coverage your tests can often exceed the compile times of the build itself; there’s nothing wrong with that. This is where you can successfully argue for more build server horsepower. The time spent on deployments is often out of your hands; you are constrained by network speeds, SAN or local disk speeds, and hard-timed delays waiting for services to stop and restart. The deployment tasks in your build quickly add up; even if deployment tasks are only 10% of your build time, it’s increasing your environment outage by 10%. If you asked QA “Hey, would it be worth trimming 10% off a complete environment rebuild?” they would jump on it. Time is money; a 10% decrease in time is literally dollars – often it’s many dollars added up over dozens of builds a day – being saved.  That’s something senior leadership can definitely understand and get behind.

Test coverage

Not slamming the build hot into the environment provides incentive to build up the automated test coverage percentage. Nobody wants a DOA build, and if they are now getting them 4 times a day max (let’s say) they want them to count. Assuming a 9AM/Noon/3PM/6PM deployment cycle, developers at 10AM might have an easier time making sure their change is covered in unit tests when they know the build won’t be deployed until Noon. I have seen project managers and directors literally standing behind a developer waiting for them to commit because “everyone is blocked!” or “The client is waiting to test!”. That’s not a conducive work environment, nor one anyone would want to work in. Taking the deployments out of the build also gives you the ability to put in longer, slower, deeper automated tests and post-build reports, like test coverage reports, in each and every build. It sounds counterintuitive to speed up the build by removing the deploy stage and then slow it down with more automated tests and post-compile reports, but those tests and reports will increase quality, which will decrease the amount of time spent in the whole “code it-throw it over the wall-have it thrown back-fix and throw it over the wall again” cycle.

Continuous Production Deployments

That’s where I am trying to get to, and that’s why I’m removing deployments from our build cycle. I need two major results out of this – our automated tests need to inc rease in quality & importance, and the decoupled deployment stage - broken out into a separate stage – can be perfected in the lower environments and then used in Production. The only reason I haven’t strung together our piecemeal deployment scripts from each build and run them against Production already is we don’t have good automated tests running (so we can’t automatically trust the build), and we don’t always have good visibility on what needs to be released due to how much shared code we use. Yanking deployments out and relying on and improving our test coverage and perfecting our deployment stage gives me half of this; my next post will be on how I’m going to solve the release build of materials question.

-Kelly Schoenhofen