How do you retrieve test results from Bamboo via a REST call?

Sorry for the lack of updates; I was recently in a pretty bad biking accident and I've only recently returned to work. So!

I was asked by someone how they can retrieve test results somehow from Bamboo, because they have a separate, custom dashboard they use for displaying test results.

I know Atlassian has a REST API for all of their main products, is it possible to get test results back using it?
Yes, it is possible and not too difficult. The main caveat is that the smallest (and largest) group of test results you can retrieve is job sized.

Here's the latest documentation on Bamboo's API:
https://docs.atlassian.com/bamboo/REST/5.6-SNAPSHOT/
https://developer.atlassian.com/display/BAMBOODEV/Using+the+Bamboo+REST+APIs
https://developer.atlassian.com/display/BAMBOODEV/Bamboo+REST+Resources

To retrieve results, we'll use curl (though Python and many other tools are available) and xmllint to output the results in pretty handsome xml. (so if you're using Ubuntu (and who isn't?) "sudo apt-get install curl libxml2-utils" will fix you up)

Our example Bamboo server's base rest url is:

https://bamboo.company.com/rest/api/latest/

We can get build results back by adding

/result/projectkey-buildkey-job/buildnumber 

to the url.

Test results will be brief until we expand the test results, so we add

?expand=testResults.allTests 

to the end of that. If you're just looking for failures, or a slew of other filters, they have that too.

It defaults to xml output I believe, but you can output JSON format by adding .json to the end of the url, before the arguments. Word is that JSON will be deprecated at some point, but for now you can still get results back that way. I'll show how in a minute.

I don't want to specify the build number, I just want the latest results, and Atlassian supports "latest" as the build number. So, here's our complete url, using a made up plan with unit tests as our example:

https://bamboo.company.com/rest/api/latest/result/PLAN-STAGE-JOB/latest?expand=testResults.allTests

If you want JSON output, it would be this:

https://bamboo.company.com/rest/api/latest/result/PLAN-STAGE-JOB/latest.json?expand=testResults.allTests

Now, it outputs a huge block of unformatted xml by default. If you want to to look nice, we pass it through xmllint, by appending this to the end:

  | xmllint --format -

By default, anonymous is disabled in our install, so we need to pass authentication. The simplest example is using basic auth, so the complete command line would be this, using your username and password:

curl -u username:password https://bamboo.company.com/rest/api/latest/result/PLAN-STAGE-JOB/latest?expand=testResults.allTests | xmllint --format -

And it outputs something like this:

There's much more you can do beyond retrieving test results. You can also use this to kick off plans, get deployment results, etc. And, each release of Bamboo expands the REST API.

-Kelly G. Schoenhofen

Code Review Tools - Stash "versus" Crucible

We've been using Atlassian Crucible for code reviews, but Atlassian Stash becomes the scm/git server standard over the last 12 months, some of the teams are starting to use Pull Requests in Stash as their code review vehicle. As the Gospel According To Stash spreads, inevitably managers are asking "Hey, what is the deal? Is Crucible going away? Should we be switching to Stash Pull Requests?"

So, let's look at it. Some background - this is the state of the products of early 2014. We're on Crucible 3.3 (March 2014) and Stash 2.10 (Feb 2014) - there is a newer release of Stash out right now, and that's mentioned. We're also almost entirely git, but using other scm products is brought into play. 

Stash

• Stash reviews work best when all your changes can be encapsulated in single pull request from a single repository. 

Crucible

• Crucible reviews work best when your changes contain commits from multiple repos - Stash cannot span multiple repos in a single review. 
• Crucible also supports pulling in changes from multiple source control systems (cvs/svn/git/et al) & multiple scm servers (i.e. two different svn servers) in a single review - Stash code reviews have to be pinned to a pull request in Stash only (so git is your only scm choice), and only from a single repo local to the Stash server you create the request on. It's sort of a fine point that 99% of people wouldn't care about, but it needs to be pointed out if you require a certain flexibility. 
• Crucible will let you review any (set of) commit(s), Stash only lets you review pull requests. So if you wanted to do a formal review of an entire release, or maybe a point in time snapshot before you upload the changes to a public github repo -Crucible would be your tool. Stash cannot free-review commits. 

Conclusion

Crucible is a very established, stable product with a wide audience; it has a very active development team behind it but the audience it caters to is very broad. The Crucible platform probably won’t radically change in the future. Crucible has a long future in front of it. 

That said, Stash is a new product and has a very focused audience - git users only, and catering to the most popular git workflows. It also has very active development team behind it, but because the audience is so narrow compared to Crucible, the change velocity appears much higher. The upgrade to Stash (to 2.11 - the March 2014 release) I'm doing this weekend adds side by side diff, commit commenting outside of pull requests/code reviews, comments at the file level in a commit (for instance, the deletion of a file, or a binary file that can’t be parsed), and a much tighter integration in Jira. Pending pull-requests tied to an issue will now be visible in Jira after this weekend. 

Stash has evolved insanely quickly; Stash 2.0 came out last January, and Stash 2.11 came out a few weeks ago - that’s 11 major(ish) releases in 14 months. Many of the features in Stash that our development teams use on a daily basis simply weren’t there just 12 months ago. I can’t imagine what the product will look like 12 months from now.  

Pull Request code reviews really shine when you can embrace the idea; I highly recommend using them if it fits your team’s needs. If you've gotten this far and pull requests are an unfamiliar term, a quick refresher:

Pull requests are like saying "request to merge". 

1. You branch server side on any change - a feature, a bug, a hotfix. 
2. You make your changes, commit them and push your changes back up to the server.
3. Your Continuous Integration system shows your peers that your change builds & passes the automated test framework.
4. You create the Pull Request to merge it back into the parent branch.
1. Add appropriate reviewers. 
2. They review your change, if they comment & request changes you make those changes and this iterates until they approve the pull request. 
3. You complete the pull request and Stash automagically merges your changes back to the parent source line and deletes your branch.

If you do this, or could do this, pull request code reviews can be a very powerful thing. 

Tl;dr - Crucible & Stash are both awesome, use what works best for your team.

-Kelly Schoenhofen

Commit/Push Driven Builds Between Atlassian Stash & Bamboo

By default build systems poll scm every so often; every 180 seconds is probably the most common setting. Very quickly though, between your different systems - your build systems (Jenkins, Bamboo, TeamCity, etc), defect systems that keep an eye on commits like Jira, or code review tools like Fisheye/Crucible, you can be querying your git server several times a second, 24/7. 

To reduce that load and make things a bit more accurate (For instance, a best practice in continuous integration is to test every push individually; this helps pin down which push broke a build, decreased the code coverage, or increased the Coverity detected defect count. Polling every 3 minutes usually gets each push individually but not always, not even close.) git has had post-receive web hook scripting built in for some time. 
I use Stash as an enterprise git server, and digging into the raw folder structure of git (housed on the Stash server) is not a best practice. Not forbidden, just not recommended.  Luckfully, there is a post-receive webhook plugin for Stash.  

To enable commit-driven builds in Bamboo, it’s just two easy steps. 

Step #1) In Bamboo, switch your build trigger from “Repository Polling” to  “Repository triggers the build when changes are committed” (that rolls right off the tongue); you have to fill in the trigger IP address; leaving it blank disables repository triggering. Enter your Stash server's IP address here. The screenshot should explain the preceding text better than the text itself:

Step #2) In Stash, enable the post-receive webhook and give it the Bamboo url to call to trigger the build. Under the repo’s Settings, go to Hooks, Post-Receive WebHooks, and the url is:

https://bamboo/api/rest/updateAndBuild.action?buildKey=PROJECT-PLAN

Change the PROJECT to your project key, and PLAN to your plan key. You can get those values off the Bamboo page url you just left on the prior tab in your browser. See the picture below for a picture.

Caveat #1) If either the git server and Bamboo fail to communicate when the post-receive hook is called, Stash/git will not re-fire the hook until the next commit, so when using post-receive webhooks it’s considered a best practice to add a second build trigger to poll your git system every few hours for changes. That wasn’t in relation to Stash’s stability/availability, I think it’s just the general git ecosystem in general. So far I’ve never seen Bamboo<->Stash post-receive webhooks fail, but I don’t have a lot of datapoints yet. I am currently not adding a second build trigger (polling) as a backup until I start seeing data indicating relying on just webhooks isn’t robust enough. 

Caveat #2) Bamboo will compare the hash of the last scm pull to the notification hash, and if they aren’t different, it won’t trigger a build. So when setting this up and/or troubleshooting, your Bamboo build has to be on different commit than when the rest call is made or no build will occur (and no error will be posted). 

Automatic Static Code Analysis of Branches

Hey. Let's setup the scenario - let's say you are in a c/c++/java shop, and you also use Coverity, and you have your main source line setup with a nightly Coverity static analysis run. Awesome, you are way ahead of most places.
Let's say development uses branches - either a home-grown solution, or an older model of trunk-branch-tag or maybe even something newish like git-flow - and the benefits of static code analysis is starting to show returns. So maybe some of the development time for the release is being allocated for clearing some of the backlog of defects discovered from prior static code analysis runs, or you want static code analysis done on the feature/development branch in near-real time and eliminating new defects found by static code analysis during the development process is now what the doctor ordered, but it's very time expensive for you to either be setting up new Coverity projects & builds, or maybe you're modifying the single Coverity build you have in place to point to different branches endlessly.
That's no fun for anyone - you're now a gateway (i.e. bottleneck) and if you have multiple branches in development, this can be a serious time sink.
So you've probably already said to yourself, "Self, I sure wish it was easy for Coverity and our build system to auto-create new Coverity runs when a branch is created!"  Well luckfully for you, there is a solution! Atlassian Bamboo 5 has excellent automatic plan cloning tied to branch detection, combine that with Coverity 6.5's project & stream management at the command line, and you get automatic Coverity builds as branches are detected, all using the same triage store so when one defect is triaged, it's reflected across all branches.

Rough workflow:

1. Set your project name, stream name, triage store, and code branch. 
2. Attempt to make the Coverity project with the branch name. 
3. Attempt to make the project stream with the branch name.
4. Link the project stream with the Coverity project. 
5. Run the Coverity static analysis. 
6. Upload the results to your Coverity server. 

Pre-reqs:
In your Bamboo project, enable Branch Detection.

Note: Bamboo does not automatically go back and setup builds for older branches; it only detects new branches going forward. If you need to  have it detect a branch created prior to you enabling branch detection, just go to the "Manual" branch creation and select the existing branch.
If you want every branch going forward to be used, leave the regex expression field blank. If you use something like the git-flow model, then you only want branches prefixed with "feature-", "release-", or "hotfix-" to be considered; in that event, you would use something like this:
(feature|hotfix|release)/.*

In Coverity, the "dummy" user you use for static analysis will need enough privileges to create projects and streams.

I use a separate Bamboo build from the normal continuous integration build for Coverity runs; usually Coverity takes 3X to 10X longer than your CI build, so while the CI build runs every commit, Coverity analysis only runs with a 3AM SCM polling trigger - if there wasn't any changes that day, Coverity analysis doesn't run. There's little point in re-analyzing code that hasn't changed since the prior run.

Since we have branch detection enabled, our Coverity static analysis user is now above to create projects & streams, here's an actual Bamboo task (inline script) to do a branch-named Coverity run. This is for a Java application being built on a Linux build server. The comments are extra - currently Bamboo doesn't like comments in its scripts, but I added them here make it clear what is going on.
The setup:
The application is called App-X, and the jar build we're following here is the CoreComponent.jar.

 PATH=~/cov-analysis-linux64/bin:$PATH  
 # setup vars  
 proj=AppX  
 stream=CoreComponent  
 covlang=java  
 covtriage=Common-Java  
 covproj=$proj-${bamboo.repository.branch.name}  
 covstream=$proj-$stream-${bamboo.repository.branch.name}  
 # create project, create stream, associate stream with project.  
 cov-manage-im --mode projects --add --set name:$covproj  
 cov-manage-im --mode streams --add --set name:$covstream --set lang:$covlang --set triage:$covtriage  
 cov-manage-im --mode projects --update --name $covproj --insert stream:$covstream  
 # finally do actual analysis  
 rm -rf imed  
 cov-build --dir imed ant -buildfile build-ci.xml  
 cwd=$(pwd)  
 cov-analyze-java --dir imed -j auto --strip-path $(pwd)  
 cov-commit-defects --dir imed --stream $covstream --user username --password password --host coverity.yourcompany.com  

Assuming you're using git, and this is the master branch (default), this makes a project called "AppX-master", and a stream under it called "AppX-CoreComponent-master".
Let's say they create a branch called "develop", meant for the next major release of App-X. Bamboo will detect the branch, clone the trigger, and that night it will run a copy of your main Coverity plan and it will create a Coverity project called "AppX-develop", and a stream under it called "AppX-CoreComponent-develop", using the same triage store the -master streams are using.

Eventually branches go silent; Bamboo will clean it up the branch copy of the plan 30 days after the last detected commit, and Coverity 6.5 will clean up detailed snapshot results (at the code level) in its database within ~120 days by default, I believe, so no worries about exploding disk usage and long term database growth.

This should give your company an advantage by being able to get immediate (within 24 hours at least) static analysis results on branches which should translate to a higher quality of code before you release, which should mean a better release.

-Kelly Schoenhofen

Webhook Call On Git Branch Change

Most modern build systems support webhook calls; simply put, your source control system does a command or makes a web service/url call to initiate a build, rather than the traditional SCM polling or timed/cron based build cycle from your build system.
The advantages of using hooks are that sometimes polling your SCM system is expensive in either time, bytes, or logging; or sometimes your build software doesn't support your SCM system directly and can't poll it. Or if you're trying to refine your continuous integration and you want to try to ensure each commit is processed separately rather than in batches, per your polling timer.
Usually the build system exposes this via a job-specific url; different SCM systems expose this in different ways. Git uses a "hooks" subfolder in your repo, here's fairly complete documentation on it:
http://git-scm.com/book/en/Customizing-Git-Git-Hooks

On our main git server, we use post-receive text files to initiate builds for the projects that ask for webhooks vs cron or scm polling. Here's a typical post-receive text file, calling a Jenkins job:

#!/bin/bash
/usr/bin/curl -u username:password http://jenkins.company.com/job/Top-Level/job/Build-Job/build?token=SecretToken

Fairly straightforward. The other day I was asked the other day to setup a webhook for branch commits in git. I did some googling around, stole some ideas and this is the result:

#!/bin/bash
while read oldrev newrev refname
do
    branch=$(git rev-parse --symbolic --abbrev-ref $refname)
    if [ "master" == "$branch" ]; then
        /usr/bin/curl -u username:password http://jenkins.company.com/job/Top-Level/job/Master-Build/build?token=SecretToken
    fi
    if [ "feature" == "$branch" ]; then
        /usr/bin/curl -u username:password http://jenkins.company.com/job/Top-Level/job/Feature-Build/build?token=SecretToken
    fi

done


This will watch for changes on "master" and a feature branch called "feature", and kick off different builds respectively.  You could also only look for master, or just for feature, or extend this considerably.
Prior to this I had only thought of the hook scripts as web calls, not as full-blown executable scripts with the ability to do logic. I had a harder time trying to figure out how to google the question than putting the pieces together once I started getting relevant results back.

-Kelly Schoenhofen

Duplicate Notification Emails In Jira

Once the dust settles from a new Jira install and you have users active in it on a daily basis, the little things start to get noticed. A frequent complaint is how Jira seems to spam users with email notifications. If you google around for "why am I getting duplicate jira notifications" you find alot of results, such as:
https://answers.atlassian.com/questions/69131/duplicate-notification-emails-from-jira
https://confluence.atlassian.com/display/JIRAKB/Notifications+Troubleshooting
https://answers.atlassian.com/questions/44064/how-to-prevent-duplicated-outgoing-mail-on-notification to name a few of the best search returns.

Atlassian even put a wizard into Jira for admins to figure out why a specific user got a duplicate email on a specific issue and action. However, the wizard doesn't work for notifications that aren't about a specific action on a specific issue, for instance when someone uses the "Contact Administrators" form. Luckfully, duplicate emails are almost overwhelmingly not caused by bugs in Jira, but the way your user directory is setup and the way your notification schemes are setup.  Doubly luckfully, here's the two rules that will help troubleshoot most duplicate email issues in my experience.

1) Jira, when it sends emails, only checks that the username is unique, not the email address.
2) Certain events get forwarded to everyone with the Jira Administrators global permission.

A messy user directory will make what looks like duplicate emails because of rule #1; if you have an internal directory with jsmith@company.com as a username and you then connect Jira to active directory and importes domain\jsmith as a user, Jira will treat jsmith@company.com and jsmith as two separate users (because they are!) even though they have the same smtp email address. If they are both in the same group that has a project role that gets notifications, it will appear that Jira is sending out duplicates when it's actually not.

For rule number two, let’s take the ContactAdministrators form for instance; it’s one of those system events that emails everyone with the Jira Administrator’s global permission.  If you check your Global Permissions, you can see which user groups have the Jira Administrators permission. Then check the email addresses for everyone in every group with the Jira Administrator global permission; you will almost certainly find either duplicate email addresses (remember, Jira checks usernames for duplicity when it sends emails - not email addresses!) or unique email addresses that have the same destination (for instance, jsmith@company.com and john.smith@company.com may both be valid and go to the same person, depending how your email scheme was setup), or unique email addresses that result in a distribution group; perhaps you're jsmith@company.com and you're a member of jira-admins@company.com. Jira will email both and not feel bad in the slightest.

Long term, you need to:
1) Be aware of Jira's behavior so you can be more tolerant of what looks like duplicate emails/spam.
2) Be very wary of internal directory accounts using the smtp email address of a distribution group. Jira will not parse the contents of a distribution group - you literally can't over SMTP.

-Kelly Schoenhofen

Linux App + ports below 1024 + not root = permission denied

Issue: you set your application on a Linux server (say Jira, Fisheye, Stash, Bamboo, etc) to port 80 and/or 443, and you get "permission denied" right away during startup. 

This has been a growing issue for the last couple years - it appears that sometime around the Linux 3.0 kernel release, they made the decision to deny port binding/usage below 1024 to applications not running as root. Example: a Tomcat based application can't use port 80 with running as root. I get some of their reasoning - it's harder to run a rogue website on port 80 on a box you've slightly compromised and steal people's data/identity/etc, if you don't have access to port 80 unless you're a super-privileged user. But it sure makes things difficult when your user base wants to type in http://application.company.com internally and it doesn't come up. 

Keeping an application on its default port (let's say port 1234 for simplicity's sake) has alot of advantages, for instance:

• Future upgrades are a breeze, because you don't have to "fix" configuration files the upgrade overwrites. In fact, some upgrades detect your changes and if the migration of your changes fail, the upgrade can get very messy. (i.e. new features that need to be part of your server.xml file that you modified to use port 80 aren't written during the upgrade because it's too freaked out by your small change). 
• Documentation & QA from the vendor is all based around a clean, out of the box install. When you start messing with ports, you start playing with fire. 
• Staying with default ports means you can run your application as a non-root user; it's a really good idea in the first place, and as more corporations embrace Suse/Ubuntu/etc as viable application servers, security standards are being set & enforced, and not having access to root/sudo is becoming more common. 
• Port proxying and reverse-proxies aren't always a panacea, because they kill gzip compression of the web stream and while they do their best, they are an imperfect software solution; you are going to take an increasingly performance hit the more web traffic you have to deconstruct, reconstruct and forward, and the whole deconstruct/reconstruct process isn't always transparent to the client or server ends; weird behavior or bugs can result. And if you call for support, the first thing they will have you do is bypass your proxy and reproduce the issue. And if the issue is from the proxying, they won't support you. 

But sometimes you have users (or management) that really want to type in http://app.company.com and have it work. 

What I've done in the past is use apache2 to setup a simple port redirection (not proxying) so users can type in http://app.company.com and it redirects them seamlessly to http://app.company.com:1234. 

Sample setup using a clean Ubuntu 12.04 LTS server, with an existing application listening for http requests on port 1234 and https requests on 1443:

1) Install apache2 (as in sudo apt-get install apache2)
2) Enable ssl - sudo a2enmod ssl
3) Edit the now-existing /etc/apache2/httpd.conf - 


NameVirtualHost *:80

<VirtualHost *:80>
   ServerName app.company.com
   DocumentRoot /home/appuser
   Redirect permanent / http://app.company.com:1234
</VirtualHost>

<VirtualHost *:443>
   ServerName app.company.com
   DocumentRoot /home/appuser
   Redirect permanent / https://app.company.com:1443
   SSLEngine on
   SSLCertificateFile /home/appuser/app.crt
   SSLCertificateKeyFile /home/appuser/app.key
</VirtualHost>


This works fairly well - it leaves the original ports intact for direct calls (i.e Jira needing to integrate with Stash or Fisheye - it will not work via a redirect), and users will automatically use the new port when they bookmark pages.  
---
However, let's say you need to run on port 80 & 443. Authbind is probably your best bet, I've never had success with setcap. I'll use Jira under Tomcat as an example, but you can adapt this to anything. 

The following steps assumes a nice clean Ubuntu 12.04 LTS server, a dedicated jira user and an existing install of Jira 5.2.

1) If you have anything using port 80/443 already, you'll have to remove it. In my prior example I used apache2 to do some redirects for user browsers; you might as well uninstall apache2 completely now. (i.e. sudo apt-get --purge remove apache2.2-common)

2) Install authbind (i.e. sudo apt-get install authbind)

3) Bind your ports to your dedicated Jira user with touch, chown and chmod. Here's port 80:

sudo touch /etc/authbind/byport/80
sudo chown jira /etc/authbind/byport/80
sudo chmod 755 /etc/authbind/byport/80

Repeat as needed for other ports like 443.

4) Disable ipv6 in Jira/Tomcat - this was not clear to me for quite while; I glossed over that authbind does not currently support IPv6, and I didn't realize ipv6 was the primary protocol in Tomcat when it's enabled on your nic. (i.e. add CATALINA_OPTS="-Djava.net.preferIPv4Stack=true" in the setenv.sh in your jira/bin folder). You cannot skip this step! Use ifconfig to see if Ipv6 is enabled; if it is, you have to disable it in Tomcat.

5) Don't forget to change your /jira/conf/server.xml to port 80 and/or port 443 instead of 8080 & 8443 respectively. To use Jira/Tomcat's basic http to https redirection, make sure your 80 block contains redirectPort="443" in it. If your users use http against anything but the exact url your SSL cert is for, they will get an ssl warning about a certificate mismatch. 

6) Reconfigure Jira to have authbind launch it with the --deep option so child spawns are covered. 

The simplest spot to set this is at your /etc/init.d/jira script, but if someone stops and starts Jira manually in the /jira/bin folder, then you will get permission denied on ports below 1024 and Jira won't be accessible. I got a little messier and edited the start-jira.sh script, so the /etc/init.d/jira script is covered (because it execs start-jira.sh) and other people who jump on your server and restart Jira don't have to know to start it with authbind --deep. Change your start-jira.sh to the following (my additions on line 4 & 6 - specifically the "authbind --deep" portion): 

if [ -z "$JIRA_USER" ] || [ $(id -un) == "$JIRA_USER" ]; then
    echo executing as current user and authbind
    if [ "$PRGRUNMODE" == "true" ] ; then
        exec authbind --deep $PRGDIR/catalina.sh run $@
    else
        exec authbind --deep $PRGDIR/startup.sh $@
    fi

Bonus 7) If you want to get extra fancy, so users get redirected from whatever http address they try your Jira instance with (say http://jira when it should be http://jira.company.com) to the correct, FQDN using https (so no ssl cert mismatch warnings), add the following rule to the /jira/WEB-INF/urlrewrite.xml instead of redirectPort="443" in the server.xml:

<rule>
   <name>Http Check</name>
   <condition type="scheme" operator="equal">^http$</condition>
   <from>^/(.*)</from>
   <to type="permanent-redirect" last="true">https://jira.company.com/$1</to>
</rule>

and then remove the redirectPort="443" from your server.xml. 

For brevity, I glossed over alot of bits that hopefully are trivial. Hopefully this helps other people get Jira (or any other product) working on ports below 1024, and not run with sudo/root. 

-Kelly Schoenhofen