com.atlassian.bamboo.repository.RepositoryException : Failed to checkout source code :(

While needing to do some stuck-build-cleanup the other week, I did an across-the-board upgrade of our core build & source control servers and clients.
We were scattered between a few versions of Atlassian Bamboo, TortoiseSVN, the CollabNet Subversion Command Line Client (for Windows), VisualSVN Server and VisualSVN Visual Studio Client, though all were Svn 1.6.x compatible and I thought it was a good time to get us brought up to a common, consistent & recent release level since I was going to be in the guts of a few of our servers anways.

With the Subversion project still working on the major 1.7 release, 1.6.x has pretty much ended at 1.6.11, and all our tools (for a change) have released 'final' versions for 1.6.11. I thought it was a good stopping point (since I was doing maintenance anyways) before 1.7 comes out, which we'll need to let mature for a few break fix cycles before we go to anyways, so I thought this would catch us up enough to tide us over for ~six months at most. (I had the same thought 6 months ago, thinking there was no way the Subversion project would take six more months before 1.7 came out).
I also threw on a point release upgrade of Atlassian Bamboo to catch us up on the last 3 months of minor break fixes, an upgrade which wasn't technically tied to Svn 1.6.11.

So I grabbed new versions of all the tools we use, all compatible for Subversion 1.6.11 -

• TortoiseSVN 1.68 x86 & 64-bit Windows clients (for the XP & Windows 7 PC's, respectively)
• VisualSVN 2.01 Client for Visual Studio 2008
• CollabNet Subversion Windows Client 1.6.11
• VisualSVN Server 2.1.2 (for Svn 1.6.11)
• Atlassian Bamboo 2.5.5

I upgraded everything server-side, sent out emails to all the content creators and developers to upgrade their client tools, caught all the server boxes up on Windows Updates and everything got fresh reboots across the board. I disabled Basic Authentication on the Svn server and enabled NTLM Authentication only. Amazingly, it looked like everything still worked, and best of all nobody was getting authentication popups to retype their credentials anymore in TortoiseSVN  ;)

Fast-forward a week, until this morning; our secondary build server, while still showing totally green, is no longer building anything. I only noticed when a developer brought it to my attention that one of the products that is seto to build only on that server hasn't built in the 48 hours after his commit :(
I pick at it and finally figure out it's failing to successfully authenticate with the Svn server when I finally find this error:

Mainline - Product : Error occurred while executing the build for MAINLINE-PRODUCT-184
(com.atlassian.bamboo.repository.RepositoryException : Failed to checkout source code to revision '47527' for https://visualsvn/svn/Company/Mainline/Product)

I've seen similar errors in the past and it's normally a quick fix - somewhere in the maze of credentials that keep us compliant with the auditors, something expired or was cleared. Normally I just on the server in question, manually connect to the Svn repo using TortoiseSVN or CollabNet's client, re-type in credentials or re-accept an SSL cert that somehow got cleared, and done.
This time, everything was working fine already when I manually connected. I cleared all the cached credentials anyway, re-pulled down a small project to get the ball rolling, re-typed in credentials by hand, re-accepted the SSL certs, and re-kicked off a small build in Bamboo that only runs on the secondary server, to no avail. When I do it manually, everything works. When Bamboo does it, nothing works, it fails somewhere around the svn auth.

I decide that the Bamboo remote agent must need upgrading (even though a dusty note on Atlassian's site claims agents upgrade themselves, the local .jar & .exe files for the agent are still dated from 9 months ago, from the original ).
I try to upgrade the agent in place, give up on that when both (?) agents start fighting (!), remove the Bamboo agent completely from the secondary build server, remove it from the Bamboo administrative level completely, and reinstall from scratch on the secondary server. I have to re-setup the builders (msbuild & script), and go into each of the build projects that soley use the secondary server and re-select the build agent.

I tried running one of the failing builds (still green!), and still no joy, same errors as before around svn auth. I verified the remote agent was upgraded to Bamboo 2.5.5, reinstalled it as a windows service, played with the service account for abit, still not working.

I was able to pull a full stack trace out of the Bamboo agent logs, and started googling off bits of the full stack trace. I gleaned enough to go just to the Atlassian wiki/forum and search on bits and pieces of the error message.

Finally, I found this:
Authentication Failure With NTLM Subversion Authentication
http://confluence.atlassian.com/display/BAMKB/Authentication+Failure+With+NTLM+Subversion+Authentication

Grr, grr, grrr. It turns out the Java SVNKit that Bamboo uses is flaky as heck doing NTLM authentication. And if you remember,  I had turned off Basic Authentication and turned on NTLM Authentication 7 days prior, during the upgrade of VisualSVN Server :(
My real irritation is it's only partially failing - our primary build server is fine apparently doing NTLM - it's just our secondary build server that is flaking out.

My steps to rectify this closely follow the article:

1. Set Basic Authentication to be the primary method of authentication in the Bamboo remote agent by adding this to the agent's config file and restarting the service:
• wrapper.java.additional.3=-Dsvnkit.http.methods=Basic,Digest,Negotiate,NTLM
2. Turn on Basic Authentication again in the VisualSVN Server setup and recycle the VisualSVN services.
3. Clear cached Svn credentials on the secondary build server, and manually do a check-out and commit (a whitespace change) of a simple project using the CollabNet command line client, specifying the username and password of the service account the Bamboo remote agent runs as, to establish Basic Authentication credentials instead of an NTLM token. 
4. Manually kick off the builds tied to the secondary build server in the Bamboo console. 

Fixed...

Initial upgrade cost: 2 hours.
Later troubleshooting: 4+ hours on my part, + developer time wasted waiting for their build.

Sigh :(

Setting A Secure Host Header on IIS6/Server 2003

This is easily findable on the Google, but I thought I should include it here in case the prior post drew you in via a search engine. If you want to accomplish the same thing in IIS6, do the following (your AdminScripts folder mileage may vary, I'll use the default on a Windows 2003 server here):


High level example:
C:\Inetpub\AdminScripts>cscript.exe adsutil.vbs set /w3svc/<siteID)/SecureBindings "IP.Of.The.Site:443:Web.Site.Com"

Low Level Example:
C:\Inetpub\AdminScripts>cscript.exe adsutil.vbs set /w3svc/123456789/SecureBindings "127.1.2.3:443:WebServices.SecureCompany.com"

-Kelly

How to set a host header on an SSL binding in IIS7

How to set a host header on an SSL binding in IIS7


If you google for this you'll see alot of answers, but none of them seemed production-ready or correct.
Situation: 

• You have a Windows 2008 web server that is hosting multiple sites on different IP addresses. 
• You want/need to return the site name rather than the box name in a WSDL call to one of your web services.
• You may have multiple SSL certs installed on your server, for each site (i.e. VeriSign certs bound to the domain name of the site). 

Every example I looked was for some other situation, typically a developer's PC using self-generated SSL certs, and "*" is the IP address of the site, but we don't roll like that.
Also, they usually added a binding which works if you don't care which SSL cert you're using, but again, we don't roll like that. We want to edit an existing https binding so we know it's using the right SSL cert, not add a new binding. 

1. Set up a plain-jane https binding on your site with the right ssl cert. Hostheader is probably grayed out, don't worry about it.
2. Navigate to the C:\Windows\System32\inetsrv folder in a command window. 
3. Run this command line - replace everything in #'s with your value:

appcmd set site /site.name: #SiteName# /bindings.[protocol='https',bindingInformation='#IPAddress#:443:'].bindingInformation:#IPAddress#:443:#HostHeader.YouWant.com#


example:
appcmd set site /site.name: Intranet /bindings.[protocol='https',bindingInformation='10.12.1.10:443:'].bindingInformation:10.12.1.10:443:Intranet.MyCompany.com


You may need to recycle IIS for the binding to correctly show up or work, but calling your service via https should now correctly return the host header rather than the server name!

Computer Forensics: The Impact of Electronic Evidence on Modern Corporate Investigations

I attended a 2-hour lunch & learn hosted by the The Business Bank & RJ Ahmann Company at the Golden Valley Country Club, titled “Computer Forensics: The Impact of Electronic Evidence on Modern Corporate Investigations”. We ate while 3 speakers talked. Mark Lanterman was the last and primary speaker - I didn't catch the first two speaker's but they did excellent jobs. Mark is the CTO of a company called "Computer Forensic Services", a pretty high-end firm brought in by plaintiffs, defendants and the court system in general to figure out electronic evidence. His recent most high-profile cases have been Denny Hecker, Tom Petters and Paul McCartney's divorce. 

The initial speaker worked for The Business Bank and talked about banking/e-commerce fraud and what the e-commerce field is doing about it. 

Since you all love write-ups, here's some cleaned up notes and some observations of mine on the talks - some tentative best practices being put into place by the industry. Since we run an e-commerce platform ourselves, it behooves us to take a look at what the trends are, where our gaps are and maybe where we want to go ourselves. 

He primarily spoke regarding people who have admin accounts with their companies financial institution, or people with the role of orderers in an e-commerce platform.

 

1)    When a client’s admin account creates additional accounts, the account is held pending until a separate confirmation is made. They said it’s the hottest source of fraud right now – an admin’s account gets hacked, and the only thing the hackers do is create additional orderer accounts, and do their fraud from the fake orderer accounts. If it’s done properly, you never catch on that an admin’s account has been compromised (or which one), so they can keep doing it and doing it and doing it.
2)    As a result, have admin accounts password expiration be much faster than a normal user. They even referenced daily/every login expiration they have implemented. If the password changes every time the account it used, it makes the account harder to keep compromised and/or resell on the black market. It’s better to have less complex passwords (to keep users OK with 1-day expirations) then to have super-complex passwords that never expire.
3)    When activities of an orderer are outside the norm, insert challenge questions into the web page they have to answer before they can continue. If they can’t answer them, their account is locked until they call in.
4)    Geotracking (IP) restrictions on orderer/admin logins.
5)    Orderers/admins select a background watermark picture when their account is first created/logged into. This background watermark ensures they are in the “real” website when ordering/doing admin tasks.
6)    Issue RSA token fobs for admin & orderer accounts. It’s cheaper than a single fraud investigation. Usually you can charge the fob to the client and they are happy paying for it.

Corporate-wise:
1)    This may sound draconian, but no file attachments in (externally bound) emails. None. Email is not secure. Secure email is still not secure. No file attachments, no documents, period.
2)    If you need to share files, use an external service. They threw out “ShareDefender” as an example, for (casual) secure file transmission between you & clients.

Mark Lanterman gave the longest talk, and he mainly used "warstories" to illustrate where computer forensics have gone, and the role corporations play there.

1)    Deleting evidence highlights what needs to be looked at by investigators. You can’t find a needle in a haystack but you can often see where the needle was by the missing hole, and then you know where to start looking.
2)    In the last couple years, for the first time ever in Minnesota, plaintiffs have been sanctioned for “evidence spoliation”.
3)    Evidence collected for one court case has led to many other cases. I.e. two executives sue each other and the 3rd party court appointed forensic investigators uncover internal fraud from IT purchasing, etc.
4)    Give out rich/smart devices to employees; they are much easier to monitor and collect much richer evidence. They named the iPhone as top of the list as greatest un-intentioned evidence collector on employees.
5)    Even if the corporation didn’t do the theft, if an employee brings IN illegally obtained data/software and uses it, or forwards it on for other departments to use, the corporation can and will be help liable in a court of law. This happened in the Pioneer Press vs Star Tribune court case a couple years ago. You cannot let new employees bring in shady data, shady software or shady devices. You will be held liable for their use. The Star Tribune was fined & sanctioned for this, they also had to pay Pioneer Press’s complete court costs and all expert testimony fees.

That last point is highly relevant to your employee onboarding process. There is absolutely no allowance to bring in home laptops, home software, USB drives from home, nothing. You have to actively prevent it, not just say “hey guys don’t bring in your favorite software from home, or the client list from your last place of work”. They are literally exposing your entire company to successful litigation & sanctions.

-KellyS

Web.config inheritance madness!

We’re slowly migrating to Server 2008/IIS 7, and we hit on a curious issue the other day. We have a marketing "subsite” that’s incredibly basic – think a handful of .html and .asp pages -  in the same domain space as our flagship asp.net application, just taking up an isolated subfolder (and I actually use a virtual directory, because there’s no good reason to physically host one project inside another project, let alone an html/asp project inside a .net 3.5 project where you have two totally disparate groups making changes). Other than in Production nobody makes a habit of hitting the marketing site. So when a break-fix for one of the few pieces of functionality it has showed up, Development wasn’t able to pull up the marketing subsite in Test. They got this instead:

Server Error in '/Child' Application.

---

Could not load file or assembly 'Super.Awesome.v3.3, Version=3.3.2.0, Culture=neutral, PublicKeyToken=abc123youandme' or one of its dependencies. The system cannot find the file specified.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.IO.FileNotFoundException: Could not load file or assembly 'Super.Awesome.v3.3, Version=3.3.2.0, Culture=neutral, PublicKeyToken=abc123youandme' or one of its dependencies. The system cannot find the file specified.

The assembly that whacked us is an http module used by the parent asp.net website, referenced in the parent folder’s web.config in httpModules and system.webServer modules. IIS 7 has some great inheritance features, but they never seem to work when you want them to (MSEntLib database connection strings) and work when you don’t want them to (referenced modules).

Googling around, there are a few solutions, but the one that seems to work best is the one I didn’t like at all, initially. They want you to modify the web.config of the parent application to stop inheritance. I’d rather modify the child application to block inheritance. MS’s method seems backwards to me, more of a security risk and a hinderance than a help. If you’re on a locked down server farm (say, rented web space at a 3rd party co-location facility), anyone with physical rights above your folder can make you inherit whatever you don’t explicitly call out. They could write a http, network or file stream logger that steals logins and passwords, for instance. Securing your site via https wouldn’t help because they would be on the inside of the protection. And you wouldn’t be able to do anything about it; you wouldn’t even know what you were inheriting.

Alright, soap box time over. Here’s the simplest way to block web.config section inheritances downstream:
Insert
<location path="." inheritInChildApplications="false">
and
</location>
Around the section you want to stop child inheritance for.

I used system.web & system.webServer in this example, as that’s where extension mapping modules seem to be loaded most commonly.

<?xml version="1.0"?>
[…]
  <configuration>
    <location path="." inheritInChildApplications="false">
      <system.web>
      […]
        <httpModules>
          <add type="SuperAwesome.HttpHandlerModule, Super.Awesome.v3.3, Version=3.3.2.0, Culture=neutral, PublicKeyToken=abc123youandme" name="SuperAwesomeHttpHandlerModule" />
        </httpModules>
      </system.web>
    </location>
  […]
    <location path="." inheritInChildApplications="false">
      <system.webServer>
        <modules>
          <add type="SuperAwesome.ASPxHttpHandlerModule, Super.Awesome.v3.3, Version=3.3.2.0, Culture=neutral, PublicKeyToken=abc123youandme" name="SuperAwesomeHttpHandlerModule" />
        </modules>
      </system.webServer>
    </location>
  </configuration>

Not the way I would have handled this (to repeat, I would have had the child web.config be able to block inheritance) but it could be worse.

WCF endpoint over https = no wsdl response?

We've had a torturous time getting a client connecting to an externally exposed (http) WCF endpoint. The biggest "fix" was enabling rich error reporting which sped up troubleshooting incredibly. To enable the real error to come through to your client consuming your endpoint (this probably isn't a great Production setting, but it's invaluable in Test), add this to your endpoint behavior:

<behavior name="MetadataBehavior">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="True" />
</behavior>

So we get out of our sandbox environment and get closer to production, and the client starts reporting they can't add our almost-production-but-not-quite endpoint as a service reference. We pick at it and we're seeing errors like this:

The document at the url https://sanitized/sanitized.svc was not recognized as a known document type.
The error message from each known type may help you fix the problem:
- Report from 'https://sanitized/sanitized.svc' is 'The document format is not recognized (the content type is 'text/html; charset=UTF-8').'.
- Report from 'DISCO Document' is 'Discovery document at the URL https://sanitized/sanitized.svc?disco could not be found.'.
- The document format is not recognized.

(etc etc)

It turned out if you called https://sanitized/sanitized.svc?wsdl via your browser, you didn't get a wsdl response. You got the regular "You have created a service." front page. That's what the error message from above is actually saying - it was expecting a text wsdl response and it got a nice html Hello World page from your service.

With that clue, our lead dev on this tracked it down quickly - you have to enable https on the endpoint behavior. We had already set the Security from None to Transport at an earlier point when we had the client go over SSL, but it wasn't enough apparently for this service.
I set the endpoint behavior to this:

<behavior name="MetadataBehavior">
<serviceMetadata httpGetEnabled="true" httpsGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="True" />
</behavior>

and things got much better.

Strong Naming Assemblies in MSBuild

For a few years, it's been requested our core dll's be strongly named so they can be invoked by 3rd party tools. It's always been a bigger effort (we thought) then the payback so we kept putting it off. Now we need them strongly named so they can be used by say, SSIS.

I spent a ton of time on this just going in circles, and when I actually got it working it was pretty simple. You can strong name your assemblies many ways, but two stand out.
First off, I had problem after problem using the pkcs#12 key I generated using OpenSSL for various tasks (such as signing the manifest for a ClickOnce install, another MS technology). Eventually I read that Microsoft’s strong naming algorithm is just too tight on unnamed standards, and it’s pretty common to get stuck in the endless “Import Key” popup during a manual build, which I did. The fallback is to just generate a key using sn.exe, which I ultimately did.

Using sn.exe to generate a strong naming key is simple, if you’re not concerned about key security. We’re not a shrink-wrapped software place and we have tight controls about the migration of compiled code so I wasn’t concerned about that.

On your build server just do the following out of the bin folder of the appropriate Windows SDK you have installed:

sn -k CompanyBuildKey.snk

and now you have a MS-compliant strong naming key.

The first way to strong name assemblies is to do it in the msbuild script, at compile time.
A good practices is to copy the key to the C:\Program Files\MSBuild folder on the build server, and then you can reference it like this in your msbuild scripts:

$(MSBuildExtensionsPath)\CompanyBuildKey.snk

In your msbuild file on your build server change the following sample line from this:

<MSBuild
Projects="@(ProjectList)"
Properties="Configuration=$(Configuration)"
Targets="Build" />

to this:

<MSBuild
Projects="@(ProjectList)"
Properties="Configuration=$(Configuration);SignAssembly=true;DelaySign=false;AssemblyOriginatorKeyFile=$(MSBuildExtensionsPath)\CompanyBuildKey.snk" Targets="Build" />

This worked fairly well, but it started to generate two worlds – the developers are all unsigned but the build server is signing everything. If a developer wants to use compiled dll’s as references in another project, they can run into issues mixing strong (from the build server) and non-strong dll’s – a strongly named dll cannot call a non-strongly named dll.

So the second method is probably for the best, as it puts everyone on the same page.

Second method:

Move the CompanyBuildKey.snk into a neutral reference. If you have a common library folder in source control that the build server & developers all use, this is a good spot for it.

Move your strong name compiling into each project file and out of the msbuild script.
Add this to each project file:

<AssemblyOriginatorKeyFile>\Common\Folder\For\All\CompanyBuildKey.snk</AssemblyOriginatorKeyFile>
<SignAssembly>true</SignAssembly>

I hate to put this out without a clean followup, but I can’t do the bulk of the strong naming I need to do for another couple weeks. I’ll try to do a follow-up post on how that works out.