My company has just started using Team Foundation Server. And whilst it is awesome in many ways it's support for testing frameworks other than MSTest is a bit rubbish. Well completely non-existent actually. Unless your test results are in the MSTest format then you're not going to be uploading them to TFS anytime soon. Fortunately others have already come across this problem... There is a project on CodePlex called NUnit for Team Build that is basically an XSL transform to take results from NUnit and transform them into MSTest's format.
However we still have to get those results into TFS. Which means a manual upload from Visual Studio or using MSTest with a few well chosen command line arguments. So if you want to automatically upload tests from your build server to TFS you'll need to install MSTest on your build server. MSTest isn't exactly and easy-to-install self contained tool so you're either gonna have to stop copying random assemblies around until it works or install Visual Studio on your build server. Bollocks to that!
Given that TFS is supposed to be all nice and web servicey it shouldn't be too hard to write something to upload the results to TFS should it? Well it's gonna be a little tricky; the web services in question aren't documented by Microsoft and they don't really like people reverse engineering their stuff. (Plus I've had a quick peek with Reflector and there is a lot of code to wade through...) Given that it's all web services why don't we arm ourselves with Fiddler and see what MSTest is doing exactly?
The first thing to notice is that MSTest adds build information to your .trx file. Try uploading an MSTest run then downloading the run and comparing the two .trx files; you'll notice it adds a element called Build just before the TestDefinitions element:
<Build flavor="Release" platform="Any CPU" />
This just contains the build properties you specified with the
/platform switches when calling MSTest. Simple enough to add.
The first web service call MSTest makes is a call to the
QueryBuilds method of
/Build/v2.0/BuildService.asmx to get some information about the build we identified with the
/publishbuild switches when calling MSTest. This service gives us (amongst other things) a build ID that we will need to call other web services later on.
The next step is a call to the
GetVersion method of
/Build/v1.0/PublishTestResultsBuildService2.asmx to get the version of the publish web service. I never used this information in my final application; I had the publish version hardcoded. If you were working with another version of TFS (I'm using 2008) then you might need to do something with this information. I've left the call in for completeness sake.
Now we can publish. Publishing involves three steps:
/Build/v1.0/PublishTestResultsBuildService2.asmxto tell TFS that we want to publish some test results. We give it the build ID we retrieved earlier and the run ID for the test. (A random GUID in the test result XML) The web service returns a directory on the build server where we should copy our test results to.
/Build/v1.0/PublishTestResultsBuildService2.asmxto tell TFS that we've copied the results across. We give it the build and run IDs along with the filename of the file we've copied.
And that's it. Pretty easy to do ourselves in a little application wouldn't you say? Which is exactly what I've done. You lucky people!
Feel free to download the source code and give it a try. It's a simple console application that you can call from your TFSBuild.proj file when doing an automated build. It also includes an updated version of the XSL from NUnit for Team Build that fixes a couple of problems, namely names for TestCases and names over 255 characters. (Which TFS really doesn't like and will happily fail when you upload tests without telling you why causing you to rip out half your hair and go insane. TFS can be a right tosser at times.) The arguments are as follows:
||No||Path to the NUnit output file.|
||No||Team Foundation project name, e.g. MyProject.|
||Yes||Platform. Defaults to "Any CPU".|
||Yes||Flavour. Defaults to "Release".|
||No||Name of the build to publish to, e.g. Release_20090709.5|
||Yes||Name of the test run output file. Defaults to "NUnitOutput.trx".|
||Yes||Path of the NUnitToMSTest.xslt file. Defaults to "NUnitToMSTest.xslt"|
I've used the excellent CommandLine Parser library to parse the arguments; I've used the library a few times now in private projects and recommend you give it a try.
Note that you'll also need to update the configuration file with the location of your TFS; I never bothered to write code to create the WCF configuration on the fly using a command line option...
So how do you go about using it with automated builds? Well I have a folder called Build in the root of my project. In that I have the TFSBuild.proj an a folder called NUnitTFS with the NUnitTFS.exe and associated files in. You can then simply call it from your build script using the
<Exec Command=""$(SolutionRoot)\Build\NUnitTFS\NUnitTFS.exe" -n "$(OutDir)NUnitOutput.xml" -t "$(TeamProject)" -b "$(BuildNumber)" -f "Release" -p "Any CPU" -x "$(SolutionRoot)\Build\NUnitTFS\NUnitToMSTest.xslt"" />
I'll leave it to you to wrap the task in a nice build step. Didn't expect the moon on a stick did you?
Note that I've only tested this using my company's setup; there is no guarantee it will work anywhere else! That's why I've included the (not very polished) source code, so you can tweak things if necessary. If you do fix any problems please drop me an email so I can update the code; I intend to publish it on CodePlex when I get a chance.