MrKWatkins

NUnitTFS Integrated With NUnit For Team Build

In my last post I talked about the CodePlex project called NUnit for Team Build that transforms the output from NUnit into a format that Team Foundation Server can understand. My article then discussed how to upload these results to TFS without using the MSTest program.

Well Richard Banks, the owner of the NUnit for Team Build project, got in touch and the source from my last article has been integrated with his work. Head over to the CodePlex site now to get the latest version! A big thanks to Richard for actually uploading the code for me; although I meant to I got distracted by the cricket...

The project doesn't end here though... I'm currently working on improving the code base, mainly by getting some unit tests in there. Seems to me a project all about unit tests should probably have some of it's own... I'm also trying to transform and upload NCover 1.5.8 output, although that is proving a little trickier. Will keep you posted...

Making NUnit Play Nicely With Team Foundation Server

UPDATE! The code from this article has now been integrated into the NUnit for Team Build project. See NUnitTFS Integrated With NUnit For Team Build for more information.

My company has just started using Team Foundation Server. And whilst it is awesome in many ways it's support for testing frameworks other than MSTest is a bit rubbish. Well completely non-existent actually. Unless your test results are in the MSTest format then you're not going to be uploading them to TFS anytime soon. Fortunately others have already come across this problem... There is a project on CodePlex called NUnit for Team Build that is basically an XSL transform to take results from NUnit and transform them into MSTest's format.

However we still have to get those results into TFS. Which means a manual upload from Visual Studio or using MSTest with a few well chosen command line arguments. So if you want to automatically upload tests from your build server to TFS you'll need to install MSTest on your build server. MSTest isn't exactly and easy-to-install self contained tool so you're either gonna have to stop copying random assemblies around until it works or install Visual Studio on your build server. Bollocks to that!

Given that TFS is supposed to be all nice and web servicey it shouldn't be too hard to write something to upload the results to TFS should it? Well it's gonna be a little tricky; the web services in question aren't documented by Microsoft and they don't really like people reverse engineering their stuff. (Plus I've had a quick peek with Reflector and there is a lot of code to wade through...) Given that it's all web services why don't we arm ourselves with Fiddler and see what MSTest is doing exactly?

The first thing to notice is that MSTest adds build information to your .trx file. Try uploading an MSTest run then downloading the run and comparing the two .trx files; you'll notice it adds a element called Build just before the TestDefinitions element:

<Build flavor="Release" platform="Any CPU" />

This just contains the build properties you specified with the /flavor and /platform switches when calling MSTest. Simple enough to add.

The first web service call MSTest makes is a call to the QueryBuilds method of /Build/v2.0/BuildService.asmx to get some information about the build we identified with the /teamproject and /publishbuild switches when calling MSTest. This service gives us (amongst other things) a build ID that we will need to call other web services later on.

The next step is a call to the GetVersion method of /Build/v1.0/PublishTestResultsBuildService2.asmx to get the version of the publish web service. I never used this information in my final application; I had the publish version hardcoded. If you were working with another version of TFS (I'm using 2008) then you might need to do something with this information. I've left the call in for completeness sake.

Now we can publish. Publishing involves three steps:

1. A call to the PrepareToPublish method of /Build/v1.0/PublishTestResultsBuildService2.asmx to tell TFS that we want to publish some test results. We give it the build ID we retrieved earlier and the run ID for the test. (A random GUID in the test result XML) The web service returns a directory on the build server where we should copy our test results to.
2. Copy the .trx file to the directory we've just been given.
3. A call to the PublishRun method of /Build/v1.0/PublishTestResultsBuildService2.asmx to tell TFS that we've copied the results across. We give it the build and run IDs along with the filename of the file we've copied.

And that's it. Pretty easy to do ourselves in a little application wouldn't you say? Which is exactly what I've done. You lucky people!

Feel free to download the source code and give it a try. It's a simple console application that you can call from your TFSBuild.proj file when doing an automated build. It also includes an updated version of the XSL from NUnit for Team Build that fixes a couple of problems, namely names for TestCases and names over 255 characters. (Which TFS really doesn't like and will happily fail when you upload tests without telling you why causing you to rip out half your hair and go insane. TFS can be a right tosser at times.) The arguments are as follows:

I've used the excellent CommandLine Parser library to parse the arguments; I've used the library a few times now in private projects and recommend you give it a try.

Note that you'll also need to update the configuration file with the location of your TFS; I never bothered to write code to create the WCF configuration on the fly using a command line option...

So how do you go about using it with automated builds? Well I have a folder called Build in the root of my project. In that I have the TFSBuild.proj an a folder called NUnitTFS with the NUnitTFS.exe and associated files in. You can then simply call it from your build script using the Exec command:

<Exec Command="&quot;$(SolutionRoot)\Build\NUnitTFS\NUnitTFS.exe&quot; -n &quot;$(OutDir)NUnitOutput.xml&quot; -t &quot;$(TeamProject)&quot; -b &quot;$(BuildNumber)&quot; -f &quot;Release&quot; -p &quot;Any CPU&quot; -x &quot;$(SolutionRoot)\Build\NUnitTFS\NUnitToMSTest.xslt&quot;" />

I'll leave it to you to wrap the task in a nice build step. Didn't expect the moon on a stick did you?

Note that I've only tested this using my company's setup; there is no guarantee it will work anywhere else! That's why I've included the (not very polished) source code, so you can tweak things if necessary. If you do fix any problems please drop me an email so I can update the code; I intend to publish it on CodePlex when I get a chance.

Fluent Argument Validation Part 3 - Numbers

In parts one and two I discussed the approach I'm going to make to create a fluent argument validation library. Now the implementation.

Well it turns out most of it is pretty straightforward. And a little boring. There are a couple of things that warrent comment, the first being numbers. There are a lot of number types in .NET. There are precise types; byte, 3 sizes of int and decimal. Plus of course their evil partners the unsigned ints and the signed byte. And then on top of that you have imprecise types, the single and double. For the sake of completeness I'm going to add validation methods for all of them. Which means a lot of code to write; numeric types don't have a common interface we could exploit, or anything like type classes we can operate on. How can we be lazy and write less code?

By auto-generating it of course. One of the best kept secrets in Visual Studio is the Text Template Transformation Toolkit, or just T4 for short. T4 is a code generator that builds code from templates. I first heard about it in a blog post by Scott Hanselman a while ago and it's nice to finally have a chance to use it!

But before we get generating, what sort of interfaces/methods will we need? Basic range checking, greater than, less than, etc., will be needed for all number types. I find myself having to validate numbers are positive a fair bit so I'm going to add quick methods to check for positive and negative. These are only necessary for the signed types; there is no point in checking to see if a uint is negative because it cannot be. It sounds like we immediately have two interfaces, one for signed types and one for unsigned. Both contain greater/less methods but the signed one also contains positive/negative methods

As discussed in part one one of the problems I had with some of the other fluent interfaces available is that they allow you to make non-sensical and/or pointless calls, e.g:

public static void SomeMethod(int number)
{
    Validate.Argument(number, "number")
        .IsPositive()
        .IsNegative()
        .IsNegative();

A number cannot be both positive and negative, and there is no point in checking a number is negative twice. Therefore my inteface needs to return the interface for validating unsigned types after a call to positive or negative. (Even for singles and doubles! So if I talk about unsigned doubles I am making sense. Honest.) Ideally we would also like to stop calls such as:

public static void SomeOtherMethod(int number)
{
    Validate.Argument(number, "number")
        .IsGreaterThan(5)
        .IsGreaterThan(3)
        .IsGreaterThan(3);

There is no point in checking if something is over both 3 and 5 as just checking 5 would be enough. Similarly checking 3 twice is not necessary. Unlike the positive/negative we cannot enforce this at compile time so I'm going to have to let this one slide. (Although I will come back to this in part five)

The only other methods I'm going to implement are for the imprecise types only. These will be overloads of all the methods so far but with a tolerance to allow us to check two imprecise numbers follow the condition within a given amount.

Now how does T4 come into play? Well I can define generic interfaces for all the methods but I will need concrete classes for each numeric type. These concrete classes basically have the same implementation (operator comparisons such as >, <=, etc.) so instead of duplicating the same code over and over I'll create a template that generates the methods for each numeric type. If we can define the various properties of our numeric types (e.g. signed/unsigned, precise/imprecise, etc.) then our template can query those definitions to work out what interfaces and methods need implemented.

Definitions of the numeric types could come in handy in future templates. Therefore I defined the details of the numeric types in a file called NumericTypeDefinitions.ttinclude, which looks something like:

<#
    var numericTypes = new []
    {
        new
        {
            Name = "Byte",
            Zero = "0",
            CLSCompliant = true,
            Signed = false,
            Precise = true
        },
        new
        {
            Name = "Decimal",
            Zero = "0m",
            CLSCompliant = true,
            Signed = true,
            Precise = true
        },
        ...

Sadly the syntax highlighting is mine; rather annoyingly VS2008 doesn't highlight .tt or .ttinclude files... We can include this file in our actual T4 template. Basic templates just have a blob of code. The include means our definitions will be added to this blob. (You can create custom generator classes for more complex templates) Our T4 file will look something like this: (the ASP.NETters amongst you will probably recognise some of the syntax)

<#@ template language="C#v3.5" #>
<#@ output extension="cs" #>
<#@ assembly name="System.Core.dll" #>
<#@ import namespace="System.Linq" #>
<#@ include file="NumericTypeDefinitions.ttinclude" #>
using System;
using KWatkins.Validation.Argument;

namespace KWatkins.Validation
{
    public static partial class Validate
    {
<#
    // Partial part of Validate first.
    foreach(var type in numericTypes.Where(t => t.Precise))
    {
        if (!type.CLSCompliant)
        {
#>
        [CLSCompliant(false)]
<#
        }
#>
        public static <#= GetInterfaceType(type.Signed, type.Name) #> Argument(<#= type.Name #> argument, string parameterName)
        {
            return new <#= type.Name #>Validation(argument, parameterName);
        }
<#
    }
#>
    }
}
...

What's happening here? Well we're basically adding the method Argument(T argument, string parameterName) to our Validate class. The directives at the start of the template define what we need for the code in our template, e.g. the language our template uses, the assemblies and namespaces we want to use, etc. Then the template starts; the text in grey is the text that will appear in the generated .cs file. In the snippet above I am looping through all the precise numeric types. If they are not CLS compliant then I add a CLSCompliantAttribute to the method. I then spit out the method definition, which returns a validation class of the appropriate type, casted to an interface of the appropriate type. I'll spare you the rest of the details; you can have a look at the source when I've finished.

Now the tests! How on Earth are we gonna test generated code? We could I suppose test the generator itself, i.e. test the output created by our T4 template is correct. But this doesn't actually test our final validation code! We could write a test for each generated method. But that would take ages besides being a bit rubbish. We could auto-generate our tests using a T4 template. Not a bad idea, but I personally prefer to have my test code as different to my code being tested as possible. So how to avoid writing a thousand separate tests for all our generated methods?

Use NUnit 2.5's excellent new generic test fixtures! NUnit 2.5 allows you to use generic classes for your test fixtures. You can then specify multiple TestFixtureAttributes each with a different generic type. The fixture will be executed multiple times, one for each TestFixtureAttribute. Add a splash of reflection to create our T4 generated validation class and we can write test fixtures like this:

[TestFixture(typeof(Byte))]
[TestFixture(typeof(UInt16))]
[TestFixture(typeof(UInt32))]
[TestFixture(typeof(UInt64))]
public class UnsignedPreciseNumberTests<TNumber> : NumberTests<TNumber>
    where TNumber : struct
{
    [TestCase("5", "TestParameterName", "10",
        ExpectedException = typeof(ArgumentOutOfRangeException),
        ExpectedMessage = "Value must be greater than 10.\r\nParameter name: TestParameterName\r\nActual value was 5.")]
    [TestCase("5", "TestParameterName", "5",
        ExpectedException = typeof(ArgumentOutOfRangeException),
        ExpectedMessage = "Value must be greater than 5.\r\nParameter name: TestParameterName\r\nActual value was 5.")]
    [TestCase("5", "TestParameterName", "0")]
    public void IsGreaterThan(string argument, string parameterName, string minimum)
    {
        IUnsignedPreciseNumber<TNumber> validator = IUnsignedPreciseNumber<TNumber>)CreateValidationType(argument, parameterName);
        validator.IsGreaterThan(ConvertFromString(minimum));
    }
    ...

Here we are testing the IsGreaterThan method for the unsigned, precise number types. The test method gets an instance of the validation type using the CreateValidationType which does some reflective magic to create the correct type. It then calls the method using the values passed into the method via NUnit 2.5's TestCaseAttribute which allows you to run the same test multiple times with different values.

So that is the numbers done. I urge you to have a play with T4 and NUnit 2.5; both are fantastic tools to have in your arsenal. In the next post I will get irritated with enumerations...