MrKWatkins

Making NUnit Play Nicely With Team Foundation Server

UPDATE! The code from this article has now been integrated into the NUnit for Team Build project. See NUnitTFS Integrated With NUnit For Team Build for more information.

My company has just started using Team Foundation Server. And whilst it is awesome in many ways it's support for testing frameworks other than MSTest is a bit rubbish. Well completely non-existent actually. Unless your test results are in the MSTest format then you're not going to be uploading them to TFS anytime soon. Fortunately others have already come across this problem... There is a project on CodePlex called NUnit for Team Build that is basically an XSL transform to take results from NUnit and transform them into MSTest's format.

However we still have to get those results into TFS. Which means a manual upload from Visual Studio or using MSTest with a few well chosen command line arguments. So if you want to automatically upload tests from your build server to TFS you'll need to install MSTest on your build server. MSTest isn't exactly and easy-to-install self contained tool so you're either gonna have to stop copying random assemblies around until it works or install Visual Studio on your build server. Bollocks to that!

Given that TFS is supposed to be all nice and web servicey it shouldn't be too hard to write something to upload the results to TFS should it? Well it's gonna be a little tricky; the web services in question aren't documented by Microsoft and they don't really like people reverse engineering their stuff. (Plus I've had a quick peek with Reflector and there is a lot of code to wade through...) Given that it's all web services why don't we arm ourselves with Fiddler and see what MSTest is doing exactly?

The first thing to notice is that MSTest adds build information to your .trx file. Try uploading an MSTest run then downloading the run and comparing the two .trx files; you'll notice it adds a element called Build just before the TestDefinitions element:

<Build flavor="Release" platform="Any CPU" />

This just contains the build properties you specified with the /flavor and /platform switches when calling MSTest. Simple enough to add.

The first web service call MSTest makes is a call to the QueryBuilds method of /Build/v2.0/BuildService.asmx to get some information about the build we identified with the /teamproject and /publishbuild switches when calling MSTest. This service gives us (amongst other things) a build ID that we will need to call other web services later on.

The next step is a call to the GetVersion method of /Build/v1.0/PublishTestResultsBuildService2.asmx to get the version of the publish web service. I never used this information in my final application; I had the publish version hardcoded. If you were working with another version of TFS (I'm using 2008) then you might need to do something with this information. I've left the call in for completeness sake.

Now we can publish. Publishing involves three steps:

1. A call to the PrepareToPublish method of /Build/v1.0/PublishTestResultsBuildService2.asmx to tell TFS that we want to publish some test results. We give it the build ID we retrieved earlier and the run ID for the test. (A random GUID in the test result XML) The web service returns a directory on the build server where we should copy our test results to.
2. Copy the .trx file to the directory we've just been given.
3. A call to the PublishRun method of /Build/v1.0/PublishTestResultsBuildService2.asmx to tell TFS that we've copied the results across. We give it the build and run IDs along with the filename of the file we've copied.

And that's it. Pretty easy to do ourselves in a little application wouldn't you say? Which is exactly what I've done. You lucky people!

Feel free to download the source code and give it a try. It's a simple console application that you can call from your TFSBuild.proj file when doing an automated build. It also includes an updated version of the XSL from NUnit for Team Build that fixes a couple of problems, namely names for TestCases and names over 255 characters. (Which TFS really doesn't like and will happily fail when you upload tests without telling you why causing you to rip out half your hair and go insane. TFS can be a right tosser at times.) The arguments are as follows:

I've used the excellent CommandLine Parser library to parse the arguments; I've used the library a few times now in private projects and recommend you give it a try.

Note that you'll also need to update the configuration file with the location of your TFS; I never bothered to write code to create the WCF configuration on the fly using a command line option...

So how do you go about using it with automated builds? Well I have a folder called Build in the root of my project. In that I have the TFSBuild.proj an a folder called NUnitTFS with the NUnitTFS.exe and associated files in. You can then simply call it from your build script using the Exec command:

<Exec Command="&quot;$(SolutionRoot)\Build\NUnitTFS\NUnitTFS.exe&quot; -n &quot;$(OutDir)NUnitOutput.xml&quot; -t &quot;$(TeamProject)&quot; -b &quot;$(BuildNumber)&quot; -f &quot;Release&quot; -p &quot;Any CPU&quot; -x &quot;$(SolutionRoot)\Build\NUnitTFS\NUnitToMSTest.xslt&quot;" />

I'll leave it to you to wrap the task in a nice build step. Didn't expect the moon on a stick did you?

Note that I've only tested this using my company's setup; there is no guarantee it will work anywhere else! That's why I've included the (not very polished) source code, so you can tweak things if necessary. If you do fix any problems please drop me an email so I can update the code; I intend to publish it on CodePlex when I get a chance.

Fluent Argument Validation Part 5 - Version One

So that's it for fluent argument validation for the time being. I've got enough done to spit out version one, which is more than enough for my current needs. Afraid you don't get the source just yet... I plan to host the source on CodePlex eventually but I might just wait for version two for that...

"Version two?" I hear you say. "But what is wrong with version one?" Nothing, really. It suits me quite well. However whilst writing these articles (I'd actually written the library several months before...) I've thought of several ways to improve the thing...

Improve Number Chains

I touched on this one in part three. Currently the following is perfectly acceptible:

public static void SomeMethod(int number)
{
    Validate.Argument(number, "number")
        .IsGreaterThan(5)
        .IsGreaterThan(3)
        .IsGreaterThan(3);

Now are you ever going to validate a number is greater than another number twice? Surely you'd just validate it against the higher of the two. Therefore should I allow more than one call to IsGreaterThan in a validation chain?

Okay, you might be validating against variables and therefore have to validate against two variables to check they are greater than both. But in this case I think we need a different method anyways because one would want the exception method to express that the argument has to be greater than the variable rather than the current value of the variable. For example the following code:

public int SomeProperty
{
    get
    {
        return 5;
    }
}

public static void SomeMethod(int number)
{
    Validate.Argument(number, "number")
        .IsGreaterThan(SomeProperty);

would give the error message "Value must be greater than 5." A more useful message would be "Value must be greater than SomeProperty.". So perhaps we need another set of methods that allows you to specify something (other than a constant) to validate against?

Improve The Fluent 'Sentences'

The English made by reading the Fluent validation could be a bit better. For example the following:

Validate.Argument(value, "value")
    .IsNotNull()
    .IsNotEmpty()
    .StartsWith("Something");

would sound better as:

Validate.Argument(value, "value")
    .IsNotNull()
    .OrEmpty()
    .AndStartsWith("Something");

Default Tolerance

Unless you specify a tolerance then currently the validation methods for imprecise numbers don't use a tolerance which basically means they're not very useful; in general they will fail for some valid cases. NUnit's constraint model, which is a great example of a good fluent interface, has a default tolerance to use in imprecise comparisons. I think my library should do that too...

More Than Arguments

I'd like to add more validation than just arguments, e.g. type parameter validation, IO validation, XML validation, etc..

Anyways...

I'm sure I'll think of lots more for version two. But in the meantime version one is finished. Have a look at parts one, two, three and four if you haven't already. Feel free to download the library and use it in your own projects. If you do find the library useful then please ping me an email with details of the project you used it on and any ideas you have for improving it. And if you use it to make loads of cash then feel free to donate. No, in fact, feel compelled.

Fluent Argument Validation Part 4 - Enums

I've been creating a fluent argument validation library; see parts one, two and three if you need to catch up.

Generics are brilliant. Wonderful things. But there are a couple of things about them that annoy me. For example, why can't I do the following?

public class SomeClass
{
    public void SomeMethod<T>(T value) where T : class...
    
    public void SomeMethod<T>(T value) where T : struct...
}

I.e. use a different method overload depending on whether my type is a reference or a value type? A type cannot be both a reference and value type at the same time so surely we have two different method signatures? Yes, I know there are good reasons why not. But still... I want to be able to do this!

Another one that annoys me (and is relevant to this post!) is not being able to to specify my type must be an enum, i.e. something like:

public void Argument<T>(T value) where T : enum

I want to be able to create a validation method that checks if an enum value is defined. But that the same time I don't want to have people pass any value type they like in. Whilst I can quite happily check at runtime that a type is an enum it just feels like something I should be doing at compile time...

Of course what makes this really annoying is that you can specify an enum as a type constraint in IL! Whilst I'm quite happy for C# to restrict me in some ways (e.g. in IL you can throw an object of any type whereas C# restricts it to Exception and sub-classes) this one I'm not happy about. Mainly because it affects me personally, I'll admit. But is that not a good enough reason?

Guess I could rewrite the thing in a language that does allow it. Nah. I like C#. I could write my own language that is an extension of C#. Whilst I'd like to do that one day I think it might be overkill for this... What if I decompiled the assembly the C# compiler generated and changed the offending line? That might work... C# would have to respect the type constraint even if it cannot be expressed in C#. Let's try that...

Turns out it works too. Almost. Turns out I bump into my first generic annoyance, that of methods with the same signature but different type constraints. So I cannot do the following pseudo-C#:

public class Validate
{
    public void Argument<T>(T argument, string parameterName) where T : class...
    
    public void Argument<T>(T argument, string parameterName) where T : enum...
}

Guess I'll just have to live with that one and create an EnumArgument method on my Validate class. So what IL do I need to rewrite? Well firstly I'll have to rewrite the initial generic method on my Validate class. Luckily IL method signatures are fairly readable. The C# method signature:

public static IEnum<T> EnumArgument<T>(T argument, string parameterName) where T : struct

Becomes:

.method public hidebysig static class KWatkins.Validation.Argument.IEnum`1<!!T> 
        EnumArgument<valuetype .ctor ([mscorlib]System.ValueType) T>(!!T argument, string parameterName) cil managed

The type constraints are quite complex compared to C#. The type has to be a value type, has to have a parameterless constructor and has to inherit from System.ValueType. C# hides the second two things from us as in C# we can infer that if something is a value type then it must have a parameterless constructor and inherit from System.ValueType. Anyways its simple enough to change this to have an enum type constraint:

.method public hidebysig static class KWatkins.Validation.Argument.IEnum`1<!!T> 
        EnumArgument<valuetype .ctor ([mscorlib]System.Enum) T>(!!T argument, string parameterName) cil managed

Done. Change ValueType to Enum and walk away. Nice and simple. I also have to rewrite the IL on the class that actually does the validation but that's basically the same replacement as above.

How do I rewrite the IL though? Luckily .NET comes with a tool called ildasm that decompiles any given assembly into IL. And then Visual Studio comes with a tool called ilasm that compiles IL into an assembly. Therefore I can create a command line application that can run as a post build command for my validation assembly. It can call ildasm to disassembly the assembly, do some crude string replacement on the IL and then pass it back into ilasm to turn the thing back into an assembly.

And that's that. I now have a Validate.Argument<T> method that will only accept an enum for it's type parameter. And that's pretty much it for the vaguely interesting stuff in my argument validation library too. So I'll wrap things up for version one of the library in the next post.

Fluent Argument Validation Part 3 - Numbers

In parts one and two I discussed the approach I'm going to make to create a fluent argument validation library. Now the implementation.

Well it turns out most of it is pretty straightforward. And a little boring. There are a couple of things that warrent comment, the first being numbers. There are a lot of number types in .NET. There are precise types; byte, 3 sizes of int and decimal. Plus of course their evil partners the unsigned ints and the signed byte. And then on top of that you have imprecise types, the single and double. For the sake of completeness I'm going to add validation methods for all of them. Which means a lot of code to write; numeric types don't have a common interface we could exploit, or anything like type classes we can operate on. How can we be lazy and write less code?

By auto-generating it of course. One of the best kept secrets in Visual Studio is the Text Template Transformation Toolkit, or just T4 for short. T4 is a code generator that builds code from templates. I first heard about it in a blog post by Scott Hanselman a while ago and it's nice to finally have a chance to use it!

But before we get generating, what sort of interfaces/methods will we need? Basic range checking, greater than, less than, etc., will be needed for all number types. I find myself having to validate numbers are positive a fair bit so I'm going to add quick methods to check for positive and negative. These are only necessary for the signed types; there is no point in checking to see if a uint is negative because it cannot be. It sounds like we immediately have two interfaces, one for signed types and one for unsigned. Both contain greater/less methods but the signed one also contains positive/negative methods

As discussed in part one one of the problems I had with some of the other fluent interfaces available is that they allow you to make non-sensical and/or pointless calls, e.g:

public static void SomeMethod(int number)
{
    Validate.Argument(number, "number")
        .IsPositive()
        .IsNegative()
        .IsNegative();

A number cannot be both positive and negative, and there is no point in checking a number is negative twice. Therefore my inteface needs to return the interface for validating unsigned types after a call to positive or negative. (Even for singles and doubles! So if I talk about unsigned doubles I am making sense. Honest.) Ideally we would also like to stop calls such as:

public static void SomeOtherMethod(int number)
{
    Validate.Argument(number, "number")
        .IsGreaterThan(5)
        .IsGreaterThan(3)
        .IsGreaterThan(3);

There is no point in checking if something is over both 3 and 5 as just checking 5 would be enough. Similarly checking 3 twice is not necessary. Unlike the positive/negative we cannot enforce this at compile time so I'm going to have to let this one slide. (Although I will come back to this in part five)

The only other methods I'm going to implement are for the imprecise types only. These will be overloads of all the methods so far but with a tolerance to allow us to check two imprecise numbers follow the condition within a given amount.

Now how does T4 come into play? Well I can define generic interfaces for all the methods but I will need concrete classes for each numeric type. These concrete classes basically have the same implementation (operator comparisons such as >, <=, etc.) so instead of duplicating the same code over and over I'll create a template that generates the methods for each numeric type. If we can define the various properties of our numeric types (e.g. signed/unsigned, precise/imprecise, etc.) then our template can query those definitions to work out what interfaces and methods need implemented.

Definitions of the numeric types could come in handy in future templates. Therefore I defined the details of the numeric types in a file called NumericTypeDefinitions.ttinclude, which looks something like:

<#
    var numericTypes = new []
    {
        new
        {
            Name = "Byte",
            Zero = "0",
            CLSCompliant = true,
            Signed = false,
            Precise = true
        },
        new
        {
            Name = "Decimal",
            Zero = "0m",
            CLSCompliant = true,
            Signed = true,
            Precise = true
        },
        ...

Sadly the syntax highlighting is mine; rather annoyingly VS2008 doesn't highlight .tt or .ttinclude files... We can include this file in our actual T4 template. Basic templates just have a blob of code. The include means our definitions will be added to this blob. (You can create custom generator classes for more complex templates) Our T4 file will look something like this: (the ASP.NETters amongst you will probably recognise some of the syntax)

<#@ template language="C#v3.5" #>
<#@ output extension="cs" #>
<#@ assembly name="System.Core.dll" #>
<#@ import namespace="System.Linq" #>
<#@ include file="NumericTypeDefinitions.ttinclude" #>
using System;
using KWatkins.Validation.Argument;

namespace KWatkins.Validation
{
    public static partial class Validate
    {
<#
    // Partial part of Validate first.
    foreach(var type in numericTypes.Where(t => t.Precise))
    {
        if (!type.CLSCompliant)
        {
#>
        [CLSCompliant(false)]
<#
        }
#>
        public static <#= GetInterfaceType(type.Signed, type.Name) #> Argument(<#= type.Name #> argument, string parameterName)
        {
            return new <#= type.Name #>Validation(argument, parameterName);
        }
<#
    }
#>
    }
}
...

What's happening here? Well we're basically adding the method Argument(T argument, string parameterName) to our Validate class. The directives at the start of the template define what we need for the code in our template, e.g. the language our template uses, the assemblies and namespaces we want to use, etc. Then the template starts; the text in grey is the text that will appear in the generated .cs file. In the snippet above I am looping through all the precise numeric types. If they are not CLS compliant then I add a CLSCompliantAttribute to the method. I then spit out the method definition, which returns a validation class of the appropriate type, casted to an interface of the appropriate type. I'll spare you the rest of the details; you can have a look at the source when I've finished.

Now the tests! How on Earth are we gonna test generated code? We could I suppose test the generator itself, i.e. test the output created by our T4 template is correct. But this doesn't actually test our final validation code! We could write a test for each generated method. But that would take ages besides being a bit rubbish. We could auto-generate our tests using a T4 template. Not a bad idea, but I personally prefer to have my test code as different to my code being tested as possible. So how to avoid writing a thousand separate tests for all our generated methods?

Use NUnit 2.5's excellent new generic test fixtures! NUnit 2.5 allows you to use generic classes for your test fixtures. You can then specify multiple TestFixtureAttributes each with a different generic type. The fixture will be executed multiple times, one for each TestFixtureAttribute. Add a splash of reflection to create our T4 generated validation class and we can write test fixtures like this:

[TestFixture(typeof(Byte))]
[TestFixture(typeof(UInt16))]
[TestFixture(typeof(UInt32))]
[TestFixture(typeof(UInt64))]
public class UnsignedPreciseNumberTests<TNumber> : NumberTests<TNumber>
    where TNumber : struct
{
    [TestCase("5", "TestParameterName", "10",
        ExpectedException = typeof(ArgumentOutOfRangeException),
        ExpectedMessage = "Value must be greater than 10.\r\nParameter name: TestParameterName\r\nActual value was 5.")]
    [TestCase("5", "TestParameterName", "5",
        ExpectedException = typeof(ArgumentOutOfRangeException),
        ExpectedMessage = "Value must be greater than 5.\r\nParameter name: TestParameterName\r\nActual value was 5.")]
    [TestCase("5", "TestParameterName", "0")]
    public void IsGreaterThan(string argument, string parameterName, string minimum)
    {
        IUnsignedPreciseNumber<TNumber> validator = IUnsignedPreciseNumber<TNumber>)CreateValidationType(argument, parameterName);
        validator.IsGreaterThan(ConvertFromString(minimum));
    }
    ...

Here we are testing the IsGreaterThan method for the unsigned, precise number types. The test method gets an instance of the validation type using the CreateValidationType which does some reflective magic to create the correct type. It then calls the method using the values passed into the method via NUnit 2.5's TestCaseAttribute which allows you to run the same test multiple times with different values.

So that is the numbers done. I urge you to have a play with T4 and NUnit 2.5; both are fantastic tools to have in your arsenal. In the next post I will get irritated with enumerations...