Yesterday we pushed the shiny red button to send you all the bits for xUnit.net 1.6. It turns out there were some interesting technical challenges behind the scenes, so after I do the obligatory feature list, I'll talk about all the work we did (and sometimes undid) along the way.
New Features
We added a new assertion: Assert.Single(). It takes a collection, and ensures that there is a one and only one item in it. It's a little bit like Assert.Empty(). The additional value of Assert.Single() is that, after success, it returns that single item to you (correctly typed if you're using a strongly typed collection). We wrote a lot of tests that looked like this (using LINQ syntax from .NET 3.5):
Assert.Equal(1, myCollection.Count); MyItem item = myCollection.Single();
Which now become:
MyItem item = Assert.Single(myCollection);
We added support for .NET 4. This includes support that allows your unit test projects to target the .NET 4 Client Profile. To do this, we had to remove a dependency we had in xunit.dll on System.Web; more about that in the next section.
We've shipped two new console runners (xunit.console.clr4.exe and xunit.console.clr4.x86.exe) as well as two new GUI runners (xunit.gui.clr4.exe and xunit.gui.clr4.x86.exe). The existing runners are still linked against CLR 2 and usable for unit tests which target .NET 2.0, 3.0, and 3.5. There's more about this decision in the next section as well.
We added support for MVC 2, in all of VS 2008 SP1, VS 2010, and VWD 2010 (sorry, we decided not to include VWD 2008 SP1, since VWD 2010 is a free product). The templates, for both MVC 1.0 and MVC 2, all now include automatic TestDriven.net support when using TD.NET 2.24 or later. Registration is no longer required to use TD.NET; it just works out of the box!
We extended support to the console runner to consume .xunit project files. Along the way of cleaning up that code, we also did an overhaul of the way output works from the console runner... no more dots! Who was counting the dots anyway? :)
We've deprecated a few assertion APIs that currently take IComparer<T>, but actually do equality comparison. Instead, we've added new versions which take IEqualityComparer<T>. The old verisons are still available, but they will be removed from the next build of xUnit.net, so please take the time to convert your assertions over to the new APIs. There are a couple assertions that continue to use IComparer<T> when it's appropriate (for example, Assert.InRange()).
Of course, we fixed a bunch of bugs. Check out the Issue Tracker for a full list of things we fixed.
Technical Goo
We approached a couple substantial technical problems for this release. One we got fixed how we wanted it to, and one we didn't. Both stories are educational.
Supporting the .NET Client Profile
We had a request from both Microsoft employees and customers for us to support the .NET 4 Client Profile.
xUnit.net has always run tests in a separate AppDomain, to ensure that the tests are isolated from one another as well as from the test runner. When you run in separate AppDomains, you need a way to communicate across between the two sets of code. What you need is a bit of code that's shared between the two AppDomains. For us, we needed to ask xunit.dll to run tests for us, and get status back about what's going on with those test runs. The runner would also need to let xunit.dll know whether it should cancel the run or continue.
To complicate the matter, we didn't want to load xunit.dll into the AppDomain of the test runner, because we want our runners to be version independent. This means with the GUI runner, you can load tests from various versions of xunit.dll, and it all just works. So whatever shared code we were going to have between the two AppDomains had to be something we could reuse from the .NET Framework itself.
In xUnit.net 1.0, that bit of shared code was ICallbackEventHandler. It was basically perfectly defined for our needs: the version independent runner would implement this, and xunit.dll would call RaiseCallbackEvent for each testing event (starting a test, finishing a test, etc.) and then call GetCallbackResult to find out if it should continue.
This worked great, until someone complained that their unit tests couldn't run in the .NET 4 Client Profile, because System.Web.dll wasn't in the client profile. Oops!
We set about looking for an alternative, when Scott found IMessageSink. Yes, it's a little strange that we're using the remoting infrastructure inside of itself, but it seemed to work fine. We could stuff the notifications and responses inside the dictionary of the message, and all was well and good.
So that's all well and good, until you remember that the idea behind version independent runners is to ensure that any runner could run any tests written against any version of xUnit.net. If we made this change -- which would affect both xunit.dll as well as xunit.runner.utility.dll -- then we've broken backward and forward compatibility.
In order to fix xunit.runner.utility.dll, we needed it to behave differently based on the version of xunit.dll that the test library is running against: for 1.5 and earlier, the callback object implements ICallbackEventHandler, and for 1.6 and later, it implements IMessageSink. Additionally, we can't directly link against System.Web.dll when we need to support xunit.dll from 1.5 or earlier, so we stole a trick from MVC 2's MvcHtmlString class to dynamically implement the interface at runtime.
In order to fix xunit.dll, we needed it to be able to accept callback objects that implement either interface. We can explicitly cast it to IMessageSink to see if it supports that, but in the case of ICallbackEventHandler, we need to just do runtime reflection in order to prevent the hard link against System.Web.dll.
Complicated as it sounds, it does all work. :) And we anticipate this should be the only time we'll ever need to do it, so it was worth a little extra effort not to kill the backward & forward compatibility story for any custom-built runners.
If you want to look at the code that does all the dirty work, you should see ExecutorWrapper and DynamicTypeGenerator on the xunit.runner.utility.dll side, and ExecutorCallback on the xunit.dll side.
The Failed External Runner Experiment
We've never been happy with the extra executables for the console and GUI runner (to force running in 32-bit mode), and the addition of .NET 4 was just going to double our unhappiness. Additionally, when you're using a runner that's in a DLL rather than a standalone EXE, you're stuck with whatever version of the .NET framework that your hosting process chose for you.
We implemented a potential solution to this in the form of external runner executables. We embedded the executables into xunit.runner.utility.dll, and extracted them automatically when running your tests. We would allow you to choose whether the tests ran in the native bitness of the machine or in x86 32-bit mode, as well as choosing which version of the CLR to run in (2 or 4). Aside from some oddities that I'm sure we could've worked around, the solution actually worked really well.
Until you tried to debug.
The problem is, you'd attach the debugger to xunit.console or xunit.gui (or even the debugging support in TD.NET or Resharper), and it wouldn't include the actual tests, because they lived in an entirely new process. There is no way that we could find to programmatically say "if I am being debugged, then please also debug my child process". The developer can choose to turn this feature on in WinDBG, but nothing exists in Visual Studio like it, and even asking the developer to be aware that they needed to turn this feature on seemed like a showstopper.
So in the end, we tossed the solution away. We considered using COM to get the new CLR 2 and CLR 4 side-by-side feature working, but with it only being half of the solution (still needing separate executables for 32- vs. 64-bit) we decided it wasn't worth the extra effort this time around. We also only ever anticipate supporting two versions of the CLR at any given time; by the time a new CLR comes around, we'll retire CLR 2 support, if we hadn't already.
