xUnit.net 1.0 RC2 Released

Today, we released xUnit.net 1.0 RC2.

Before I get into why it took us 10 weeks to go from RC1 to RC2, let's cover what's available in this release:

• We merged the xunit and xunitext projects. After some evaluation of our two-project system, we've decided to kill the xunitext project. We have already moved all the source over; later today, I'll be copying the work items from xunitext over into xunit, and then sometime this week we'll kill the xunitext project. The single Build 1223 download released today contains all the libraries from xunit and xunitext. The DLL names and namespace names haven't changed, so you should simply be able to drop them in and use them as is.
• We obsolesced most of the Assert methods which take user messages. The only ones we left are those on Assert.True and Assert.False, which tend to be catch-all asserts which might require documentation. We will be removing the obsolesced methods in 1.0 RTM, so please move your calls to the message-less variants.
• We added an MSBuild task. We are now using this MSBuild task in our automated builds, so you can look in xunit.tests.msbuild to see how it's used (there's also an MSBuild sample in the Samples zip file). One of the biggest drawbacks to using xunit.console from MSBuild was the lack of feedback about progress, since the console runner printed a single dot for each run test, but MSBuild waits for whole lines of output before displaying anything. Our MSBuild task prints the name of each test after it has been run: in "low priority" gray for passing tests, in "warning" yellow for skipped tests, and in "error" red for failing tests.
• We added Assert.IsAssignableFrom. Based on feedback, we decided that there was value in two different asserts which tested for exact type (Assert.IsType) vs. inheritance/implementation (Assert.IsAssignableFrom).
• We added an IUseFixture<T> sample. This sample is designed to illustrate the use of IUseFixture<T> as a replacement for test fixture setup and teardown. It attaches to a SQL Express database and populates some data into it, and then provides the SQL connection and data IDs to the tests to be consumed.
• The console and MSBuild runner output version information for xunit.dll and the full pathname of the assembly under test. It should be self explanatory why this is valuable information. :)
• We added the Executor and ExecutorWrapper classes. More about this in a minute.
• We fixed several bugs. See the release for a list.

If you find any issues or have any questions, please use our forums or bug tracker.

Why did it take so long?

A lot happened in the 10 or 11 weeks since we released RC1. We had intended the RC to really be a release candidate, but as more people have been using xUnit.net, we found that there were a lot of problems centered around the runners. To the point, since runners are linked against a specific version of xUnit.net, they were not capable of running tests for other versions.

This wasn't too big an issue with the console runner, as you tended to upgrade all your libraries at once, but it did come into play with runners where you couldn't necessarily match the runner version against the version of xUnit.net that you were using (most commonly this was a problem with the runners inside Visual Studio: the TD.NET and Resharper runners).

We decided that tackling version-resilient test runners was a problem worth solving for v1.0. We started down the path in the context of writing the new MSBuild task. We added a simple class to xunit.net called Executor, which acts as an intermediary between xUnit.net specific types (like PassedResult, FailedResult) and the runners. It does this by changing the communication mechanism to XML, so that the runners are not required to link against version-specific types.

The Executor class was not intended to be consumed directly, as this would link the runner against a specific version of xUnit.net. Instead, we added a new assembly named "xunit.runner.utility", and into this assembly we placed a class named ExecutorWrapper. This class provides an abstraction away from linking to xunit.dll directly. Runner authors can link to this DLL and use this class to help them run tests.

Constructing an instance of ExecutorWrapper requires an assembly filename, a configuration filename, and a flag as to whether the system should shadow copy the DLLs for it. When it's created, it automatically creates an AppDomain on behalf of the runner and loads xunit.dll into it. When ExecutorWrapper is disposed, it cleans up the AppDomain.

ExecutorWrapper today offers a single property -- XunitVersion -- and a single method -- RunAssembly. Using this, you can run all the tests in an assembly, as well as discover which version of xunit.dll was loaded into the AppDomain. The XML callback is done in terms of the XML nodes that are normally part of the xUnit.net XML output file. As each test is run, for example, the callback will result in a single "test" XML node; once all the tests in a class are run, the result is a "class" node; etc.

Both the console runner and the MSBuild task runner are using ExecutorWrapper now. You can see that they are linked against xunit.runner.utility and not xunit.dll. The MSBuild task is the simplest illustration of how to write a version-resilient runner. We also converted our acceptance tests to use ExecutorWrapper, as this provides better end-to-end coverage that simulates the use of an actual test runner.

Executor and ExecutorWrapper are still a work in progress. Today they are capable of supporting simple assembly-level runners. We will be finishing them to support more complex runners, including the existing TD.NET and Resharper runners, as well as interactive (GUI) runners. If you intend to consume the ExecutorWrapper classes because you are a runner author, we would be interested in hear about the needs you would have for this API. We'd like to get it more or less right the first time around. :)

Hopefully this will eventually resolve issues people have been having with runner versions not matching the exact xunit.dll version of the assembly under test.

Comments

[Jeff Brown]

As a runner author, I need to be able to enumerate the tests ahead of time. I also want to be able to provide my own IMethodInfo objects and such.

So RunAssembly won't be enough. I'm okay with that as long as it is still possible to achieve deeper integration through other APIs.

[Brad Wilson]

Right, that's exactly the kind of stuff we want to get done before 1.0 RTM. I think I'm going to mock up a simplified version of the NUnit GUI just to help figure out what those interfaces should look like.

We knew RunAssembly would only be suitable for the console and MSBuild runners. :)

[Gabe]

Good work, but where is the all important UI? Without that, I find it hard to leave NUnit.

[Brad Wilson]

We just started working on a GUI today, and checked in about an hour ago. It's only capable of running an assembly (no selection of "which tests to run"), but it is functional.

We're using it to help flesh out the Executor(Wrapper) API.

[Brad Wilson]

Jeff,

IMethodInfo and friends are not going to be supported. We looked at the problem today in the context of the Resharper runner, and decided it wasn't a problem worth tackling (for performance reasons, primarily, but also because of other technical challenges).

On the other hand, today we finished implementing RunClass, RunTest, and RunTests; we should be able to finish EnumerateTests soon and that will be our RC3 (with no planned changes for RTM other than critical bug fixes).

More information is available here:

http://bradwilson.typepad.com/blog/2008/04/xunitnets-resha.html