Brad Wilson

xUnit.net 1.0 RC3 Released Today

We've finally finished off all the work we wanted to achieve with the version-independent runner support, so we cut a new release candidate today. There were a few bugs fixed, and we added support for CruiseControl.net and ASP.NET MVC Preview 2, but otherwise this release should function more or less like the RC2 Refresh.

We believe we are now v1 feature complete.

At this point, we will only be fixing critical bugs. We're especially looking for feedback from runner authors on the functionality of ExecutorWrapper. If you're looking for examples on how to use ExecutorWrapper, the best places to start are the source code for the MSBuild task and the unit tests, although all the runners (except the Resharper runner, as I talked about yesterday) are using ExecutorWrapper now. In fact, all of our dynamic-compilation acceptance tests are also use ExecutorWrapper.

Barring any major bugs found, we're planning to release v1 at the end of the month.

xUnit.net's Resharper Runner - No Version Independence

Our version independent runner scheme has served us well so far: all of our runners have converted over to it except for the Resharper runner. Jim and I intended to tackle the Resharper runner today and make it version independent as well, but ran into some stumbling blocks that caused us to postpone the work indefinitely.

The API for the version independent runners is relatively simple, and everything is communicated in terms of simple strings and XML. I explained a bit about it in my previous post about RC2, so I won't re-cover old ground.

The problem with Resharper that's different from all the other runners is that it operates in a mode where there is no assembly file. You are asked to answer questions about whether classes contain tests based on the source code that's in the editor, not the bits that are on the disk. This makes perfect sense: they want to provide "run this" chicklets as soon as you add a test to your assembly.

Unfortunately, our API is based on assemblies on the disk. Most of the decision making is done on your behalf; you're just asking to run assemblies, classes, or tests based on filenames and fully qualified type and method names.

In order to support Resharper, we would essentially have to serialize (potentially incomplete) type information into XML and pass it across an app domain boundary to get it inspected in order to make decisions about whether something is a test or not. Aside from the obvious performance implications, we were also concerned with the tremendous complexity it would introduce for essentially one runner. There were also issues about how to properly locate the xunit.dll that they would eventually be linking against.

In the end, we decided it wasn't going to happen.

Unfortunately, this means that the Resharper runner is going to suffer from version dependence, so you'll only ever be able to test against one version of xunit.dll with Resharper (whichever version the Resharper support was installed with).

Users who want integrated Visual Studio test runner capabilities with version independence are encouraged to use TestDriven.net.

P.S. Only Resharper 3.1 is currently supported. We won't be supporting 4.0 until the final RTM. Sorry for the inconvenience.

xUnit.net RC2 Refresh + ASP.NET MVC Preview 2

Scott and Phil has been patiently asking us when we would add support for ASP.NET MVC Preview 2 for xUnit.net. After we released this refresh this morning, we decided to see if we could quickly add support for them.

Turns out it wasn't that painful, and we finished it. To use this:

• Go to the RC2 Refresh release page
• Download the file named xunit-build-1235-installer.zip
• Unzip its contents alongside the RC2 Refresh binaries
• Run xunit.installer.exe

That's it! In future builds, this new installer will replace the existing xunitext.runner.installer.exe (which we renamed, since the installer isn't just about test runners any more).

Enjoy!

xUnit.net RC2 Refresh

Today, we issued a refresh build for xUnit.net RC2.

There were a couple significant issues that we wanted to fix. First, we'd been having some strange behavior regarding TestDriven.net which we finally tracked down; second, we had an issue when we merged the source code for the extensions project which caused the MSBuild task to be broken.

Additionally, we've started work on a GUI runner which is included in the refresh. The actual GUI that you see today is just a placeholder on the way to the eventual GUI runner, so it's very rough. Today, you can use the GUI to run all the tests in an assembly; in addition, it doesn't lock the assembly, so you can leave the GUI open while building with Visual Studio (it even reloads the assembly when it detects that it has been changed).

Onwards toward the 1.0 release!

Update: We released support for ASP.NET MVC Preview 2 as well today!

xUnit.net 1.0 RC2 Released

Today, we released xUnit.net 1.0 RC2.

Before I get into why it took us 10 weeks to go from RC1 to RC2, let's cover what's available in this release:

• We merged the xunit and xunitext projects. After some evaluation of our two-project system, we've decided to kill the xunitext project. We have already moved all the source over; later today, I'll be copying the work items from xunitext over into xunit, and then sometime this week we'll kill the xunitext project. The single Build 1223 download released today contains all the libraries from xunit and xunitext. The DLL names and namespace names haven't changed, so you should simply be able to drop them in and use them as is.
• We obsolesced most of the Assert methods which take user messages. The only ones we left are those on Assert.True and Assert.False, which tend to be catch-all asserts which might require documentation. We will be removing the obsolesced methods in 1.0 RTM, so please move your calls to the message-less variants.
• We added an MSBuild task. We are now using this MSBuild task in our automated builds, so you can look in xunit.tests.msbuild to see how it's used (there's also an MSBuild sample in the Samples zip file). One of the biggest drawbacks to using xunit.console from MSBuild was the lack of feedback about progress, since the console runner printed a single dot for each run test, but MSBuild waits for whole lines of output before displaying anything. Our MSBuild task prints the name of each test after it has been run: in "low priority" gray for passing tests, in "warning" yellow for skipped tests, and in "error" red for failing tests.
• We added Assert.IsAssignableFrom. Based on feedback, we decided that there was value in two different asserts which tested for exact type (Assert.IsType) vs. inheritance/implementation (Assert.IsAssignableFrom).
• We added an IUseFixture<T> sample. This sample is designed to illustrate the use of IUseFixture<T> as a replacement for test fixture setup and teardown. It attaches to a SQL Express database and populates some data into it, and then provides the SQL connection and data IDs to the tests to be consumed.
• The console and MSBuild runner output version information for xunit.dll and the full pathname of the assembly under test. It should be self explanatory why this is valuable information. :)
• We added the Executor and ExecutorWrapper classes. More about this in a minute.
• We fixed several bugs. See the release for a list.

If you find any issues or have any questions, please use our forums or bug tracker.

Why did it take so long?

A lot happened in the 10 or 11 weeks since we released RC1. We had intended the RC to really be a release candidate, but as more people have been using xUnit.net, we found that there were a lot of problems centered around the runners. To the point, since runners are linked against a specific version of xUnit.net, they were not capable of running tests for other versions.

This wasn't too big an issue with the console runner, as you tended to upgrade all your libraries at once, but it did come into play with runners where you couldn't necessarily match the runner version against the version of xUnit.net that you were using (most commonly this was a problem with the runners inside Visual Studio: the TD.NET and Resharper runners).

We decided that tackling version-resilient test runners was a problem worth solving for v1.0. We started down the path in the context of writing the new MSBuild task. We added a simple class to xunit.net called Executor, which acts as an intermediary between xUnit.net specific types (like PassedResult, FailedResult) and the runners. It does this by changing the communication mechanism to XML, so that the runners are not required to link against version-specific types.

The Executor class was not intended to be consumed directly, as this would link the runner against a specific version of xUnit.net. Instead, we added a new assembly named "xunit.runner.utility", and into this assembly we placed a class named ExecutorWrapper. This class provides an abstraction away from linking to xunit.dll directly. Runner authors can link to this DLL and use this class to help them run tests.

Constructing an instance of ExecutorWrapper requires an assembly filename, a configuration filename, and a flag as to whether the system should shadow copy the DLLs for it. When it's created, it automatically creates an AppDomain on behalf of the runner and loads xunit.dll into it. When ExecutorWrapper is disposed, it cleans up the AppDomain.

ExecutorWrapper today offers a single property -- XunitVersion -- and a single method -- RunAssembly. Using this, you can run all the tests in an assembly, as well as discover which version of xunit.dll was loaded into the AppDomain. The XML callback is done in terms of the XML nodes that are normally part of the xUnit.net XML output file. As each test is run, for example, the callback will result in a single "test" XML node; once all the tests in a class are run, the result is a "class" node; etc.

Both the console runner and the MSBuild task runner are using ExecutorWrapper now. You can see that they are linked against xunit.runner.utility and not xunit.dll. The MSBuild task is the simplest illustration of how to write a version-resilient runner. We also converted our acceptance tests to use ExecutorWrapper, as this provides better end-to-end coverage that simulates the use of an actual test runner.

Executor and ExecutorWrapper are still a work in progress. Today they are capable of supporting simple assembly-level runners. We will be finishing them to support more complex runners, including the existing TD.NET and Resharper runners, as well as interactive (GUI) runners. If you intend to consume the ExecutorWrapper classes because you are a runner author, we would be interested in hear about the needs you would have for this API. We'd like to get it more or less right the first time around. :)

Hopefully this will eventually resolve issues people have been having with runner versions not matching the exact xunit.dll version of the assembly under test.

We Were Crashing, DLLs Were Missing, Fusion Log Didn't Help... Now What!?

One of the more difficult-to-debug scenarios in .NET is failure to load an assembly, especially as you create new App Domains programmatically.

The system which is responsible for resolving the loading of assemblies is called Fusion. The work of assembly loading can often be complex, so there is a tool which ships with the Windows SDK (and Visual Studio) call the Fusion Log Viewer (fuslogvw.exe). Suzanne Cook's blog contains extensive information about Fusion and its inner workings.

Most assembly loading issues can usually be resolved by judicious use of the Fusion Log Viewer. You can ask the tool to log binding successes and/or failures and then review the logs as necessary to see exactly what the system did and did not do on your behalf. Note that any changes you make to Fusion logging settings only affect an application when it starts up, so you will need to stop and restart your application after making changes.

Fusion Log Viewer never failed me until Wednesday, when Jim and I spent several hours debugging code that it took us about 10 minutes to write. I wanted to document the process we went through here, in case others have the same problem, since we weren't really able to find sufficient information about what was going on, and had to do a lot of guess work and spelunking to find the solution.

Here's the problem as it manifested itself to us. If you're not interested in the arcane details of creating App Domains and using cross-App Domain remoting, then go on outside and play and be happy that these issues don't affect you. :)

When you create an App Domain and ask to load assemblies into it, those assemblies usually need to reside on the disk in a path that is relative to the App Domain's ApplicationBase path. So when we create an App Domain for running the tests, we set the ApplicationBase path to the path containing the DLL that is being tested, which should also contain xunit.dll (adding a reference to the DLL copies it into your BIN directory during build).

The xUnit.net console runner (xunit.console.exe) creates a separate App Domain to run the test code in, so that the test code is completely isolated from the runner. When you run the console runner, it contains a copy of xunit.dll with it as well in the same directory, so the base App Domain and test App Domain both contain copies of xunit.dll within the appropriate ApplicationBase path areas. So far, so good. (Another day, we'll talk about what might happen if those two xunit.dll files are not the same version.)

When we create our test App Domain, we create an instance of a class named RemoteAssemblyCommand in the test App Domain. Short answer is that this class runs the tests for us, but it lives in the test App Domain instead of the base App Domain. We use Assembly.CreateInstanceAndUnwrap() against the RemoteAssemblyCommand type, which derives from MarshalByRefObject, so it stays there and we get a transparent proxy that allows us to call it remotely.

Whew! Trust me, setting this stuff up to work the first time is non-trivial. Even though Jim and I were both very familiar with App Domains, we still spent several weeks making sure all scenarios worked correctly.

The problem we hit came when we making the MSBuild task. The key difference here is that the base App Domain is created by MSBuild.exe, not xunit.console.exe, so the base App Domain's ApplicationBase path is actually C:\Windows\Microsoft.NET\Framework\v2.0.50727. This is important. (Where's the blink tag when you need it? :-p)

We built our MSBuild task using code purloined copied from the console runner, with calls to Console.WriteLine() replaced with Log.LogXxx(). We kept it really simple to start, just to make sure it was working correctly.

It failed. Spectacularly, and strangely.

Our call to CreateInstanceAndUnwrap() was telling us that the thing we created was not castable to RemoteAssemblyCommand. So we fired up the debugger and watched the whole process, including watching as RemoteAssemblyCommand was indeed instantiated properly from inside the test App Domain. It's just that when we popped back out the other side, the thing we had said it was a MarshalByRefObject but knew nothing of its RemoteAssemblyCommand heritage.

We couldn't really understand why. Identical versions of xunit.dll were loaded into both App Domains successfully. We thought for a while that it might be because there were two different xunit.dll files being loaded (one from the directory that contained the MSBuild task, and one from the directory that contained the test DLL), but that wasn't the problem, either.

We fired up Fusion Log Viewer, and poked through some of the resolution failures, but there wasn't actually any failures going on during the time when we ended up with the casting problem. On a lark, Jim suggested that we copy xunit.dll along side MSBuild.exe, which resolved the problem. We definitely had a Fusion binding problem, but it wasn't being logged.

That didn't make any sense, though. The xunit.dll assembly was already loaded in our App Domain. Why would it feel the need to attempt to re-load it?

To confirm our suspicions that this was an unlogged binding failure, we attached to the AssemblyResolve event from the base App Domain (System.Reflection.AppDomain.CurrentDomain.AssemblyResolve). Fusion will call this to allow you to help resolve any failed assembly lookups. Sure enough, it got called during the point where we would normally have failed, and as we suspected, it was trying to load xunit.dll. We inserted code in there to returning a copy of the existing assembly, and everything worked again.

Joy!

We're still not sure why Fusion decided it wanted to re-load the xunit.dll assembly when it was already loaded into the App Domain. It is clear now, though, that Fusion had always been doing it, except in the case of the console runner, it always found xunit.dll sitting right there next to the runner and so it succeeded. It wasn't until we tried to do the same thing from MSBuild that it failed, because our base App Domain's ApplicationBase path was now pointing under Windows instead of at the console runner.

Hopefully this will help someone else when they hit the same problem. :)