Brad Wilson

xUnit.net 1.5 Shipped

Jim and I released xUnit.net 1.5 a couple weeks ago.

This was the first full RTM release in about 9 months, an unfortunate side-effect of our busy schedules. :) The major new feature of 1.5 is the new GUI runner, which has evolved over several CTP and beta builds.

Here is a screenshot of the new GUI runner in 1.5:

If you're used to using the GUI runners in other testing tools, you'll notice that our GUI runner is significantly different. Rather than the traditional "tree" view, we've opted for a flat, integrated list of tests (the large box in the upper right of the window). Whatever is visible in this list will be run if you click "Run All"; you can also select specific tests, which turns the button into "Run Selected".

These test methods can be filtered in several ways, using the Refinements section on the left:

• The search box will search and filter methods whose method name, namespace name, or display name match the provided text
• The assemblies list allows you to filter down to just the tests in one or more of your selected assemblies
• The traits list shows all the [Trait] attribute name/value combinations in your tests.
• The count buttons at the top of the method list also work as filters, so you can quickly just see the tests that have passed, failed, or were skipped.

Assemblies are not locked, so you can leave the GUI runner open while working on your test code, and when test assemblies are recompiled, the GUI will automatically reload the DLL. Tests are run on a background thread so that the UI remains responsive while running tests.

There are lots of other small new features, plus a bunch of bug fixes, which we cover in the release notes (shown below):

xUnit.net release 1.5

Build #1479
Change set #42400

Important notes for Resharper users:

Resharper support has been moved to the xUnit.net Contrib project.

Important note for TestDriven.net users:

If you are having issues running xUnit.net tests in TestDriven.net, especially on 64-bit Windows, we strongly recommend you upgrade to TD.NET version 2.20 or later. It is also important to uninstall and reinstall TD.NET support using xunit.installer.exe.

This release adds the following new features:

• Better support for 64-bit machines, including forced 32-bit runners
• Introduced a new GUI runner, which allows:
  • Multiple assemblies
  • Filters on text, assembly, traits, and last run state
  • Project (.xunit) files
  • Multithreaded UI so that tests aren't slowed down by UI updates
  • Assemblies are not locked and are automatically reloaded so that you can continue to work on your tests in Visual Studio while leaving the GUI runner running
  • Automatically remembers UI window placement and recent assemblies/projects
• Additional MSBuild task which consumes project (.xunit) files
• Support for MVC 1.0 and MVC 2 Preview 1, including both C# and VB.NET
• A new test runner object model for runner authors (supporting multiple assemblies)
• IEnumerable/IEnumerable<T> support for Assert.Equal
• Better support for Assert.Throws for expressions which return values
• Support for installation-free running with TestDriven.net (2.23 or later)

This release fixes several issues, including:

• Added an output property named ExitCode to the MSBuild runner (0=success, -1=fail)
• Expected/actual messages are improved when values are the same but types don't match
• Fixed an issue where assemblies without config files run from MSBuild were using MSBuild.exe.config as their config file.
• Fixed several broken samples.
• Fixed an issue where long-running tests would be aborted in the console runner.
• Fixed an issue with CDATA handling in test results.

For a complete list of fixed issues, see the Issue Tracker.

This release includes:

• Core library (xunit.dll)
• Support for ASP.NET MVC 1.0 and 2.0 Preview 1
• Support for CruiseControl.net (see the wiki for more information)
• Support for TeamCity (see the wiki for more information)
• Version-resilient runner support (xunit.runner.utility.dll)
• Installer utility for runners and ASP.NET MVC (xunit.installer.exe)
• Runners:
  • Console runner (xunit.console.exe and xunit.console.x86.exe)
  • GUI runner (xunit.gui.exe and xunit.gui.x86.exe)
  • MSBuild runner (xunit.runner.msbuild.dll)
  • TestDriven.net 2.x runner (xunit.runner.tdnet.dll) see note above
• Extensions for .NET 2.0+ (xunit.extensions.dll):
  • Data theories using [Theory] with support for data from inline values, properties, Excel spreadsheets, SQL Server, and OLE DB data sources
  • [AssumeIdentity] extension
  • [AutoRollback] extension
  • [FreezeClock] extension
  • [Trace] extension
  • TestClass and Assertions base class for assertion extensibility

xUnit.net 1.5 Beta Released

Jim and I just shipped xUnit.net 1.5 Beta.

We're pretty much done with the features for 1.5 at this point. Other than a non-functional recent item list for projects in the GUI, we believe we are feature complete. We will probably be restructuring some of the GUI runner code so that it can have a long and maintainable life. :)

The big new feature for 1.5 Beta is the introduction of xUnit.net Test Project files (.xunit files). These files contain information about multiple assemblies and their run parameters, and can be consumed from the GUI and MSBuild runners (as well as created in the GUI). To accommodate this, the GUI got a small makeover in the menus, and now behaves like a single document application (where the document is your project file). We also support launching .xunit files from Windows Explorer (run the installer utility, xunit.installer.exe, to enable this support).

The other big feature is that we've included builds of the console and GUI runner which always run in 32-bit mode, even on 64-bit OSes. The default versions will run in whatever is native. We also included environment information (32- vs. 64-bit and the .NET version) in several places, including the native XML output files.

Finally, we added preliminary support for ASP.NET MVC 2 Preview 1. This support may need to evolve with later releases of MVC.

As always, we look forward to usage reports, especially with the new GUI. Your help lets us know that we're ready to ship!

xUnit.net 1.5 CTP 1

This morning, we published the first CTP of xUnit.net 1.5. This release includes a new GUI runner, support for ASP.NET MVC 1.0 RTM (for both C# and VB.NET), and several bug fixes.

A note for Resharper users: support for xUnit.net in Resharper has been moved to the xUnit.net Contrib project, and is no longer part of the core xUnit.net distribution.

New GUI Runner

Our goal with the new UI is to streamline the test running process for test-first developers. It is common for test-first development to result in several test assemblies and thousands of tests. The traditional thinking of test runners and tree representations of tests seems to fall apart when your test list becomes large.

We wanted to give developers a quick way to filter test lists that don’t involve navigating large trees. This first CTP allows users to filter by loaded assembly, as well as using a search box which live-filters the lists of tests. Additionally, the user can use the test list to select a subset of tests to run.

What’s Left To Do?

This runner is still in a very early state. We wanted to get it out and get user feedback. We have several features (and some significant code restructuring) left to implement before it will be ready. We are intending to re-add the automatic assembly reloading feature, as well as additional test list filtering (for example, filtering by traits). Mostly, though, we would like to see the community provide feedback on the general effectiveness of the design for their day to day runner needs.

We will be releasing at least one more CTP before we release the final 1.5 bits. Please take this opportunity to try out the new runner and let us know what you think!

It’s Not TDD, It’s Design By Example

The word “test” in software is a very loaded term. The first time I came across a tester (as in, a person performing the quality assurance role) was in my first professional job. We were never taught about testing at college, as we were all there to learn to be “developers”.

It turns out there are lots of different kinds of testing. In a series of blog posts titled Agile Testing Directions, Brian Marick came up with a 2x2 matrix of test types, and describes the activities which happen in each corner of the matrix. If you haven’t read the blog posts, they can help to provide a vocabulary about the kinds of testing that most software teams undertake.

A lot has changed in the nearly 6 years since Brian wrote those posts. Test Driven Development is arguably the most well known and most popular part of the eXtreme Programming software development process. For several years, I’ve been giving talks and coaching teams on how to do Test Driven Development, and by far the most frustrating part of the process has been overcoming people’s expectations when you use the word “test”. For most developers, “testing” is something that’s done by “testers”, not something that’s done by developers. Even those developers who are savvy enough to be doing unit testing are still practicing an activity whose primary purpose is quality assurance.

What the name Test Driven Development has going against it is that it doesn’t properly express the purpose of TDD; namely, that it is a process designed to help you drive and iterate the design of your implementation at the unit level. The result of the design process is unit tests, but their primary purpose is not one of quality assurance; rather, it is an expression of the intended usage of the component under design. In this way, the “tests” that you are writing become the first client of your component, and come into being just before the component’s code is written. The rhythm in TDD is “write a test, watch it fail; write the production code, watch the test now pass; when prudent, refactor the code to increase clarity and remove duplication”.

The unit tests written by TDD have some quality assurance value as a secondary effect of the test, but that is not their primary goal. Their primary goal is to help you design the code, and to give you a safety net with which to refactor your code. Note that when I use the word refactor, I mean it in the classical sense: to change the internal implementation of a component without changing its externally observable behavior; that is, if you find yourself needing to change a unit test, you are not doing refactoring.

Frustrated by the misunderstanding of the purpose of TDD, my friends and I (Peter, Jim, Brian, Scott, and many others, all agile practitioners and coaches) decided to start calling it Test Driven Design. A small change, but it starts to focus on the fact that the process is about design. Unfortunately, that “test” word baggage is still in there, so our next iteration was then Example Driven Design. This worked well too, but “EDD” and “TDD” were still too close together and confusing.

The final iteration ended up being Design By Example (DbE). Now when I talk about TDD, I always call it Design By Example, and explain why we like this name better than TDD. Where TDD(esign) or EDD failed to get traction, people really seem to resonate to Design By Example.

~ ~ ~

As an aside, you’ll note that most of the unit testing frameworks on .NET have the word “test” in them a lot. NUnit started this with the [Test] attribute, which MbUnit adopted, and MSTest converted into [TestMethod]. When Jim and I set out to design xUnit.net, it originally had a [Test] attribute in it as well, which is how it was when we first talked about it publicly at the first ALT.NET event in Austin. Even before ALT.NET, we’d been lamenting the name [Test] for the attribute exactly because of this baggage with the word “test”. We wanted xUnit.net to be a framework that was first and foremost for TDD (erm, DbE) practitioners.

At the Austin event, we decided to rename [Test] to [Fact]. While some users have complained that this is an arbitrary change, we felt it was right because it removed the focus of the word “test” from the code that you were writing when doing DbE. It also lined up very well with the [Theory] feature we added to xUnit.net Extensions to support David Saff’s data theories. The essential difference is: a [Fact] is an expression of some condition which is invariant, whereas a [Theory] is an expression of a condition which is only necessarily true for the given set of data. As such, [Theory]s are driven with external data; if you provide invalid data, then the theory will potentially produce invalid results.

An example of a [Fact] is the condition which says, “if you have failed to properly initialize this object, when you call this method on it, it will throw InvalidOperationException”. There are no variations on this rule; it is always true. An example of a [Theory], using the classic FizzBuzz interview question, is the condition which says, “if you pass a number which is divisible by 3, but not by 5, the result should be ‘Fizz’.” For this theory, you pass values which meet the initial requirement; if you were to pass 4 or 15, though, the result would be a failure, but that doesn’t mean the theory is bad, it simply means the preconditions for the theory weren’t properly met.

Announcing xUnit.net Contrib

Jim and I have a relatively limited amount of time to work on xUnit.net, since we’re not together in the same group any more. One of the things that has suffered as a result was the Resharper runner support, particular since neither of us uses Resharper. This has been complicated by the fact that the Jetbrains team seems intent to breaking their extensibility APIs with every single release.

Matt Ellis approached us about contributing some fixes for the Resharper runner, and we all though it might be a great opportunity to launch an xUnit.net Contrib project on CodePlex. Matt has very graciously agreed to run the project, whose first deliverable is a new Resharper runner with several bug fixes.

Thanks, Matt!

Everybody who uses xUnit.net and Resharper should go download the new runner bits. :)

xUnit.net 1.1 Released

Yesterday, we published xUnit.net release 1.1.

In addition to new features like TeamCity support, there are also several breaking changes; please review the breaking changes blog post for more information. We also documented the XML format that is used for xUnit.net's native XML output, as well as used for version-resilient test runner authors.

Look for a separate blog post soon on authoring your own test runner for xUnit.net.

Breaking Changes in xUnit.net 1.1

Updated November 28 with the release of xUnit.net 1.1.

We've just released xUnit.net version 1.1, and we wanted to take the opportunity to do a little cleanup work (that will have breaking changes).

We've added the following features:

• Support for TeamCity; requires:
  • TeamCity v3.1 or later
  • Tests linked against xunit.dll v1.1 or later
  • xunit.console.exe or xunit.runner.msbuild.dll v1.1 or later
• Higher level version-independent runner APIs (more below).

We also made the following breaking changes from v1.0x:

• Renamed XunitExt namespace to Xunit.Extensions.
• Renamed DLLs:
  • xunitext.dll => xunit.extensions.dll
  • xunitext.runner.msbuild.dll => xunit.runner.msbuild.dll
  • xunitext.runner.jetbrains.dll => xunit.runner.resharper.dll
  • xunitext.runner.tdnet.dll => xunit.runner.tdnet.dll
• Dropped the xunitext35.dll project entirely (see more below)
• Added a <start> XML element to the version independent runners (see more below)

TeamCity Support

The console and MSBuild runners will automatically detect when they're running in the context of TeamCity, so no changes are necessary to your automated build files, and you can use the same build targets for interactive use and continuous integration.

Dropped xunitext35.dll

The unit test extension methods for .NET 3.5 have never been a fully planned or developed feature, and as such, we moved them to the Samples project.

The new <start> XML element

The <start> XML element was added to xunit.dll v1.1 for version independent runners to be notified that a test is beginning to run. This should allow runners to show which test is currently running, which can be helpful when diagnosing issues with long-running tests. Version independent runners which are linked against ExecutorWrapper need to be written in such a way to cope with the presence or absence of the <start> element, since the it will only be provided when the tests are linked against xunit.dll v1.1 or later.

Higher level version-independent test runner APIs

One of the things we've had on our list for a while is documenting the XML that's used to communicate between version independent test runners and the xUnit.net execution engine. Today, you can link against xunit.runner.utility.dll and use the ExecutorWrapper class to gain access to these XML-based APIs, but since the XML was undocumented, it is a rarely used API. We have since documented the XML format on the project wiki.

Second, we introduced a higher level abstraction for version independent runner authors, which "cracks" the XML elements and converts them into calls against a new interface named IRunnerLogger. The version independent runner author implements this new interface, and uses the callbacks to the logger to present information to the user about the progress of the test run, as well as allowing the user to cancel the test run.

In the process of fleshing out these new runner APIs, we've converted all our runners (except ReSharper) to use them. Until I can write a blog post specifically designed to assist test runner authors, the best place to start understanding how runners are written is to look at the source code for our MSBuild runner.

It will still be important to understand which XML elements are supported based on which version of xunit.dll the user linked against. For example, if the user is linked against a version of xunit.dll prior to v1.1, the XML will not contain any of the <start> elements (and therefore IRunnerLogger.TestStart will never be called). Authors of version independent runners will need to be able to deal with the variance present between the available versions of xunit.dll. The XML format documentation shows which version of xunit.dll introduced each element and attribute.

We will continue to support the XML-based APIs as well as these newer APIs.

xUnit.net 1.0.3 Released

We shipped xUnit.net 1.0.3 today. Primarily this release is about adding support for ReSharper 4.1, but it also fixes a couple of annoying ReSharper-related bugs.

Enjoy!

xUnit.net 1.0.2 Released

Jim and I finished up work on xUnit 1.0.2 on Saturday and we pushed the publish button this afternoon. The one new notable feature is support for Resharper 4.0. We also fixed five bugs in the bug database (and probably a few more here and there that didn't get tracked).

It's been almost 3 months since 1.0.1, and if it doesn't seem like we've made a lot of progress for those three months, you're probably right.

• When I was on the CodePlex team, we had a dedicated 20% Slack day to work on xUnit.net, so it got ~ 40 hours/month work on it. Now that I'm on the ASP.NET team, Jim and I have to carve off a Saturday or Sunday, and we're probably running more like 12-18 hours/month right now.
• It's summer, and the end of the fiscal year at Microsoft, so there has been inevitable vacations, plus speaking gigs for both of us where travel eats into the weekend.
• We've been working on trying to make progress with our GUI runner, as well as doing some investigative work around supporting the Visual Studio Test runner. While we are making progress of sorts, it's not the kind of "release every month" progress right now.

Hopefully we'll be able to find some more solid time to work on it soon. Now that we have an officially supported release for Resharper 4.0, we can re-focus on the other runners.

xUnit.net: Support Visual Studio Test Runner?

Jim and I have been doing some investigation into the idea of supporting the Visual Studio test runner, and we’ve come across some issues that we think requires the input of the community.

If you’re interested in using xUnit.net with Visual Studio’s test runner support (Visual Studio 2008 SP1, Professional Edition or higher), please visit the xUnit.net forum post that describes the situation and provide us with your feedback.

Thanks!