Brad Wilson

xUnit.net 1.0 Released

We released xUnit.net 1.0 RTM today.

Acknowledgements

Many people have helped us get to our 1.0 release.

First, we want to acknowledge the inventors of the core ideas behind xUnit frameworks, especially Kent Beck and Eric Gamma, without whom we probably wouldn't have anything to talk about today. :)

Second, we want to thank the people who hatched the idea of a new framework with us, including Peter Provost, Brian Button, Scott Densmore, Jonathan Wanagel, Jamie Cansdale, and Drew Miller.

Lastly, our community provided significant feedback to us during the development process by providing feedback on the web site, opening feature requests, reporting bugs, and generally helping guide the design of the framework. We especially want to thank Matt Podwysocki, Ben Hall, Harry Pierson, Scott Hanselman, Kirk Viehland, Daniel Cazzulino, Oren Eini, Jeff Brown, and Charlie Poole.

Looking Back

We release Beta 1 of our framework September 19, 2007, almost exactly 7 months ago. It took a lot longer than we thought it would to get to 1.0, especially given that we had been using the framework extensively internally for a while before that.

Our original release plans called for just a console runner, but our users quickly educated us on the value of variety. We personally are TestDriven.net users, so that became our second runner; our community quickly followed on asking us to support ReSharper and to write a stand-alone GUI runner. Our time spent with the console runner in an automated environment then drove us to write an MSBuild task which provides much better feedback on the testing process.

Extensibility was a very important feature for us. We felt that there was a lot to be desired in the landscape of extensibility for test frameworks. An important early decision we made was to push as much of our "functionality" as we possibly could out into extensions so that we were forced to exercise and improve our extensibility points.

Another unexpected issue was around test runners and their tight coupling to the version of xUnit.net that they were built against. We made a decision to push out the 1.0 release in order to better support the idea of having test runners that were independent from any specific version of xUnit.net. We succeeded for all of our runners, save the ReSharper runner, for reasons we've already documented.

Looking Forward

Our near-term focus is going to be on refining the process of actually using the framework for Test Driven Development, especially in conjunction with Visual Studio. While tools like TestDriven.net provide a lower friction environment than others, we still believe that there is room for improvement in the day to day usage for TDD.

A lot of this work will be centered around the GUI runner. What we released in 1.0 is essentially a bare-bones runner that we labeled experimental. While we would like to know if you have issues with the runner, please be aware that we are intended to make dramatic changes to it to help support our idea of zero-friction TDD.

It is our intention, therefore, to release interim drops of the GUI runner as we make progress, hopefully on a fairly regular schedule.

Additionally, there are several releases on the horizon (especially ReSharper 4.0 and ASP.NET MVC) which will likely also result in new point releases of the framework.

For the longer term (version 2.0), what we really want now is to let the framework get greater adoption so that many more users can help provide feedback on the things they love and the things they don't, the things that just work and the things that could be tweaked. If you download and use 1.0, please visit our forums and tell us about your experiences!

Small Decisions Sometimes Cause Big Hacks

When .NET was being designed and introduced, it was notable not only in the ways in which it “borrowed” from Java, but also in the ways in which it parted from it. Some decisions were obviously good (dumping checked exceptions comes to mind) while some were clearly troubling (non-virtual methods by default).

Some, though, seemed insignificant at the time until you come up against them. Today’s example:

When is the stack trace for an exception generated?

In Java, the stack trace is generated when an exception is created, and in .NET it’s generated when the exception is thrown. In practical terms, with 99% of throws looking like this, it hardly matters:

throw new SomeException();

The creation and the throw actually happen on the same line of code. The stack trace is identical in both cases.

However, this seemingly small decision has rippling impact throughout the framework and languages. How many times have you seen this code and cringed?

catch(Exception ex)
{
    DoSomething();
    throw ex;
}

We’ve all seen it. That “throw ex” line destroys the original stack trace and replaces it with your own. The C# language team even had to invent a way to “re-throw an exception without destroying the original stack trace”, which is “throw;” on a line all by itself.

This probably should’ve been the clue that the wrong decision was made.

It’s reasonable to think that there might be a time when you have an exception object, outside of a catch block, and you want to re-throw that exception as though it were never caught. In fact, the framework team themselves came across just such a situation and invented their own hack to handle it, and built it into the Exception class.

When you are talking to a class via remoting, an exception thrown by the remote component is caught by the remote end and serialized back to the transparent proxy. The transparent proxy (which is what the client code is talking to) needs to experience that exception just as though it had been thrown locally. The problem is, we’re not in a catch block; we have a hydrated exception, but need to re-throw it without trashing the stack trace.

There is a private field inside of Exception into which you can stuff a stack trace which says “when I throw you, this is your stack trace; don’t make a new one”. This sample helper method shows you how to do the work:

public static void RethrowWithNoStackTraceLoss(Exception ex)
{
    FieldInfo remoteStackTraceString =
        typeof(Exception).GetField("_remoteStackTraceString",
                                   BindingFlags.Instance | BindingFlags.NonPublic);
    remoteStackTraceString.SetValue(ex, ex.StackTrace);
    throw ex;
}

I first ran across this trick via Peter Provost pointing me to Chris Taylor’s blog post on the subject. I’ve used it several times now (in ObjectBuilder and xUnit.net), and I’m thankful for it every time I run across it.

Since we’re talking about exceptions…

One of my biggest gripes about exceptions relates to reflection. The remoting team obviously went to great lengths to hide the fact that you were talking to a transparent proxy instead of the real object. That’s really the whole reason this hack exists in the first place. You can treat a transparent proxy just like the real thing.

It’s not so easy with Reflection. If you invoke something through reflection, and it throws an exception, the reflection infrastructure catches that exception and wraps it in TargetInvocationException. It’s basically an “I was here!” exception: the thing you actually care about it partially obscured by it. Nobody ever really cares about TargetInvocationException; it’s just a reminder that you’re doing things differently than normal, and you can’t treat some thing like you normally would.

I use this re-throw trick when reflection is an unimportant implementation detail (which it almost always is). For example, when running a test in xUnit.net, the test method is invoked via reflection; the person writing the test shouldn’t ever care about that detail. If every exception that was thrown in a test was reported as a TargetInvocationException, I’m pretty sure people would be telling us it was a bug.

So our implementation of TestCommand.Execute() uses this code instead:

public MethodResult Execute(object testClass)
{
    try
    {
        testMethod.Invoke(testClass, null);
    }
    catch (TargetInvocationException ex)
    {
        ExceptionUtility.RethrowWithNoStackTraceLoss(ex.InnerException);
    }

    return new PassedResult(testMethod, Parameters);
}

We originally spun in a loop at the point where we caught exceptions and turned them into FailedResult, stripping off any TargetInvocationExceptions that were contained. But that made testing for TargetInvocationException difficult in some cases, because you couldn’t differentiate between a TargetInvocationException you didn’t care about from one you did.

As such, everywhere that we use reflection, you’ll find a try/catch like this, eradicating the less-than-useless TargetInvocationException from the exception stack. Every time we have to write it, it makes me wish the reflection team had just done the right thing in the first place.

xUnit.net 1.0 RC3 Released Today

We've finally finished off all the work we wanted to achieve with the version-independent runner support, so we cut a new release candidate today. There were a few bugs fixed, and we added support for CruiseControl.net and ASP.NET MVC Preview 2, but otherwise this release should function more or less like the RC2 Refresh.

We believe we are now v1 feature complete.

At this point, we will only be fixing critical bugs. We're especially looking for feedback from runner authors on the functionality of ExecutorWrapper. If you're looking for examples on how to use ExecutorWrapper, the best places to start are the source code for the MSBuild task and the unit tests, although all the runners (except the Resharper runner, as I talked about yesterday) are using ExecutorWrapper now. In fact, all of our dynamic-compilation acceptance tests are also use ExecutorWrapper.

Barring any major bugs found, we're planning to release v1 at the end of the month.

xUnit.net's Resharper Runner - No Version Independence

Our version independent runner scheme has served us well so far: all of our runners have converted over to it except for the Resharper runner. Jim and I intended to tackle the Resharper runner today and make it version independent as well, but ran into some stumbling blocks that caused us to postpone the work indefinitely.

The API for the version independent runners is relatively simple, and everything is communicated in terms of simple strings and XML. I explained a bit about it in my previous post about RC2, so I won't re-cover old ground.

The problem with Resharper that's different from all the other runners is that it operates in a mode where there is no assembly file. You are asked to answer questions about whether classes contain tests based on the source code that's in the editor, not the bits that are on the disk. This makes perfect sense: they want to provide "run this" chicklets as soon as you add a test to your assembly.

Unfortunately, our API is based on assemblies on the disk. Most of the decision making is done on your behalf; you're just asking to run assemblies, classes, or tests based on filenames and fully qualified type and method names.

In order to support Resharper, we would essentially have to serialize (potentially incomplete) type information into XML and pass it across an app domain boundary to get it inspected in order to make decisions about whether something is a test or not. Aside from the obvious performance implications, we were also concerned with the tremendous complexity it would introduce for essentially one runner. There were also issues about how to properly locate the xunit.dll that they would eventually be linking against.

In the end, we decided it wasn't going to happen.

Unfortunately, this means that the Resharper runner is going to suffer from version dependence, so you'll only ever be able to test against one version of xunit.dll with Resharper (whichever version the Resharper support was installed with).

Users who want integrated Visual Studio test runner capabilities with version independence are encouraged to use TestDriven.net.

P.S. Only Resharper 3.1 is currently supported. We won't be supporting 4.0 until the final RTM. Sorry for the inconvenience.

Upcoming Events

Man, the time has flown by! I can't believe it's almost Tax Day.

Next weekend (Fri April 18 through Sun April 20) I'll be hanging out at DigiPen for the ALT.NET Seattle event. Since this event was purposefully timed to coincide with the MVP Summit next week, I imagine I'll be out and about meeting up with people during the week prior; if you're in town and want to hang out, my e-mail is bradwils at microsoft.

The P&P Summit in Quebec is coming up in just a few weeks, May  6-8. At the moment I'm scheduled to do a talk on Dependency Injection with Scott Densmore and a talk on Open Source with Peter Provost. I was supposed to be at the summit in Redmond last fall, but I ended up injuring my knee and unable to walk (which makes it pretty tough to present, too). Scott's talk on DI in the fall was centered around our work moving ObjectBuilder towards 2.0; I expect our talk this time will be more focused on the final OB2 and Unity bits that just shipped.

Opinions, Everybody's Got Them (Including Me)

I don't often get interviewed. Part of that is because I'm not really a super-important kinda guy, but part is also because my cat doesn't have the required credit card necessary to get his own blog. Yet.

A couple months ago, Scott Swigart and Scott Campbell asked for an interview. Suckers. :) Ignoring the voice of Patton Oswalt in my head (re: demon monkeys), I sat down and gave an interview that most of my friends would call "uncharacteristically lucid and on-topic". Yeah, I'm a rambler. Sue me.

Theit summary makes it seem weightier than I recalled:

In this interview we talk with Brad Wilson - Software Developer in Microsoft’s OfficeLabs team. In specific, we talk about:

• CodePlex, the Microsoft open source repository
• Community participation other than coding
• Open-source governance at Microsoft
• Open-source inroads and future at Microsoft

Enjoy!

xUnit.net RC2 Refresh + ASP.NET MVC Preview 2

Scott and Phil has been patiently asking us when we would add support for ASP.NET MVC Preview 2 for xUnit.net. After we released this refresh this morning, we decided to see if we could quickly add support for them.

Turns out it wasn't that painful, and we finished it. To use this:

• Go to the RC2 Refresh release page
• Download the file named xunit-build-1235-installer.zip
• Unzip its contents alongside the RC2 Refresh binaries
• Run xunit.installer.exe

That's it! In future builds, this new installer will replace the existing xunitext.runner.installer.exe (which we renamed, since the installer isn't just about test runners any more).

Enjoy!

xUnit.net RC2 Refresh

Today, we issued a refresh build for xUnit.net RC2.

There were a couple significant issues that we wanted to fix. First, we'd been having some strange behavior regarding TestDriven.net which we finally tracked down; second, we had an issue when we merged the source code for the extensions project which caused the MSBuild task to be broken.

Additionally, we've started work on a GUI runner which is included in the refresh. The actual GUI that you see today is just a placeholder on the way to the eventual GUI runner, so it's very rough. Today, you can use the GUI to run all the tests in an assembly; in addition, it doesn't lock the assembly, so you can leave the GUI open while building with Visual Studio (it even reloads the assembly when it detects that it has been changed).

Onwards toward the 1.0 release!

Update: We released support for ASP.NET MVC Preview 2 as well today!