Mocking Still Not Quite There For Me

The release of Moq (and the subsequent updating of Rhino Mocks in response) has made mock frameworks much more approachable to me, but I still find situations where hand-rolled stubs seem to work best.

Wednesday, Jim and I were trying to see what it might look like if we replaced our stubs with a mocking framework. We have two interfaces which we are hand stubbing today:

public interface ITestCommand
{
  string Name { get; }
  bool ShouldCreateInstance { get; }

  MethodResult Execute(object testClass);
}

public interface ITestClassCommand
{
  object ObjectUnderTest { get; }
  ITypeInfo TypeUnderTest { get; set; }

  int ChooseNextTest(ICollection<MethodInfo> testsLeftToRun);
  Exception ClassFinish();
  Exception ClassStart();
  IEnumerable<ITestCommand> EnumerateTestCommands(IMethodInfo testMethod);
  IEnumerable<IMethodInfo> EnumerateTestMethods();
  bool IsTestMethod(IMethodInfo testMethod);
}

A quick background on some of the types involved here:

• ITestClassCommand represents a “class with tests”. It’s used by the execution engine when running the tests. Most of what’s used there isn’t important, but what is important for this test is EnumerateTestCommands().
• ITestCommand represents a “single execution of a test”. Each test method may yield multiple executions (think data-driven testing), which is why ITestClassCommand.EnumerateTestCommands() returns a collection of these rather than just one.
• IMethodInfo is an abstraction of the CLR’s MethodInfo class, which is necessary because sometimes when you’re being asked to enumerate these things, you don’t actually have a real MethodInfo (f.e., when ReSharper is asking you to tell it what tests exist in the source file that user is looking at).

The actual class under test is TestCommandFactory. Its single responsibility is: given a ITestClassCommand instance and a MethodInfo, generate the ITestCommand instances to run all the tests. It does more than just get the instances from the class command: it wraps them using the chain of responsibility pattern to provide all the extra work that goes into running and reporting on a test command, which includes:

• Running all the BeforeAfterAttributes found decorating the test method
• Creating a new instance of the type under test (if the inner ITestCommand.ShouldCreateInstance is true)
• Turning Trace.Fail and Debug.Fail statements into thrown exceptions
• Catching exceptions thrown by the test and turning them into FailResult objects
• Timing how long it takes to run the class

As you can see, the presence and absence of these things is important, and order is also important. Our hand-rolled stubs record the values that were passed in, and allow you to control values that were returned. Our test, in pseudo-code, is approximately:

Arrange:

Create a stub of ITestCommand (stubTestCmd) - In response to ShouldCreateInstance, return true

Create a stub of ITestClassCommand (stubClassCmd) - In response to EnumerateTestCommands, return StubTestCmd

Get the MethodInfo for an empty, void-returning method (methInfo)

Act:

Call TestCommandFactory.Make, passing stubClassCmd and methInfo

Assert:

Ensure that the IMethodInfo passed into stubClassCmd.EnumerateTestMethod wrapped methInfo

Ensure that we returned a wrapped ITestCommand (test all the levels of wrapping described above)

Because our stubs record the passed-in values and we can inspect them, our first assertion is easy to write:

Assert.Same(methInfo, stubClassCmd.EnumerateTestCommands_TestMethod.MethodInfo);

Note that we’re not just testing the value (that would be stubClassCmd.EnumerateTestCommands_TestMethod) but we’re actually diving into it and testing a specific value (.MethodInfo). This is because ITestClassCommand takes an IMethodInfo, but we actually pass a MethodInfo to the method under test.

When we tried to convert this to Moq, we were stuck. We could make a Mock<ITestClassCommand>, but we could neither specify (via .Expect) nor verify (via .Verify) against the data item it recorded. We actually need the actual value; the wrapper isn’t important, but what it wrapped was.

We found a way to write the test, which was to cascade a .Callback after the .Expect and do the assertion. However, this put assert logic up inside our arrange. Alternately, we could’ve stashed the value into a temporary and did the assertion afterwards, but that made things start to feel a little unwieldy compared to the hand-rolled stubs.

We also found the use of “Expect” to be unfortunate, particularly the syntax that we ended up with (approximately):

var mockClassCommand = new Mock<ITestClassCommand>;
mockClassCommand.Expect(x => x.EnumerateTestCommands(It.IsAny<IMethodInfo>()))
                .Returns(new List<ITestCommand>(mockTestCmd));

Our problem is with the monstrosity of “It.IsAny<IMethodInfo>()”. It looks bad enough with a single parameter, but would look positively awful with several parameters. The truth is that I rarely want to say “if given A, return B”; I almost always say “return B, and I’ll verify in the Assert that I was passed A”. This would be a much better syntax:

var mockClassCommand = new Mock<ITestClassCommand>;
mockClassCommand.Stub(x => x.EnumerateTestCommands)
                .Returns(new List<ITestCommand>(mockTestCmd));

The compiler error I get from this tells me that the Lambda system in C# 3 probably won’t ever let me get to this kind of a syntax. I’m also not even sure about what the syntax would look like that lets me inspect the executions of each call to EnumerateTestCommands, so that I could ensure it was only called once and that the wrapper wrapped the right thing.

I think Wednesday we came to the realization that a code generator was probably what we were going to end up with. I’ve come around several times to the problem and, even after seeing Lambdas in C# 3, concluded that a generator which could creates stubs of any interface/abstract class is still the ideal situation.

Comments

[Jeremy Gray]

Were this Rhino, could the "It.IsAny" portion not be replaced with a chained .IgnoreArguments() call? I'm not sure on the Moq equivalent, if any, but I'm sure there has to be a more succinct way to clean up that part.

As an aside: To be honest, by the end of your post it ends up having read like you bumped into the "It.IsAny" problem and then promptly gave up. Was that really how things went?

[Brad Wilson]

I think that It.IsAny<>() is a symptom of the larger problem with mocking and .NET (or perhaps mocking and early bound languages). It forces me into unnatural and undesired constructs just to satisfy compiler requirements.

To be honest, C# 3 was required to even get close. NMock was horrible, and Rhino's Record/Replay stuff was slightly better but still quite awful. And I'm not faulting the framework authors, because they're clearly trying to make the best of a bad situation, but it's still just plain awkward and ugly to use these things sometimes.

[jdn]

This is very well argued, and helps to give coherent form to some of my own vague misgivings.

Thanks for this.

[Daniel Cazzulino]

I think I can get this to work (it's compiling now :)):

mockClassCommand.Expect>(x => mockClassCommand.Object.EnumerateTestCommands).Returns(new List(mockTestCmd));

You still have to specify the return type (and argument types) as the compiler will not properly infer the Expect overload signature that most closely matches the argument :(

Is it an improvement at all?