Enabling parameterized tests in MSTest using PostSharp

I have blogged about the shortcoming of Microsoft’s unit testing framework in the past. It has very good Visual Studio (and TYFS) integration out of the box but it seems that in order to use it I have to suffer lack of functionality I’m used to taking for granted when using any other .NET unit testing framework out there. ON the of feature I miss most is RowTest (Theory for you XUnit users). When using MSTest I want to be able to create a test that receive parameters. It shames me to have to tell a co-worker to write 10 (or more) different tests that test the exact functionality with differed conditions.

There is a way to run parameterized tests in MSTest namely data-driven tests the downside is that you have to create an external file to host the parameters making the test less readable to my taste. Another alternative is to use MSTest that comes with VS2010 and extend it – I wrote about it in a previous blog post.

I’ve decided to create another solution for those of you that do not use VS2010 and .NET 4 or do not want to extend the testing framework – using data-driven test and the power of PostSharp.

The incidents

One test marked as a test class no parameters
One test with the same name with parameters
One simple RowAttribute to keep the parameters
One PostSharp OnInvocation attribute to do all of the magic
Mix & Run

The Test(s)

Because of MSTest limitation will need two functions with the same name. The first one without parameters will be used as a host test method to run and the second will contain the actual test code:

[TestClass]
public class MyRowTests
{
    public TestContext TestContext { get; set; }

    [TestMethod, RowTest]
    [DeploymentItem(@"..\..\SimpleRowTest.xml")]
    [DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML",
     "|DataDirectory|\\SimpleRowTest.xml",
     "Row", DataAccessMethod.Sequential)]
    public void SimpleRowTest()
    {

    }

    [Row(1, 2, 3)]
    [Row(3, 2, 5)]
    [Row(3, 4, 9)]
    [Row(3, 4, 7)]
    public void SimpleRowTest(int x, int y, int result)
    {
        Assert.AreEqual(result, x + y);
    }
}

The first method is the one MSTest know and respect and we’re going to use it to run the second method. I’ve added DeploymentItem and DataSource attribute to run the xml file that is going to be generated with the data from the Row attributes.

Note: It is important that the deployment item would point to the relative path of the test project file from the solution file – this is one nasty bit of code and I wish I could find a way to throw it away.

Note: It is crucial that you have TestContext defined in the test class as well – otherwise MSTest won’t be able to access the test’s parameters:

RowAttribute

This is a plain old C# attribute – nothing special, just a container to hold the row test data:

[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
public class RowAttribute : Attribute
{
    private readonly object[] _rowParameters;

    public RowAttribute(params object[] rowParameters)
    {
        _rowParameters = rowParameters;
    }

    public object[] Parameters
    {
        get { return _rowParameters; }
    }
}

It’s marked to be used on a method and that it can be applied multiple times and that’s it./p>

RowTestAttribute

This is where the magic happens! The attribute does two things:creates a xml data source file and run the method with the parameters. Inheriting from MethodInterceptionAspect enables hooking into two events – compilation of the class and when it runs:

[Serializable]
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class RowTestAttribute : MethodInterceptionAspect
{
    private MethodInfo _methodToInvoke;

    public override void CompileTimeInitialize(MethodBase method, AspectInfo aspectInfo)
    {
        base.CompileTimeInitialize(method, aspectInfo);

        _methodToInvoke = method.DeclaringType.GetMethods(BindingFlags.Instance | BindingFlags.Public).Single(info => info.Name == method.Name && info != method);

        var rowAttributes = _methodToInvoke.GetCustomAttributes<RowAttribute>();

        CreateDataSource(_methodToInvoke, rowAttributes);
    }

    public override void OnInvoke(MethodInterceptionArgs args)
    {
        var testContext = args.Instance.GetPropertyValue<TestContext>("TestContext");

        var parameters = _methodToInvoke.GetParameters();

        var arguments = testContext.GetParameterValues(parameters);

        try
        {
            _methodToInvoke.Invoke(args.Instance, arguments.ToArray());
        }
        catch (TargetInvocationException exc)
        {
            // fix stacktrace
            FieldInfo remoteStackTraceString = typeof(Exception).GetField("_remoteStackTraceString", BindingFlags.Instance | BindingFlags.NonPublic); // MS.Net

            remoteStackTraceString.SetValue(exc.InnerException, exc.InnerException.StackTrace + Environment.NewLine);

            throw exc.InnerException;
        }
    }        

    private string CreateDataSource(MethodBase testMethod, IEnumerable<RowAttribute> customeAttribute)
    {
        var projectFileName = PostSharpEnvironment.Current.CurrentProject.EvaluateExpression("{$MSBuildProjectFullPath}");
        var projectPath = Path.GetDirectoryName(projectFileName);
        var filename = Path.Combine(projectPath, _methodToInvoke.Name + ".xml");

        using (var sr = new XmlTextWriter(filename, Encoding.ASCII))
        {
            sr.WriteStartDocument();
            sr.WriteStartElement("Rows");
            foreach (var rowAttribute in customeAttribute)
            {
                sr.WriteStartElement("Row");

                for (var i = 0; i < rowAttribute.Parameters.Length; i++)
                {
                    sr.WriteStartElement("value" + i);
                    sr.WriteValue(rowAttribute.Parameters[i]);
                    sr.WriteEndElement();
                }

                sr.WriteEndElement();
            }

            sr.WriteEndElement();
            sr.WriteEndDocument();
        }

        return filename;
    }
}

There are three main functions:

CompileTimeInitialize – that get called when the code compiles and is responsible to create the XML file – remember we need it before the test runs
OnInvoke – called when the test runs. Reads the data from the data source and convert the parameters before running the second (parameterized) method
CreateDataSource – you’ve guessed it – this is where we create the xml file – simple and dirty… I use PostSharp’s EvaluateExpression to find where the project’s file is – remember this bit runs on compile so no tricks using reflection could be used to find the target assembly.

Utilities

Simple methods I use in the RowTestAttribute – just in case you’re wondering:

public static class Extensions
{
    public static IEnumerable<T> GetCustomAttributes<T>(this MethodBase method) where T : Attribute
    {
        return (T[])Attribute.GetCustomAttributes(method, typeof(T));
    }

    public static T GetPropertyValue<T>(this object instance, string propertyName)
    {
        var propertyInfo = instance.GetType().GetProperty(propertyName);
        return (T)propertyInfo.GetValue(instance, null);
    }

    public static IEnumerable<object> GetParameterValues(this TestContext context, ParameterInfo[] parameters)
    {
        for (var i = 0; i < parameters.Length; i++)
        {
            var parameterType = parameters[i].ParameterType;
            var inputData = context.DataRow["value" + i];

            yield return Convert.ChangeType(inputData, parameterType);
        }
    }
}

Running the test

MSTest force me to run the empty parameter-less method. Running the test from the beginning of this post will result in the following:

How cool is that – The “test” is actually 4 tests run separately and one of them failed – with a clear error message. And there’s more clicking on the test we get to see the full error message:

In fact I was so impressed with how it works that I put it to us at work despite the fact that it’s not finished yet.

Is it done?

Not yet – I would really like to throw away the DeploymentItemAttribute and if I can get away with it the DataSourceAttribute. I do use it daily and it feels just wrong.

I’ve tried creating the attributes in runtime – using PostSharp but Visual Studio was not fooled – probably because DTE is used to collect these attribute and not reflection.

By publishing this post I was hoping to achieve two goals:

Help other lost souls such as myself that are forced to use MSTest but wish they had the ability to run RowTests – BTW you’re welcome
Get feedback, suggestions and ideas how this nifty idea can be improved

And please let me know if you put this code to use – that’s what the comments are for

Happy coding…

Labels: .NET, C#, MSTest, PostSharp, Tips and Tricks, Unit tests