xUnit - dynamically skipping tests for different test-environments

While developing MyCouch, I need support for multiple test-environments targetting local CouchDb installs as well as for cloud services such as Cloudant. Each different test-environment supports different features/scenarios. E.g. When I execute my integration tests against Cloudant, I don't want to support creation and deletion of databases. This post will show you how I'm using xUnit's Skip property of the FactAttribute, to skip the execution of tests, dynamically determined by configuration.

Since I do target Windows store apps as well as .Net4.0 and .Net4.5, I don't have this different connection settings for each environment in an app.config (not supported by WinRT). Also, I don't want to check in Cloudant configuration as I then would be putting my credentials out in the open for a cloud service. Instead, I have the configuration being delivered via simple HTTP-request to a local Nancy hosted webapp, that just reads and returns JSON lying in configuration files on disk. The configuration looks like this:

{
    "Supports": [
        "attachmentscontext", "changescontext",
        "databasecontext", "databasescontext",
        "documentscontext", "entitiescontext",
        "replicationcontext", "viewscontext",
        "mycouchstore", "createdb", "deletedb"],
    "TempDbName":"mycouchtests-temp",
    "ServerClient":{
        "Url":"http://localhost:5984",
        "User":"sa",
        "Password":"p@ssword"},
    "DbClient":{
        "ServerUrl":"http://localhost:5984",
        "DbName":"mycouchtests",
        "User":"mycouchtester",
        "Password":"p@ssword"}
}

This JSON is simply deserialized to a simple entity:

public class TestEnvironment
{
    public ServerClientConfig ServerClient { get; set; }
    public DbClientConfig DbClient { get; set; }
    public string TempDbName { get; set; }
    public string[] Supports { get; set; }

    public bool SupportsEverything
    {
        get { return Supports.Contains("*"); }
    }

    public TestEnvironment()
    {
        ServerClient = new ServerClientConfig();
        DbClient = new DbClientConfig();
        Supports = new[] { "*" };
        TempDbName = "mycouchtests-temp";
    }

    public virtual bool HasSupportFor(string requirement)
    {
        return
            SupportsEverything ||
            Supports.Contains(requirement, StringComparer.OrdinalIgnoreCase);
    }
}

The important configuration here isSupports. It defines what the test-environment supports. A test then uses a custom FactAttribute called MyFactAttribute which allows me to specify "requirements/scenarios" for a test:

[MyFact(TestScenarios.DocumentsContext)]
public void When_GET_of_document_with_id_and_rev_It_returns_the_document()
{
   ...
}

I can also specify more than one requirement/scenario that has to be supported:

[MyFact(TestScenarios.DatabasesContext,
    TestScenarios.CreateDb,
    TestScenarios.DeleteDb)]
public void Can_create_and_drop_db()
{
    ...
}

The implementation of the MyFactAttribute is quite simple. The "ugliest" part of it is that it has a dependency on a static IntegrationTestsRuntime to look at the current test-environment and what it supports:

public class MyFactAttribute : FactAttribute
{
  public MyFactAttribute(params string[] scenarios)
  {
    var environment = IntegrationTestsRuntime.Environment;

    if(!scenarios.Any() || environment.SupportsEverything)
      return;

    if (!scenarios.All(r => environment.HasSupportFor(r)))
      Skip = string.Format(
             "TestEnvironment does not support ALL test scenario(s): '{0}'.",
             string.Join("|", scenarios));
  }
}

If I now execute the Can_create_and_drop_db test, against a test-environment that has no support for either: DatabasesContext, CreateDb or DeleteDb, the test-runner will skip the test and output something like:

Can_Create_and_drop_db ignored: TestEnvironment does not support test scenario(s): 'databasescontext|createdb|deletedb'.

Not sure if it's the correct solution, but it's easy and solves my problem. Perhaps it can help you to.

Cheers,

//Daniel