Set-up and tear-down

Recently as part of program run-time performance optimization effort, I was scanning through the code for occurrences of keyword new. The goal was to find unnecessary memory allocations. I was surprised to find most of news in the unit test module. Not that there was any particular need for memory allocation: it is just that the framework that was chosen (a fairly popular one) enforces on the programmers bad practices of manually controlling the life-time of their objects.

I assume you all know what the “RAII” approach to structuring program code is. The programmers in my team know it as well, but they cannot use it effectively in our unit-test framework. Typically when you define similar test cases (e.g., when testing different member functions of the same class), you want to group them (into a suite) and execute the same set-up code before each test case, and possibly, run the same clean-up (or tear-down) code after each test case. This pattern can be illustrated as follows:

{
  test_case_1()
  {
    // set up c
    // test1(c)
    // tear down c
  }

  test_case_2()
  {
    // set up c
    // test2(c)
    // tear down c
  }

  test_case_3()
  {
    // set up c
    // test3(c)
    // tear down c
  }
}

In order not to repeat the same set-up and tear-down code in each test case, the natural expectation of the programmers is to have a way to define set-up and tear-down code only once, and have the framework inject it into every test case auto-magically.

Our framework tries to follow the design implemented in JUnit. Apparently JUnit is treated as some point of reference. And the author of our framework tries to be as close as possible to the original, which includes importing some of Java habits. Thus, you define a test suite by declaring a class which inherits from some special test suite base; in order to provide a clean-up and tear-down code, you have to override virtual member functions setUp() and tearDown():

class MyTest : public UTFramework::TestSuite
{
public:
  void setUp() override;
  void tearDown() override;

  test_case_1();
  test_case_2();
  test_case_3();
}

Once you do it, and inform the framework, that it should consider MyTest when running all tests, and that test_case_1, test_case_2 and test_case_3 are in fact tests, it will perform the following sequence of operations:

MyTest t;
t.setUp();
t.test_case_1();
t.tearDown();
t.setUp();
t.test_case_2();
t.tearDown();
t.setUp();
t.test_case_3();
t.tearDown();

This is more-less what we wanted. But suppose the test cases all need to use a resource, which is represented by a RAII-like class: the resource R is initialized in constructor and cleaned up in the destructor. We need to initialize it before each test case, and unfortunately setUp() and test_case_1() can only communicate via member data. So, what the programmers are forced to do is something like this:

class MyTest : public UTFramework::TestSuite
{
  R* res_ = nullptr;

public:
  void setUp() override {
    res_ = new R{params};
  }
  void tearDown() override {
    delete res_;
  }

  test_case_1() {
    use (*res_);
  }
  test_case_2() {
    use (*res_);
  }
  test_case_3() {
    use (*res_);
  }
}

Quite horrible, isn’t it? Is it exception-safe? It is impossible to tell what the framework does. My experience is that frameworks that force programmers to resort to such things hardly ever pay attention to exception safety, and never bother to document what they do upon exception.

We can mitigate the problem by employing a smart pointer, or even better: by employing boost::optional:

class MyTest : public UTFramework::TestSuite
{
  boost::optional<R> res_;

public:
  void setUp() override {
    res_.emplace(params);
  }
  void tearDown() override {
    res_ = boost::none;
  }

  test_case_1() {
    use (*res_);
  }
  test_case_2() {
    use (*res_);
  }
  test_case_3() {
    use (*res_);
  }
}

But this is still far from ideal. It does not respect basic C++ idioms: that members are initialized in constructors and cleared up in destructors. We have to deal with the fact that res_ may not contain a value, at least technically. A far better solution would be to enforce the following usage pattern on the programmers:

class MyTest
{
  R res_;

public:
  MyTest() : res{params} {}
  ~MyTest() {} // not really needed in our case

  test_case_1() {
    use (res_);
  }
  test_case_2() {
    use (res_);
  }
  test_case_3() {
    use (res_);
  }
}

And have the framework execute our test cases in the following pattern:

{
  MyTest t;          // <-- set up
  t.test_case_1();
}                    // <-- tear down
{
  MyTest t;
  t.test_case_2();
}
{
  MyTest t;
  t.test_case_3();
}

And in fact this is what a couple of decent C++ unit-test frameworks (like Boost.Test) are doing. For an illustration, I chose framework Catch. It allows you to define a common set-up and tear-down functions in a manner similar to the one above. You define a class with custom data, default constructor representing a set-up procedure and destructor representing the tear-down procedure:

struct Fx
{
  int i = 1;
  R res {params};
  Fx() { /* set-up */ }
  ~Fx() { /* tear-down */ }
};

Because we are only initializing the member variables, we do not even need to define the default constructor or destructor, it is enough that we initialize the members in-place (which is an option in C++11). Now, we can use such fixture in the tests:

TEST_CASE_METHOD(Fx, "MyTest", "[mytest]") // test suite
{  
  SECTION("test case 1") {                 // test case
    REQUIRE(i == 1); // you have access to i
    REQUIRE(i);
  }
  
  SECTION("test case 2") {                 // test case
    use(res);
  }

  SECTION("test case 3") {                 // test case
    use(res);
  }
}

A couple of macros do the magic. This generates the following sequence of calls (in pseudo-code):

{
  _DerivedFromMyTest t;
  t.SECTION("test case 1");
}
{
  _DerivedFromMyTest t;
  t.SECTION("test case 2");
}
{
  _DerivedFromMyTest t;
  t.SECTION("test case 3");
}

In fact, Catch offers an even simpler way of defining a common set-up (provided you do not need to execute a custom clean-up). You do not have to define any fixture:

TEST_CASE("MyTest", "[mytest]")   // test suite
{  
  int i = 1;
  R res {params};
  // additional set-up

  SECTION("test case 1") {        // test case
    REQUIRE(i == 1); // you have access to i
    REQUIRE(i);
  }
  
  SECTION("test case 2") {        // test case
    use(res);
  }

  SECTION("test case 3") {        // test case
    use(res);
  }
}

Although it may look impossible, it will execute the following sequence:

{
  int i = 1;
  R res {params};
  // additional set-up
  SECTION("test case 1");
}
{
  int i = 1;
  R res {params};
  // additional set-up
  SECTION("test case 2");
}
{
  int i = 1;
  R res {params};
  // additional set-up
  SECTION("test case 3");
}

The documentation in Catch explains how this effect is achieved.

And that’s it for today. My goal was not to promote Catch as the best unit-test framework, but I found it elegant how it handles the common test case set-up in a C++ way. For a full working example, see here.

Advertisement