Set-up and tear-down

Recently as part of program run-time performance optimization effort, I was scanning through the code for occurrences of keyword new. The goal was to find unnecessary memory allocations. I was surprised to find most of news in the unit test module. Not that there was any particular need for memory allocation: it is just that the framework that was chosen (a fairly popular one) enforces on the programmers bad practices of manually controlling the life-time of their objects.

I assume you all know what the “RAII” approach to structuring program code is. The programmers in my team know it as well, but they cannot use it effectively in our unit-test framework. Typically when you define similar test cases (e.g., when testing different member functions of the same class), you want to group them (into a suite) and execute the same set-up code before each test case, and possibly, run the same clean-up (or tear-down) code after each test case. This pattern can be illustrated as follows:

{
  test_case_1()
  {
    // set up c
    // test1(c)
    // tear down c
  }

  test_case_2()
  {
    // set up c
    // test2(c)
    // tear down c
  }

  test_case_3()
  {
    // set up c
    // test3(c)
    // tear down c
  }
}

In order not to repeat the same set-up and tear-down code in each test case, the natural expectation of the programmers is to have a way to define set-up and tear-down code only once, and have the framework inject it into every test case auto-magically.

Our framework tries to follow the design implemented in JUnit. Apparently JUnit is treated as some point of reference. And the author of our framework tries to be as close as possible to the original, which includes importing some of Java habits. Thus, you define a test suite by declaring a class which inherits from some special test suite base; in order to provide a clean-up and tear-down code, you have to override virtual member functions setUp() and tearDown():

class MyTest : public UTFramework::TestSuite
{
public:
  void setUp() override;
  void tearDown() override;

  test_case_1();
  test_case_2();
  test_case_3();
}

Once you do it, and inform the framework, that it should consider MyTest when running all tests, and that test_case_1, test_case_2 and test_case_3 are in fact tests, it will perform the following sequence of operations:

MyTest t;
t.setUp();
t.test_case_1();
t.tearDown();
t.setUp();
t.test_case_2();
t.tearDown();
t.setUp();
t.test_case_3();
t.tearDown();

This is more-less what we wanted. But suppose the test cases all need to use a resource, which is represented by a RAII-like class: the resource R is initialized in constructor and cleaned up in the destructor. We need to initialize it before each test case, and unfortunately setUp() and test_case_1() can only communicate via member data. So, what the programmers are forced to do is something like this:

class MyTest : public UTFramework::TestSuite
{
  R* res_ = nullptr;

public:
  void setUp() override {
    res_ = new R{params};
  }
  void tearDown() override {
    delete res_;
  }

  test_case_1() {
    use (*res_);
  }
  test_case_2() {
    use (*res_);
  }
  test_case_3() {
    use (*res_);
  }
}

Quite horrible, isn’t it? Is it exception-safe? It is impossible to tell what the framework does. My experience is that frameworks that force programmers to resort to such things hardly ever pay attention to exception safety, and never bother to document what they do upon exception.

We can mitigate the problem by employing a smart pointer, or even better: by employing boost::optional:

class MyTest : public UTFramework::TestSuite
{
  boost::optional<R> res_;

public:
  void setUp() override {
    res_.emplace(params);
  }
  void tearDown() override {
    res_ = boost::none;
  }

  test_case_1() {
    use (*res_);
  }
  test_case_2() {
    use (*res_);
  }
  test_case_3() {
    use (*res_);
  }
}

But this is still far from ideal. It does not respect basic C++ idioms: that members are initialized in constructors and cleared up in destructors. We have to deal with the fact that res_ may not contain a value, at least technically. A far better solution would be to enforce the following usage pattern on the programmers:

class MyTest
{
  R res_;

public:
  MyTest() : res{params} {}
  ~MyTest() {} // not really needed in our case

  test_case_1() {
    use (res_);
  }
  test_case_2() {
    use (res_);
  }
  test_case_3() {
    use (res_);
  }
}

And have the framework execute our test cases in the following pattern:

{
  MyTest t;          // <-- set up
  t.test_case_1();
}                    // <-- tear down
{
  MyTest t;
  t.test_case_2();
}
{
  MyTest t;
  t.test_case_3();
}

And in fact this is what a couple of decent C++ unit-test frameworks (like Boost.Test) are doing. For an illustration, I chose framework Catch. It allows you to define a common set-up and tear-down functions in a manner similar to the one above. You define a class with custom data, default constructor representing a set-up procedure and destructor representing the tear-down procedure:

struct Fx
{
  int i = 1;
  R res {params};
  Fx() { /* set-up */ }
  ~Fx() { /* tear-down */ }
};

Because we are only initializing the member variables, we do not even need to define the default constructor or destructor, it is enough that we initialize the members in-place (which is an option in C++11). Now, we can use such fixture in the tests:

TEST_CASE_METHOD(Fx, "MyTest", "[mytest]") // test suite
{  
  SECTION("test case 1") {                 // test case
    REQUIRE(i == 1); // you have access to i
    REQUIRE(i);
  }
  
  SECTION("test case 2") {                 // test case
    use(res);
  }

  SECTION("test case 3") {                 // test case
    use(res);
  }
}

A couple of macros do the magic. This generates the following sequence of calls (in pseudo-code):

{
  _DerivedFromMyTest t;
  t.SECTION("test case 1");
}
{
  _DerivedFromMyTest t;
  t.SECTION("test case 2");
}
{
  _DerivedFromMyTest t;
  t.SECTION("test case 3");
}

In fact, Catch offers an even simpler way of defining a common set-up (provided you do not need to execute a custom clean-up). You do not have to define any fixture:

TEST_CASE("MyTest", "[mytest]")   // test suite
{  
  int i = 1;
  R res {params};
  // additional set-up

  SECTION("test case 1") {        // test case
    REQUIRE(i == 1); // you have access to i
    REQUIRE(i);
  }
  
  SECTION("test case 2") {        // test case
    use(res);
  }

  SECTION("test case 3") {        // test case
    use(res);
  }
}

Although it may look impossible, it will execute the following sequence:

{
  int i = 1;
  R res {params};
  // additional set-up
  SECTION("test case 1");
}
{
  int i = 1;
  R res {params};
  // additional set-up
  SECTION("test case 2");
}
{
  int i = 1;
  R res {params};
  // additional set-up
  SECTION("test case 3");
}

The documentation in Catch explains how this effect is achieved.

And that’s it for today. My goal was not to promote Catch as the best unit-test framework, but I found it elegant how it handles the common test case set-up in a C++ way. For a full working example, see here.

Advertisements
This entry was posted in programming and tagged , , . Bookmark the permalink.

5 Responses to Set-up and tear-down

  1. Marco Arena says:

    Curiously in Google-Test fixtures one case use both setUp/TearDown and Ctor/Dtor. I think – as you said – setUp/TearDown pair is maintained for compatibility with JUnit or similar frameworks.

    Anyway, the common set-up facility provided by Catch is really nice!

  2. Krzysiek says:

    As I started reading the article I sympathized with you and you colleagues for having to deal with verbose unit testing framework that requires writing a lot of boilerplate code. In my previous job we also used something based on JUnit and I cursed it.

    Incidentally, my first assignment in my next job was to do some research and recommend a unit test framework. I chose Catch, which makes writing unit tests easy and hassle-free, if not pleasant. At some point I tried to use it together with Google Mock, managed to get it work, but did not really need it.

  3. cieplik says:

    As to resource handling, you should be able to make use of RAII in any framework that claims to be JUnit-compatible – see this short article on JUnit creating new test suite instance for each test execution. So, the Catch sample you present could look like this (for UTFramework == cppunit):

    struct MyTest : UTFramework::TestSuite
    {
      void test_case_1();  // will get its own i and R
      void test_case_2();  // will get its own i and R
    
    private:
      int i = 1;
      R res{params};
    }
    

    If more sophisticated initialization / cleanup is needed, ctor / dtor can be used instead of setUp / tearDown. Of course it doesn’t solve the problem with UTFramework’s intrusiveness but at least makes it a bit more decent ;-).

  4. viboes says:

    An alternative is to *reset* the resource on the `tearDown`. What does *reset* means?
    Call the destructor and then use placement new on the storage. In this way you always have a fresh default resource on the setup.

    Note that not all the resources have a `none` to reset the instance, but most of them have a default constructor.

    I’ve found this reset function quite useful to reset also some global variables on exceptional cases 😦

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s