Unit Tests

1. Introduction

Recently, a set of regression unit tests has been added to Cyrus. This document explains the purpose implementation of those unit tests, and gives an example of how to add more unit tests (because there are never enough unit tests!).

2. What Is A Unit Test?

The definition on Wikipedia sheds some light:

...unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application.

In other words, unit testing is about verifying that small pieces of code, like individual functions, modules, or classes, work in isolation. It is not about testing the system as a whole.

The tests implemented here are also regression tests, which in Wikipedia's words means:

Regression testing is any type of software testing that seeks to uncover software errors after changes to the program (e.g. bugfixes or new functionality) have been made, by retesting the program. The intent of regression testing is to assure that a change, such as a bugfix, did not introduce new bugs.

In other words, the tests are designed to be easy to run and to work out fully automatically whether they have passed or failed, so that they can be run usefully by people who didn't write them.

3. Running The Tests

This section takes you through the process of running Cyrus' unit tests.

3.1. Setting Up The Machine

Cyrus' unit tests are all located in a new directory, cyrus-imapd/cunit/. They're written in C, like the remainder of Cyrus, and use the CUnit library from SourceForge, with some home grown wrappers and other improvements to make our lives easier.

Your first step is step is to ensure that the CUnit library (including the headers) is installed. Some modern operating systems already have CUnit, for example on Ubuntu you can just do:

me@ubuntu> sudo apt-get install libcunit1-dev

Alternately, you can download the CUnit source, build it and install it. It's not a complicated or difficult library, this shouldn't take long. When you've done, install it in /usr/include and /usr/lib.

3.2 Configure Script

Because of the dependency on the CUnit library, the tests are disabled by default; this means you need enable them with an option to the configure script:

me@mybox> ./configure --enable-unit-tests
...
checking for CU\_initialize\_registry in -lcunit... yes
checking CUnit/CUnit.h usability... yes
checking CUnit/CUnit.h presence... yes
checking for CUnit/CUnit.h... yes
...

3.3 Make

First you need to build Cyrus itself, using the traditional all: target.

me@mybox> make all
...

Then, use the new check: target to build and run the unit tests.

me@mybox> make check
 cd . && /bin/bash /home/me/cyrus-imapd/missing --run automake-1.11 --foreign Makefile
 cd . && /bin/bash ./config.status Makefile depfiles (a)
config.status: creating Makefile
config.status: executing depfiles commands
...
make[3]: Entering directory \`/home/me/cyrus-imapd'
make[3]: \`sieve/test' is up to date.
cunit/cunit.pl --project cunit/default.cunit --generate-wrapper cunit/mboxname.testc (b)
gcc -DHAVE\_CONFIG\_H ... -c -o cunit/mboxname.o cunit/mboxname.testc-cunit.c
rm -f cunit/mboxname.testc-cunit.c
/bin/bash ./libtool --tag=CC --mode=link gcc -fPIC -g -O2 -o
cunit/unit cunit/unit.o ... lib/libcyrus\_min.la ... (c)
...
Running unit tests (d)

    CUnit - A Unit testing framework for C - Version 2.1-0
    http://cunit.sourceforge.net/

...
Suite: mboxname (e)
  Test: dir\_hash\_c ... passed
  Test: to\_parts ... passed
  Test: to\_userid ... passed
  Test: to\_usermbox ... passed
...
--Run Summary: Type      Total     Ran  Passed  Failed (f)
               suites       34      34     n/a       0
               tests       323     323     323       0
               asserts 1079745 1079745 1079745       0
make[1]: Leaving directory `/home/me/cyrus-imapd/cunit'

Let's take a closer look at what's happening here.

  1. The check: target causes automake to re-run itself. This is normal automake behaviour. Note that the older build system used to run make recursively in sub-directories, the newer automake-based system builds everything from the top directory.

  2. The cunit/ directory contains a number of C source files (called, for reasons too complicated to explain here, whatever.testc) with test code in them. For each of those, a small wrapper C source file is generated and then compiled into an object file.

  3. Finally, all the compiled object files are linked into an executable, with a main() routine from unit.c, and a number of libraries and object files from other parts of the Cyrus tree.

  4. The resulting executable is then run.

  5. The test executable runs all the built tests one by one, telling us which ones passed and which ones failed as it runs them. You can also run it manually with the name of a test as an argument, and it will run only the named test.

  6. At the end, the text executable prints a summary of how many tests it ran and how many passed and failed. The key thing to look at here is the rightmost column, it should be all zero.

3.4 Using Valgrind

Some failure modes are subtle, and cannot be detected in the C code itself; this is where the Valgrind program comes in very handy. It detects buffer overruns and memory leaks and various other kinds of subtle errors.

To run the unit tests with Valgrind, use the new valgrind: target.

me@mybox> make valgrind
...
valgrind --tool=memcheck --leak-check=full ./unit -v (a)
==2999== Memcheck, a memory error detector
==2999== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==2999== Using Valgrind-3.6.0.SVN-Debian and LibVEX; [...]
==2999== Command: ./unit -v
==2999==
...
--Run Summary: Type      Total     Ran  Passed  Failed   (b)
               suites        9       9     n/a       0
               tests        51      51      50       1
               asserts     474     474     473       1
...
==2999== HEAP SUMMARY:   (c)
==2999==     in use at exit: 4,489 bytes in 134 blocks
==2999==   total heap usage: 715 allocs, 581 frees, 352,763 bytes allocated
==2999==
==2999== 4 bytes in 1 blocks are definitely lost in loss record 3 of 50
==2999==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
==2999==    by 0x44A0CA: xmalloc (xmalloc.c:57)
==2999==    by 0x4399D8: strconcat (util.c:631)
==2999==    by 0x40C059: test_uncast_null (strconcat.c:51)
==2999==    by 0x61B32A9: ??? (in /usr/lib/libcunit.so.1.0.1)
==2999==    by 0x61B36ED: ??? (in /usr/lib/libcunit.so.1.0.1)
==2999==    by 0x61B3827: CU_run_all_tests (in /usr/lib/libcunit.so.1.0.1)
==2999==    by 0x4066CC: run_tests (unit.c:144)
==2999==    by 0x406806: main (unit.c:283)
==2999==
...

Here's an explanation of what's happening in the example.

  1. The test executable is run as before, but using the valgrind program. The first thing we see is Valgrind's banner message.

  2. The test executable proceeds as normal and eventually emits it's run summary, then exits.

  3. After the test executable exits, Valgrind checks for memory leaks and prints both a summary of all leaks and a stack trace showing where each block of leaked memory was allocated.

I'd just like to say that I love Valgrind and I think it's immensely useful. I would have made running the tests under Valgrind the only option for the check: target, except that Valgrind is not available on all of Cyrus' supported platforms.

3.5 The Tests Are Failing

So you've noticed that some of the tests are failing. Let me make the guiding principle of unit testing as clear as possible: THE UNIT TESTS SHOULD NOT FAIL. All of the tests are designed to pass all the time, in everyone's environment. The unit tests are run automatically every twelve hours on the Cyrus Continuous Integration server, and a failing test fails the whole build and makes people unhappy.

There are a few rules which you should follow to help us all get the most benefit out of unit testing

  • If you see a test failing, investigate it.

  • If you can't investigate, complain on the mailing list or raise a bug so that somebody else can investigate.

  • When writing tests, write them to work in all environments and all combinations of configure script options. It's ok to have a test which is empty in some circumstances; it's not ok to have a test that fails.

  • When adding code, write new tests for the new code.

  • When modifying code, write new tests for the new behaviour.

  • When looking at old code, also take a look at the coverage report and consider writing tests for the existing code.

3.6 Debugging A Test

With the new Cyrus build system, the file cunit/unit is no longer an executable, it's a shell script which sets up some environment variables before running the real executable which is hidden away. This makes debugging a failing test somewhat challenging. The solution is:

me@mybox> ( cd cunit ; libtool --mode=execute gdb --args unit -t crc32 )
...
Reading symbols from /home/me/cyrus-imapd/cunit/.libs/lt-unit...done.
(gdb) list crc32.testc:1
1       /* Unit test for lib/crc32.c */
2       #include "cunit/cyrunit.h"
3       #include "crc32.h"
...
(gdb) break test_map
Breakpoint 1 at 0x44a2f8: file ./cunit/crc32.testc, line 11.
(gdb) run
Starting program: /home/me/cyrus-imapd/cunit/.libs/lt-unit -t -v crc32
[Thread debugging using libthread_db enabled]

    CUnit - A Unit testing framework for C - Version 2.1-0
    http://cunit.sourceforge.net/

Suite: crc32
  Test: map ...
Breakpoint 1, test_map () at ./cunit/crc32.testc:11
11          c = crc32_map(TEXT, sizeof(TEXT)-1);
(gdb)

Note the -t option. This turns off test timeouts, which is very useful for manual debugging.

4. Adding Your Own Tests

Adding your own tests is quite simple. Here's how.

4.1 Where To Put Your Tests

The unit test code in Cyrus is contained in a set of C source files in the cunit directory. For reasons too complex to go into here, these are named whatever.testc instead of the more usual whatever.c. If you look closely, you will see that each of those C source files maps to a "Suite" in CUnit parlance. For example, cunit/glob.testc is listed as the Suite "glob" in CUnit's runtime output.

Typically, each Suite tests a single module or a related set of functions; for example, cunit/glob.testc contains tests for the glob module in lib/glob.c.

So, if you want to add a new test for a module which already has some existing tests, the sensible thing to do is to add a new test to the existing suite. Otherwise, you'll need to add a new Suite.

4.1 Adding A New Suite

Each Suite is a single C source file in the cunit/ directory. Your first step is to create a new C source file. For this example, you'll create a new Suite to test the CRC32 routines which live in lib/crc32.c.

me@mybox> vi cunit/crc32.testc
...

The file should contain something like this.

/* Unit test for lib/crc32.c */
#include "cunit/cyrunit.h"  (a)
#include "crc32.h"  (b)

static void test_map(void)  (c)
{
    static const char TEXT[] = "lorem ipsum";  (d)
    static uint32_t CRC32 = 0x0;
    uint32_t c;  (e)

    c = crc32_map(TEXT, sizeof(TEXT)-1);  (f)
    CU_ASSERT_EQUAL(c, CRC32);  (g)
}

Here's an explanation of what all these bits are for.

  1. You need to include the header "cunit/cyrunit.h", which is a thin Cyrus wrapper around the CUnit's library's header, <CUnit/CUnit.h> with some extra conveniences.

  2. You should also include any headers you need for declarations of the functions which you'll be testing. Note that the Cyrus lib/ and imap/ directories are already in the include path, so any header in there can be included without the directory prefix, e.g. "crc32.h" for lib/crc32.h.

  3. You need to have at least one function which looks like this: it takes no arguments, returns void, and is named test_whatever. It may be static or extern, but I recommend static. Functions with this signature are automatically discovered in the source code by the Cyrus unit test infrastructure, so all you have to do is write the function. Later, a CUnit test named "whatever" will be created automatically for your test_whatever function.

  4. Here's a good place to define the test inputs and expected outputs. Note that for this example you have no idea of the actual correct output. The right thing to do there is to manually calculate the expected result from first principles, or to use a different piece of software which you believe to be working. For this example, let's just use a known incorrect value and see what happens.

  5. Here's a good place for local variables you need during the test.

  6. Call the function under test (crc32_map() in this example) with known inputs, and capture the results in a local variable c.

  7. Compare the actual result in c with the expected result in CRC32. The CU_ASSERT_EQUAL() macro checks that it's two arguments are equal (using an integer comparison), and if they're different it prints a message and records a failure. Note that unlike the libc assert() macro, control will continue even if the assert fails. The CUnit library provides a whole family of similar macros, see the online CUnit documentation for more details.

Now you need to tell the Cyrus build system about your new Suite.

me@mybox> vi Makefile.am
...

You need to add the filename of your new test to the definition of the cunit_TESTS variable.

cunit_TESTS = \
    cunit/aaa-db.testc \
    cunit/annotate.testc \
    cunit/backend.testc \
    cunit/binhex.testc \
    cunit/bitvector.testc \
    cunit/buf.testc \
    cunit/byteorder.testc \
    cunit/charset.testc \
    cunit/crc32.testc \
    cunit/dlist.testc \
    cunit/duplicate.testc \

At this point you should be able to just rebuild and rerun using make check. You can also just rebuild without rerunning by using the command make cunit/unit.

Note that sometimes this doesn't quite work right, and you may be able to work around this problem using the command rm cunit/default.cunit.

me@mybox> make check
...
../cunit/cunit.pl [...] --add-sources [...] crc32.testc
...
../cunit/cunit.pl [...] --generate-wrapper crc32.testc
gcc -c [...] -g -O2 .cunit-crc32.c
gcc [...] -o unit [...] .cunit-crc32.o ...
Running unit tests

    CUnit - A Unit testing framework for C - Version 2.1-0
    http://cunit.sourceforge.net/

...
Suite: crc32
  Test: map ... FAILED
    1. crc32.testc:12  - CU_ASSERT_EQUAL(c=1926722702,CRC32=0)

Note how the test failure told us which in source file and at what line number the failure occurred, and what the actual and expected values were. Let's go and fix that up now.

static const char TEXT[] = "lorem ipsum";
static uint32\_t CRC32 = 0x72d7748e;

Re-run make check and you'll see your test being rebuilt and rerun, and this time passing.

me@mybox> make check
...
../cunit/cunit.pl [...] --generate-wrapper crc32.testc
gcc -c [...] -g -O2 .cunit-crc32.c
gcc [...] -o unit [...] .cunit-crc32.o
...
Running unit tests

    CUnit - A Unit testing framework for C - Version 2.1-0
    http://cunit.sourceforge.net/

...
Suite: crc32
  Test: map ... passed

4.2 Adding A Test To A Suite

Adding a new test to an existing test is easy: all you have to do is add a new function to an existing C source file in the cunit/ directory. As an example, let's add a test for the crc_iovec() function.

me@mybox> vi cunit/crc32.testc
...

static void test_iovec(void)  (a)
{
    static const char TEXT1[] = "lorem";  (b)
    static const char TEXT2[] = " ipsum";
    static uint32_t CRC32 = 0x72d7748e;
    uint32_t c;  (c)
    struct iovec iov[2];

    memset(iov, 0, sizeof(iov));  (d)
    iov[0].iov_base = TEXT1;
    iov[0].iov_len = sizeof(TEXT1)-1;
    iov[1].iov_base = TEXT2;
    iov[1].iov_len = sizeof(TEXT2)-1;

    c = crc32_iovec(iov, 2);  (e)
    CU_ASSERT_EQUAL(c, CRC32);  (f)
}

Here's an explanation of what all these bits are for.

  1. Your new test function should look like this: it takes no arguments, returns void, and is named test_whatever. It may be static or extern, but I recommend static. Functions with this signature are automatically discovered in the source code by the Cyrus unit test infrastructure, so all you have to do is write the function. Later, a CUnit test named "whatever" will be created automatically for your test_whatever function. Note that the opening curly brace must be on the next line or the unit test infrastructure will not find the function.

  2. Here's a good place to define the test inputs and expected outputs.

  3. Here's a good place for local variables you need during the test.

  4. Here you set up the input conditions for the function under test.

  5. Call the function under test with your known inputs, and capture the results in a local variable, here c.

  6. Compare the actual result in c with the expected result in CRC32. The CU_ASSERT_EQUAL() macro checks that it's two arguments are equal (using an integer comparison), and if they're different it prints a message and records a failure. Note that unlike the libc assert() macro, control will continue even if the assert fails. The CUnit library provides a whole family of similar macros, see the online CUnit documentation for more details.

Now run make check and you'll see your test being built and run.

me@mybox> make check
...
../cunit/cunit.pl [...] --generate-wrapper crc32.testc
gcc -c [...] -g -O2 .cunit-crc32.c
gcc [...] -o unit [...] .cunit-crc32.o
...
Running unit tests


     CUnit - A Unit testing framework for C - Version 2.1-0
     http://cunit.sourceforge.net/

...
Suite: crc32
  Test: map ... passed
  Test: iovec ... passed

4.3 Suite Setup And Teardown

Sometimes the behaviour of the functions under test depend on external influences such as environment variables, global variables, or the presence of certain files.

These kinds of functions need special treatment to ensure that their behaviour is locked down during the running of your tests. Otherwise, all sorts of strange behaviour may confuse the results of the tests. For example, a test might succeed the first time it's run in a given directory and fail the next time. Or a test might succeed when run by the author of the test but fail when run by another user.

CUnit provides a special arrangement which helps you in such cases: the suite initialisation and cleanup functions. These are two functions that you write and which live in the suite source. They are called from CUnit respectively before any of the tests in the suite is run, and after all tests from that suite are run.

Here's how to use them. The suite setup function should set up any global state that the functions under test rely on, in such a way that their state is predictable and always the same no matter who runs the test or when or how many times. Similarly the suite teardown function should clean up any state which might possibly interfere with other test suites. Note that some suites will need a setup function but not necessarily a teardown function.

Adding these functions is very easy: you just write functions of the appropriate signature (names, arguments and return type) and the Cyrus unit test infrastructure will automatically discover them and arrange for them to be called. The functions should look like (actual example taken from cunit/mboxname.testc) this:

static enum enum_value old_config_virtdomains;

static int set_up(void)
{
    old_config_virtdomains = config_virtdomains;
    config_virtdomains = IMAP_ENUM_VIRTDOMAINS_ON;
    return 0;
}

static int tear_down(void)
{
    config_virtdomains = old_config_virtdomains;
    return 0;
}

The functions should return 0 on success, and non-zero on error. They must not call and CU_* functions or macros.

Good luck and good testing!