# Writing Postfix unit tests

## Overview

This document covers Ptest, a simple unit test framework that was introduced with Postfix version 3.8. It is modeled after Go tests, with primitives such as ptest_error() and ptest_fatal() that report test failures, and PTEST_RUN() that supports subtests.

Ptest is light-weight compared to more powerful frameworks such as Gtest, but it avoids the need for adding a large Postfix dependency (a dependency that would not affect Postfix distributors, but developers only).

- Simple example

- Testing one function with TEST_CASE data

- Testing functions with subtests

- Suggestions for writing tests

- Ptest API reference

## Simple example

Simple tests exercise one function under test, one scenario at a time. Each scenario calls the function under test with good or bad inputs, and verifies that the function behaves as expected. The code in Postfix `mymalloc_test.c` file is a good example.

After some `#include` statements, the file goes like this:

>      27 typedef struct PTEST_CASE {
>      28     const char *testname;         /* Human-readable description */
>      29     void    (*action) (PTEST_CTX *, const struct PTEST_CASE *);
>      30 } PTEST_CASE;
>      31
>      32 /* Test functions. */
>      33
>      34 static void test_mymalloc_normal(PTEST_CTX *t, const PTEST_CASE *tp)
>      35 {
>      36     void   *ptr;
>      37
>      38     ptr = mymalloc(100);
>      39     myfree(ptr);
>      40 }
>      41
>      42 static void test_mymalloc_panic_too_small(PTEST_CTX *t, const PTEST_CASE *tp)
>      43 {
>      44     expect_ptest_log_event(t, "panic: mymalloc: requested length 0");
>      45     (void) mymalloc(0);
>      46     ptest_fatal(t, "mymalloc(0) returned");
>      47 }
>     ...     // Test functions for myrealloc(), mystrdup(), mymemdup().
>     260
>     261 static const PTEST_CASE ptestcases[] = {
>     262     {"mymalloc + myfree normal case", test_mymalloc_normal,
>     263     },
>     264     {"mymalloc panic for too small request", test_mymalloc_panic_too_small,
>     265     },
>     ...     // Test cases for myrealloc(), mystrdup(), mymemdup().
>     306 };
>     307
>     308 #include <ptest_main.h>

To run the test:

>     $ make test_mymalloc
>     ... compiler output...
>     LD_LIBRARY_PATH=/path/to/postfix-source/lib ./mymalloc_test
>     RUN  mymalloc + myfree normal case
>     PASS mymalloc + myfree normal case

>     RUN  mymalloc panic for too small request
>     LOG  (expected) panic: mymalloc: requested length 0
>     PASS mymalloc panic for too small request

>     ... results for myrealloc(), mystrdup(), mymemdup()...

>     mymalloc_test: PASS: 22, SKIP: 0, FAIL: 0

This simple example already shows several key features of the ptest framework.

- Each test is implemented as a separate function (`test_mymalloc_normal()`, `test_mymalloc_panic_too_small()`, and so on). These functions take two arguments: the first argument points to test infrastructure, and the second argument is not used here but will feature in a later example.

- The first test verifies 'normal' behavior: it verifies that `mymalloc()` will allocate a small amount of memory, and that `myfree()` will accept the result from `mymalloc()`. When the test is run under a memory checker such as Valgrind, the memory checker will report no memory leak or other error.

- The second test is more interesting.

  - The test verifies that `mymalloc()` will call `msg_panic()` when the requested amount of memory is too small. But in this test the `msg_panic()` call will not terminate the process like it normally would. The Ptest framework changes the control flow of `msg_panic()` and `msg_fatal()` such that these functions will terminate their test, instead of terminating the entire test process.

  - The `expect_ptest_log_event()` call sets up an expectation that `msg_panic()` will produce a specific error message; the test would fail if the expectation remains unsatisfied. When an error message is logged as expected, it shows up in output as "`LOG (expected) ...content of expected message...`.

  - The `ptest_fatal()` call at the end of the second test is not needed; this call can only be reached if `mymalloc()` does not call `msg_panic()`. But then the expected panic message would not be logged, and the test would fail anyway.

- The `ptestcases[]` table near the end of the example contains for each test the name and a pointer to function. As we show in a later example, the `ptestcases[]` table can also contain test inputs and expected test outputs.

- The "`#include <ptest_main.h>`" at the end pulls in the code that iterates over the `ptestcases[]` table and logs progress.

- The test run output shows that the `msg_panic()` output in the second test is silenced; only output from unexpected `msg_panic()` or other unexpected `msg(3)` calls would show up in test run output.

## Testing one function with TEST_CASE data

Often, we want to test a module that contains only one function. In that case we can store all the test inputs and expected results in the PTEST_CASE structure.

The examples below are taken from the `dict_union_test.c` file which test the `unionmap` implementation in the file. `dict_union.c`.

Background: a unionmap creates a union of tables. For example, the lookup table "`unionmap:{inline:{foo=one},inline:{foo=two}}`" will return ("`one, two`", DICT_STAT_SUCCESS) when queried with `foo`, and will return (null, DICT_STAT_SUCCESS) otherwise.

First, we present the TEST_CASE structure with additional fields for inputs and expected results.

>      29 #define MAX_PROBE       5
>      30
>      31 struct probe {
>      32     const char *query;
>      33     const char *want_value;
>      34     int     want_error;
>      35 };
>      36
>      37 typedef struct PTEST_CASE {
>      38     const char *testname;
>      39     void    (*action) (PTEST_CTX *, const struct PTEST_CASE *);
>      40     const char *type_name;
>      41     const struct probe probes[MAX_PROBE];
>      42 } PTEST_CASE;

In the PTEST_CASE structure above:

- The `testname` and `action` fields are required. We have seen these already in the simple example above. The other PTEST_CASE fields are specific to the unionmap tests.

- The `type_name` field will contain the name of the table, for example `unionmap:{static:one,inline:{foo=two}}`.

- The `probes` field contains a list of (query, expected result value, expected error code) that will be used to query the unionmap and to verify the result value and error code.

Next we show the test data. Every test calls the same `test_dict_union()` function with a different `unionmap` configuration and with a list of queries with expected results. The implementation of that function follows after the test data.

>     115 static const PTEST_CASE ptestcases[] = {
>     116     {
>     ...
>     120         .testname = "propagates notfound and found",
>     121         .action = test_dict_union,
>     122         .type_name = "unionmap:{static:one,inline:{foo=two}}",
>     123         .probes = {
>     124             {"foo", "one,two", DICT_STAT_SUCCESS},
>     125             {"bar", "one", DICT_STAT_SUCCESS},
>     126         },
>     127     }, {
>     128         .testname = "error propagation: static map + fail map",
>     129         .action = test_dict_union,
>     130         .type_name = "unionmap:{static:one,fail:fail}",
>     131         .probes = {
>     132             {"foo", 0, DICT_ERR_RETRY},
>     133         },
>     ...
>     151     },
>     152 };
>     153
>     154 #include <ptest_main.h>

Finally, here is the `test_dict_union()` function that queries the `unionmap` implementation with test inputs, and verifies that the results are as expected.

>      84 static void test_dict_union(PTEST_CTX *t, const struct PTEST_CASE *tp)
>      85 {
>      86     DICT   *dict;
>      87     const struct probe *pp;
>      88     const char *got_value;
>      89     int     got_error;
>      90
>      91     dict = dict_open(tp->type_name, O_RDONLY, 0);
>      92
>      93     for (pp = tp->probes; pp < tp->probes + MAX_PROBE && pp->query != 0; pp++) {
>      94         got_value = dict_get(dict, pp->query);
>      95         got_error = dict->error;
>      96         if (got_value == 0 && pp->want_value == 0)
>      97             continue;
>      98         if (got_value == 0 || pp->want_value == 0) {
>      99             ptest_error(t, "dict_get(dict, \"%s\"): got '%s', want '%s'",
>     100                         pp->query, STR_OR_NULL(got_value),
>     101                         STR_OR_NULL(pp->want_value));
>     102             break;
>     103         }
>     104         if (strcmp(got_value, pp->want_value) != 0) {
>     105             ptest_error(t, "dict_get(dict, \"%s\"): got '%s', want '%s'",
>     106                         pp->query, got_value, pp->want_value);
>     107         }
>     108         if (got_error != pp->want_error)
>     109             ptest_error(t, "dict_get(dict,\"%s\") error: got %d, want %d",
>     110                         pp->query, got_error, pp->want_error);
>     111     }
>     112     dict_close(dict);
>     113 }

A test run looks like this:

>     $ make test_dict_union
>     ...compiler output...
>     LD_LIBRARY_PATH=/path/to/postfix-source/lib ./dict_union_test
>     ...
>     RUN  propagates notfound and found
>     PASS propagates notfound and found

>     RUN  error propagation: static map + fail map
>     PASS error propagation: static map + fail map
>     ...
>     dict_union_test: PASS: 5, SKIP: 0, FAIL: 0

## Testing functions with subtests

Sometimes it is not convenient to store test data in a PTEST_CASE structure. This can happen when converting an existing test into Ptest, or when the module under test contains functions that need different kinds of test data. The solution is to create a `_test.c` file with the structure as shown below. This can be found in the file `map_search_test.c`, which was converted from an existing test into Ptest.

- One PTEST_CASE structure definition without test data.

       50 typedef struct PTEST_CASE {
       51     const char *testname;
       52     void    (*action) (PTEST_CTX *, const struct PTEST_CASE *);
       53 } PTEST_CASE;

- One test function for each module function that needs to be tested, and one table with test cases for that module function. In this case there is only one module function (`map_search()`) that needs to be tested, so there is only one test function (`test_map_search()`).

       67 #define MAX_WANT_LOG    5
       68
       69 static void test_map_search(PTEST_CTX *t, const struct PTEST_CASE *unused)
       70 {
       71     /* Test cases with inputs and expected outputs. */
       72     struct test {
       73         const char *map_spec;
       74         int     want_return;            /* 0=fail, 1=success */
       75         const char *want_log[MAX_WANT_LOG];
       76         const char *want_map_type_name; /* 0 or match */
       77         const char *want_search_order;   /* 0 or match */
       78     };
       79     static struct test test_cases[] = {
       80         {                               /* 0 */
       81             .map_spec = "type",
       82             .want_return = 0,
       83             .want_log = {
       84                 "malformed map specification: 'type'",
       85                 "expected maptype:mapname instead of 'type'",
       86             },
       87         },
       88         {                               /* 1 */
       89             .map_spec = "type:name",
       90             .want_return = 1,
       91             .want_map_type_name = "type:name",
       92         },
      .. .        // ...other test cases...
      166     };

- In a test function, iterate over its table with test cases, using `PTEST_RUN()` to run each test case in its own subtest.

      184     for (tp = test_cases; tp->map_spec; tp++) {
      185         vstring_sprintf(test_label, "test %d", (int) (tp - test_cases));
      186         PTEST_RUN(t, STR(test_label), {
      187             for (cpp = tp->want_log; cpp < tp->want_log + MAX_WANT_LOG && *cpp; cpp++)
      188                 expect_ptest_log_event(t, *cpp);
      189             map_search_from_create = map_search_create(tp->map_spec);
      ...            // ...verify that the result is as expected...
      ...            // ...use ptest_return() or ptest_fatal() to exit from a test...
      228         });
      229     }
      ...

- Create a `ptestcases[]` table to call each test function once, and include the Ptest main program.

      183 static const PTEST_CASE ptestcases[] = {
      184     "test_map_search", test_map_search,
      185 };
      186
      187 #include <ptest_main.h>

See the file `map_search_test.c` for a complete example.

This is what a test run looks like:

>     $ make test_map_search
>     ...compiler output...
>     LD_LIBRARY_PATH=/path/to/postfix-source/lib  ./map_search_test
>     RUN  test_map_search
>     RUN  test_map_search/test 0
>     LOG  (expected) warning: malformed map specification: 'type'
>     LOG  (expected) warning: expected maptype:mapname instead of 'type'
>     PASS test_map_search/test 0

>     RUN  test_map_search/test 1
>     PASS test_map_search/test 1
>     ....
>     PASS test_map_search

>     map_search_test: PASS: 13, SKIP: 0, FAIL: 0

This shows that the subtest name is appended to the parent test name, formatted as parent-name/child-name.

## Suggestions for writing tests

Ptest is loosely inspired on Go test, especially its top-level test functions and its methods `T.run()`, `T.error()` and `T.fatal()`.

Suggestions for test style may look familiar to Go programmers:

- Use identifiers named `got_xxx` and `want_xxx`. When a test result is unexpected, log the discrepancy as "got <what you got>, want <what you want>".

- Report discrepancies with `ptest_error()` if possible; use `ptest_fatal()` only when continuing the test would produce nonsensical results.

- Where it makes sense use a table with testcases and use `PTEST_RUN()` to run each testcase in its own subtest.

Other suggestions:

- Consider running tests under a memory checker such as Valgrind. Use `ptest_defer()` to avoid memory leaks when a test should terminate early.

- Always test non-error and error cases, to cover all code paths in the function under test.

## Ptest API reference

- Managing test errors

- Managing log events

- Managing test execution

- Miscellaneous test APIs

## Managing test errors

As one might expect, Ptest has support to flag unexpected test results as errors.

`void ptest_error(PTEST_CTX *t, const char *format, ...)`

  Called from inside a test, to report an unexpected test result, and to flag the test as failed without terminating the test. This call can be ignored with `expect_ptest_error()`.

`void ptest_fatal(PTEST_CTX *t, const char *format, ...)`

  Called from inside a test, to report an unexpected test result, to flag the test as failed, and to terminate the test. This call cannot be ignored with `expect_ptest_error()`.

For convenience, Ptest can also report non-error information.

`void ptest_info(PTEST_CTX *t, const char *format, ...)`

  Called from inside a test, to report a non-error condition without terminating the test. This call cannot be ignored with `expect_ptest_error()`.

Finally, Ptest has support to test `ptest_error()` itself, to verify that an intentional error is reported as expected.

`void expect_ptest_error(PTEST_CTX *t, const char *text)`

  Called from inside a test, to expect exactly one `ptest_error()` call with the specified text, and to ignore that `ptest_error()` call (i.e. don't flag the test as failed). To ignore multiple calls, call `expect_ptest_error()` multiple times. A test is flagged as failed when an expected error is not reported (and of course when an error is reported that is not expected with `expect_ptest_error()`).

## Managing log events

Ptest integrates with Postfix `msg(3)` logging.

- Ptest changes the control flow of `msg_fatal()` and `msg_panic()`. When these functions are called during a test, they will terminate the test instead of terminating the entire test process.

- Ptest installs a log event listener to monitor Postfix logging with `msg_info()` etc. Examples of what logging may look like:

      RUN  name-of-test
      LOG  (expected) warning: some text...
      LOG  (expected) panic: some text...
      LOG  (info) some text...
      Unexpected non-info event: some text....
      FAIL name-of-test

Ptest provides the following API to manage log events:

`void expect_ptest_log_event(PTEST_CTX *t, const char *text)`

  Called from inside a test, to expect exactly one `msg(3)` call with the specified text including any `warning`, `error`, `fatal`, or `panic` prefix. To expect multiple events, call `expect_ptest_log_event()` multiple times. A test is flagged as failed when a `warning` or higher-severity message was logged but not expected, or when such a message was expected but not logged.

  There is no need to call `expect_ptest_log_event()` for msg_info() logging; such text will be displayed whether or not it is expected. Allowing arbitrary msg_info() calls makes bug hunting easier.

  There also is no need to match an entire logging message; a substring match will be sufficient. It only needs to be specific enough.

## Managing test execution

Ptest has a number of primitives that control test execution.

`void PTEST_RUN(PTEST_CTX *t, const char *test_name, { code in braces })`

  Called from inside a test, to run the `{ code in braces }` in it own subtest environment. In the test progress report, the subtest name is appended to the parent test name, formatted as parent-name/child-name.

  NOTE: because `PTEST_RUN()` is a macro, the `{ code in braces }` MUST NOT contain a `return` statement; use `ptest_return()` instead. It is OK for `{ code in braces }` to call a function that uses `return`.

`void PTEST_TRY(PTEST_CTX *t, const char *test_name, { code in braces })`

  Called from inside a test, to run the `{ code in braces }` without entering a new subtest environment. The purpose is to continue running the current test after the `{ code in braces }` calls msg_fatal*() or msg_panic(). The `{ code in braces }` should set a variable to indicate that PTEST_TRY() executed "normally".

  NOTE: because PTEST_TRY() is a macro, the `{ code in braces }` MUST NOT contain a `return` statement; use `ptest_return()` instead. It is OK for `{ code in braces }` to call a function that uses `return`.

`NORETURN ptest_skip(PTEST_CTX *t)`

  Called from inside a test, to flag a test as skipped, and to terminate the test without terminating the process. Use this to disable tests that are not applicable for a specific system type or build configuration.

`NORETURN ptest_return(PTEST_CTX *t)`

  Used inside a `{ code in braces }` block to terminate a PTEST_RUN subtest.

`void ptest_defer(PTEST_CTX *t, void (*defer_fn)(void *), void *defer_ctx)`

  Called once from inside a test, to call `defer_fn(defer_ctx)` after the test completes. This is typically used to eliminate a resource leak in tests that terminate the test early.

  NOTE: The deferred function is designed to run outside a test, and therefore it must not call Ptest functions.

## Miscellaneous test APIs

`PTEST_CTX *ptest_ctx_current(void)`

  Returns the PTEST_CTX pointer for the current test or subtest. This can be used to handle a test error in a mock function or helper function that has no PTEST_CTX argument.