|
In this document I attempt to explain the decisions made while developing
TUT.
One day I ran into need of unit test framework for C++. So, I've made a small research
and discovered C++Unit, boost::test and a bunch of similar libraries.. Though
they were usable, I was not satisfied with the approach they offered; so I
designed my own variant of unit test framework based on the following
restrictions:
- No C-style macros
- No manual registration for test groups and methods
- No libraries of any kind
- Neutrality to user interface
- No Javisms
Usually C++ Unit test frameworks define a lot of macroses to achieve the goals
other languages have as built-in features: for example, Java is able to show you
the whole exception stack; and C++ cannot. So, to achieve the same (or similar)
results, C++ frameworks often declare a macro to catch any exception and trace
__FILE__ and __LINE__ variables.
The problem is that it turns the C++ code into something that is hard to read, where
"hard to read" actually means "hard to maintain".
Macros don't recognize namespace borders, so a simple macro can expand in the user
code into something unexpected. To avoid this, we have to give macros unique
prefixes, and this, in turn, reduces code readability even more.
From bad to worse, C-style macros can't handle modern C++ templates, so comma
separated template arguments will break the macro, since preprocessor will
handle the template as two arguments (separated by the comma used in the
template) to this macro.
And the final contra for macros is that even if used they cannot achieve the same
usability level as the native language tools; for example, macros cannot
generate a full stack trace (at least, in a platform-independent manner). So it
looks like we loose readability and get almost nothing for this.
See also Bjarne Stroustrup notices about macros harmness:
So,
what's wrong with using macros?
In JUnit (Java-based Unit Tests framework) reflection is used to recognize
user-written test methods. C++ has no reflection or similar mechanism, so user
must somehow tell the framework that "this, this and that" methods should be
considered as test methods, and others are just helpers for them.
The same can be said about test groups, which have to be registered in test runner
object.
Again, most C++ frameworks introduce macros or at least methods to register a
freestanding function or method as a test instance. I find writing redundant
code rather annoying: I have to write test itself and then I have to write more code
to mark that my test is a test. Even more, if I forget to register the method,
nothing will warn me or somehow indicate that I have not done what I should.
Most of C++ frameworks require building a library that user must link to test
application to be able to run tests. The problem is that various platforms imply
different ways for building libraries. One popular C++ framework has more than
60% bugs in their bug database that sound like "cannot build library on platform
XXX" or "built library doesn't work on platform YYY".
Besides, some platforms has complexities in library memory management (e.g.
Win32).
Some frameworks provide predefined formatters for output results, for example
CSV or XML. This restricts users in test results presentation options. What if a
user wants some completely different format? Of course, he can implement his own
formatter, but why frameworks provide useless formatters then?
The ideal test framework must do only one thing: run tests. Anything beyond that is
the responsibility of the user code. Framework provides the test results, and
the user code then represents them in any desired form.
Most implementors of C++ test frameworks know about JUnit and inspired by
this exciting tool. But, carelessly copying a Java implementation to C++, we can
get strange and ugly design.
Rather obvious example: JUnit has methods for setting up a test (setUp) and for
cleaning after it (tearDown). I know at least two C++ frameworks that have these
methods with the same semantics and names. But in C++ the job these methods do is
the responsibility of constructor and destructor! In Java we don't have
guaranteed destruction, so JUnit authors had to invent their own replacement
for it - tearDown(); and it was natural then to introduce constructing
counterpart - setUp(). Doing the same in C++ is absolutely redundant
C++ has its own way of working, and whenever possible, I am going to stay at the C++
roads, and will not repeat Java implementation just because it is really good for
Java.
The solution is that simple: just
do not use any macros. I personally never needed a macro during
development.
Since C++ has no reflection, the only way to mark a method as a test is to give it a kind
of predefined name.
There would be a simple solution: create a set of virtual methods in test object base
class, and allow user to overwrite them. The code might look like:
struct a_test_group : public test_group { virtual void test1() { ... } virtual
void test2() { ... } };
Unfortunately, this approach has major drawbacks:
- It scales badly. Consider, we have created 100 virtual test methods in a test
group, but user needs 200. How can he achieve that? There is no proper way.
Frankly speaking, such a demand will arise rarely (mostly in
script-generated tests), but even the possibility of it makes this kind of
design seemingly poor.
- There is no way to iterate virtual methods automatically. We
would end up writing code that calls test1(), then test2(), and so on, each
with its own exception handling and reporting.
Another possible solution is to substitute reflection with a dynamic loading.
User then would write static functions with predefined names, and TUT would use
dlsym()/GetProcAddress() to find out the implemented tests.
But I rejected the solution due to its platform and library operations
dependencies. As I described above, the library operations are quite different
on various platform.
There was also an idea to have a small parser, that can scan the user code and
generate registration procedure. This solution only looks simple; parsing
free-standing user code can be a tricky procedure, and might easily overgrow twelve TUTs
in complexity.
Fortunately, modern C++ compilers already have a tool that can parse the user code
and iterate methods. It is compiler template processing engine. To be more
precise, it is template specialization technique.
The following code iterates all methods named test<N> ranging from n to 0,
and takes the address of each:
template <class Test,class Group,int n>
struct tests_registerer
{
static void reg(Group& group)
{
group.reg(n,&Test::template test<n>);
tests_registerer<Test,Group,n-1>::reg(group);
}
};
template<class Test,class Group>
struct tests_registerer<Test,Group,0>
{
static void reg(Group&){};
};
...
test_registerer<test,group,100>.reg(grp);
This code generates recursive template instantiations until it reaches
tests_registerer<Test,Group,0> which has empty specialization.
There the recursion stops.
The code is suitable for our needs because in the specialization preference is
given to the user-written code, not to the generic one. Suppose we have a default
method test<N> in our test group, and the user-written specialization
for test<1>. So while iterating, compiler will get the address of the
default method for all N, except 1, since user has supplied a special version of
that method.
template<int N>
void test() { };
...
template<>
void test_group::test<1>() { // user code here }
This approach can be regarded as kind of compile-time virtual functions, since
the user-written methods replace the default implementation. At the same time,
it scales well - one just has to specify another test number upper bound at compile
time. The method also allows iteration of methods, keeping code compact.
Since we dig into the template processing, it is natural to not build any
libraries, therefor this problem mostly disappeares. Unfortunately, not
completely: our code still needs some central point where it could register
itself. But that point (singleton) is so small that it would be an overkill to
create library just to put there one single object. Instead, we assume that the
user code will contain our singleton somewhere in the main module of test
application.
Our code will perform only minimamum set of tasks: TUT shall run tests. But we still
need a way to adapt the end-user presentation requirements. For some of users it
would be enough to see only failed tests in listing; others would like to see the
complete plain-text report; some would prefer to get XML reports, and some would
not want to get any reports at all since they draw their test execution log in GUI
plugin for an IDE.
"Many users" means "many demands", and satisfying all of them is quite a hard task.
Attempt to use a kind of silver bullet (like XML) is not the right solution, since
user would lack XML parser in his environment, or just would not want to put it into
his project due to integration complexities.
The decision was made to allow users to form their reports by themselfs. TUT will
report an event, and the user code will form some kind of an artefact based on this
event.
The implementation of this decision is interface tut::callback. The user code
creates a callback object, and passes it to the runner. When an appropriate event
occures, the test runner invokes callback methods. User code can do anything,
from dumping test results to std::cout to drawing 3D images, if desired.
Initially, there were plans to make TUT traits-based in order not to restrict
it with STL only, but have a possibility to use other kinds of strings
(TString, CString), containers and intercepted exceptions.
In the current version, these plans are not implemented due to relative
complexity of the task. For example, the actual set of operations can be quite
different for various map implementations and this makes writing generic code
much harder.
Thus so far TUT is completely STL-based, since STL is the only library existing
virtually on every platform.
|