Unit testing frameworks

In a recent conversation with Chris it was brought up that the LLVM project doesn't have a way to incorporate fine-grained unit tests (in other words, tests that exersize the members of a single C++ class). I was wondering what is the general feeling about adding this to LLVM.

Let me discuss some of the details:

First, there is the question of a test framework. One possibility is to create a custom framework based on the support classes already defined within LLVM; This has the benefit of not adding an extra dependency, minimizes potential license incompatibilities, and may perhaps lead to a more integrated testing experience. The downside is that writing a decent framework is a lot of work, and you aren't going to get something as full-featured or mature as some of the various open-source C++ testing frameworks out there.

The other alternative is to adopt one of those frameworks. Based on my research, I would go with either UnitTest++ or GoogleTest, as these are the most modern and supported test frameworks available. (There are many others, but the two I mentioned were written with the full knowledge of all of the others and with the benefit of hindsight.)

I've been using GoogleTest (GitHub - google/googletest: GoogleTest - Google Testing and Mocking Framework) in my own project and I am pretty happy with it. Tests are easy to write without a lot of boilerplate, and it supports automatic test discovery, test fixtures, customized assertion predicates, customizable stringification of application-specific types, debugger integration, and so on. I use the svn 'external' feature to incorporate a link to GoogleTest, so that it gets checked out automatically as a result of checking out my project's sources. I didn't bother with trying to get autoconf's embedded project support to work, I just included the source files directly in my own library's makefile and that worked fine.

A third possibility is to write an LLVM-specific framework that is effectively clone of one of the existing frameworks, but using LLVM's infrastructure as a basis. Its unlikely that LLVM will need more than 50% of the features of the open-source solutions, so it would not be necessary to clone the entire framework, merely a simplified version of it.

In any case, I would imagine that a new directory, 'unit' be added to the top level of the project, and appropriate makefile targets written so that 'make check' (or whatever target name you like) would build and run the unit tests. Only the executables defined in the 'unit' directory would have any dependency on the test framework.

It may be that the existing test infrastructure is adequate for LLVM's needs. However, one of my interests is in improving various container classes in ADT, and the current test infrastructure doesn't serve that use case very well IMHO.

-- Talin

You may want to look at Cppunit as well.
http://cppunit.sourceforge.net/cppunit-wiki

- Sanjiv