CommonLisp Testing: announcing Lisp-Unit2

I recently published lisp-unit2 which is a major refactoring of the lisp-unit library. I am one of the biggest public users of lisp-unit (accounting for 15 of the 28 libraries depending on it in quicklisp). I also have some very large internal test suites for non-public applications. I have been using lisp-unit regularly for at least the past 5 years and perhaps longer. After such extended use, I finally had an issue that made me start refactoring lisp-unit. (I could not find where a “compiler error” message was being printed from, which ended up not even being lisp-unit’s fault, but lisp-unit didn’t help me find it). This compounded with years of “wouldn’t it be nice if lisp-unit did” made me finally go about fixing all my gripes.

The things I most wanted to change were:

  • Lots of flags that are not obvious, that do odd things. I always ended up setting all of them to true from their default state, so having 3-4 flags that always need to be remembered to be set was strange
  • No go-to-definition. I always want to be able to go to definition from the test name and couldn’t
  • Tags/Suites. I always had wrapper macros that allowed me to organize my tests into suites. Eventually lisp-unit added tags, but the syntax was a bit odd and the usage was not always obvious.
  • Better Signal Testings. I like conditions and frequently need to test protocols that use them. Lisp-unit was deficient here
  • No easy context control. Databases and the like all had to be handled by wrapping the test body in layers of with-resource-context style macros
  • Tests were not compiled till they were run, so compiler errors/warnings often got caught later than needed and were harder to track down than they should be

All of these and more were accomplished in lisp-unit2. (Check out the README for documentation and more details).

I gathered some statistics about which test libraries were being used the most as judged by number of dependencies in quicklisp. The libraries from the CLiki Page on Test Frameworks were cross referenced with quicklisp (because some are defunct). The number listed is the number of systems in quicklisp which require one the testing libraries.

LIFT (48)
STEFIL (46) - there seems to be two branches
-- HU.DWIM.STEFIL (30)
-- STEFIL (16)
FIVEAM (45)
LISP-UNIT (28)
RT (25)
EOS (20)
CL-TEST-MORE (14)
UNIT-TEST (9)
PTESTER (5)
XLUNIT (3)
CLUNIT (1)
TESTBILD (1)

After later analysis, I could probably have gotten by writing extensions to STEFIL, but with my already long-term investment in lisp-unit. I don’t think I spent more time updating lisp-unit than I would have learning STEFIL, writing the extensions I needed and converting all my tests. Hopefully lisp-unit2 fills a need that others have as well.

The most interesting things for me while refactoring were handling the dynamic contexts and using signals to orchestrate output, debugging, results collection etc. These two abstractions combined really nicely to offer all the flexibility I wanted in output and debugging. I think that these abstractions will also allow lisp-unit2 to be highly extensible in an easy to manage way. I also liked that it was easy to write a meta-test-suite for lisp-unit2 in lisp-unit2 using these systems.