Migrating to LazyTest
I've been using the Expectations
testing library since early 2019 -- over six years. I love the expressiveness of
it, compared to clojure.test
, and it exists because "Classic Expectations"
was not compatible with clojure.test
tooling. At work, our tests use a
mixture of clojure.test
and Expectations, but in my open source projects,
I've mostly stuck with clojure.test
for familiarity's sake for contributors.
Being compatible with clojure.test
comes at a price, though. Expectations
uses macros to produce clojure.test
-compatible code under the hood, so it
is limited by the same reporting as clojure.test
and the same potential
problems with tooling that tries to override parts of clojure.test
's
behavior -- namely that multiple tools do not play well together, so I've
had to avoid "improving" the expressiveness or reporting in ways that would
break compatibility with that tooling.
Those limitations have gradually become more frustrating, and I've been
watching Noah Bogart's work on LazyTest
with interest as he has resurrected and evolved
S Sierra's old LazyTest project
that had been archived back in 2017. At first, I was skeptical because it
is not compatible with clojure.test
tooling: it has its own test runner
and that isn't supported by any of the editors, nor the various test runners
that exist for clojure.test
(Cognitect's,
Kaocha), nor
Polylith's incremental test runner.
We rely on the latter heavily at work, but we already use my external test runner to avoid classloader and memory issues, so I figured I could adapt that to also run LazyTest tests and experiment with it at work.
LazyTest provides its own DSL for writing tests -- describe
, it
, expect
--
and it also has extension namespaces that provide a compatibility layer for
most of Expectations and also
Nubank's matcher-combinators,
as well as experimental namespaces that provide a migration path for
clojure.test
and parts of Midje, Qunit, and Xunit.
I migrated a few of our Expectations-based test namespaces over to LazyTest and started using LazyTest's core DSL for new tests. I really like the expressiveness of the DSL and the way it reports test results. I also like the way it handles test fixtures -- as a "context" for tests that can be applied before, after, or around each test or group of tests. LazyTest has multiple reporting options, so you can pick different styles of output for working in the REPL, for running tests locally, and for running tests in CI, for example.
The next step was to see what it would take for Cognitect's test runner to
support LazyTest. As a proof of concept, I hacked up a version that followed
the pattern I'd used in my external test runner. It worked fine but the code
was ugly, and when I talked to Alex Miller
about it, he quite rightly wanted to see a more extensible approach that
would allow others in the community to provide integrations for additional
test libraries and/or runners. There's a
pull request
awaiting review that adds a protocol for test runners and a default
implementation that supports clojure.test
. I've also written a quick
lazytest-runner
implementation, so that test-runner
can run either clojure.test
tests
or LazyTest tests or both. If the PR gets merged, this implementation should
be added directly into LazyTest.
The next step for me was to migrate one of my open source projects to
LazyTest. Since LazyTest does not (currently) support ClojureScript, that
meant HoneySQL was off the table, so I picked
next.jdbc
.
It's tests all used clojure.test
, so I used the LazyTest experimental
compatibility namespace for the migration. You can look at the
pull request for the migration
for the gory details but here's the summary of how little I had to do:
- switch from
clojure -X:test
usage toclojure -M:test:runner
usage - switch Cognitect
test-runner
references to LazyTest's equivalent (deps,-m
namespace for running tests) - switch require of
[clojure.test :refer [deftest is testing]]
to[lazytest.experimental.interfaces.clojure-test :refer [deftest is testing]]
- where I used
clojure.test/use-fixtures :once
, I added a require for[lazytest.core :refer [around set-ns-context!]]
and changed theuse-fixtures
call toset-ns-context!
- where I used
clojure.test/use-fixtures :each
, I added a require for[lazytest.core :refer [around]]
, removed theuse-fixtures
call, and added anaround
context to each test
What's with the -X
/ -M
change, you ask? I like that I can have a :test
alias in my deps.edn
that includes the test runner and the test dependencies,
but uses :exec-fn
to identify the function to run -- so that the same alias
can be used with -M
as part of commands that run a different main function
(such as starting an nREPL server). If I had stuck with -M
for Cognitect's
test-runner
, I would have already had a separate :runner
alias that
provided just the :main-opts
for -m
(and the main namespace to run).
LazyTest doesn't have a -X
-compatible API, but it does expose functions to
run tests programmatically, so it doesn't really need to support :exec-fn
.
Along the way, I did hit some bumps: version 1.5.0 of LazyTest did not
support Clojure 1.10 and I still needed to test next.jdbc
against that.
LazyTest provides throws?
(and throws-with-msg?
) but it didn't support
clojure.test
's thrown?
macro. lazytest.core/throws?
expects an exception
type and a no-arg function to call, containing the code under test.
In clojure.test
, thrown?
is just "syntax" that is
understands but it
takes an exception type and a form to evaluate, making the migration a bit
harder: (is (thrown? Exception (some-expr :here)))
changed to
(throws? Exception #(some-expr :here))
. Noah is very receptive to feedback
(and pull requests) so version 1.6.1 of LazyTest addresses both of those
concerns: it supports Clojure 1.10 and it provides a thrown?
macro in the
experimental compatibility namespace (that maps the code to use throws?
).
As I work on next.jdbc
tests moving forward, I'll be able to mix'n'match
LazyTest's more expressive DSL for tests with the existing clojure.test
style of tests, and I'll probably migrate the latter to use the former over
time.
FYI: when I'm working with VS Code and Calva, I use a custom REPL snippet
to run tests in the REPL, in a way that supports both clojure.test
and
LazyTest tests -- see this
snippet for details.
It runs clojure.test/run-tests
first for the current namespace, then it
runs lazytest.repl/run-tests
for that namespace, and merges the results
summary.