My hopes and dreams for 'zig test'

I love Zig, I use it daily in multiple capacities, especially in developing Mach engine which I've been working on for well over a year now. I plan to continue using it well into the future, and I want it to be even nicer for others to adopt in the future. That's why I think it's about time I wrote this slight hate letter to zig test that has been on my mind for a while now.

I'll start with my weakest points, and end with my strongest-so bear with me.

Command documentation

I think as a user coming to a language, you're very likely to explore the CLI. Hopefully that CLI will describe to you best-practices. zig -h today gives something like:

Usage: zig [command] [options]

Commands:

  build            Build project from build.zig
  init-exe         Initialize a `zig build` application in the cwd
  init-lib         Initialize a `zig build` library in the cwd

  ast-check        Look for simple compile errors in any set of files
  build-exe        Create executable from source or object files
  build-lib        Create library from source or object files
  build-obj        Create object from source or object files
  fmt              Reformat Zig source into canonical form
  run              Create executable and run immediately
  test             Create and run a test build
  translate-c      Convert C code to Zig code

  ar               Use Zig as a drop-in archiver
  cc               Use Zig as a drop-in C compiler
  c++              Use Zig as a drop-in C++ compiler
  dlltool          Use Zig as a drop-in dlltool.exe
  lib              Use Zig as a drop-in lib.exe
  ranlib           Use Zig as a drop-in ranlib

  env              Print lib path, std path, cache directory, and version
  help             Print this help and exit
  libc             Display native libc paths file or validate one
  targets          List available compilation targets
  version          Print version number and exit
  zen              Print Zen of Zig and exit

...

I wish there was a distinction here between commonly-used commands like zig build, zig fmt, etc. and less commonly used ones like zig test and zig ranlib. As it stands today these are all are on equal footing, and zig build and zig test are on even less than equal footing due to the fact one has 'test' in the name and the other does not.

As a new user, it looks like zig test would test my code and zig build would build it. I was baffled by the help output when I encountered Zig and was looking at zig test -h, confused that it worked on only a single file, and I imagine people today trying it on established projects and running into missing dependencies or similar because they didn't use zig build test instead:

$ zig test ./src/gfx2d/font_builder.zig 
./src/gfx2d/font_builder.zig:9:26: error: no package named 'freetype' available within package 'root'
const freetype = @import("freetype");
                         ^~~~~~~~~~
referenced by:
    test.font_builder: ./src/gfx2d/font_builder.zig:229:24
    remaining reference traces hidden; use '-freference-trace' to see all reference traces

General documentation

The official docs suggest using zig test directly. I personally think this is leading users down the wrong path, but further I would suggest there is a greater mistake here: there is no clearly worded 'how do I test every file in my project?' section describing the zig build test pattern, the magical std.testing.refAllDecls incantation you need to get your tests to run in nested containers, etc. There is a section on the latter, but it's title 'Nested Container Tests' leaves a lot to be desired in terms of discoverability. I would propose 'Running all tests in a project' or something to that effect, else we end up with these questions on Reddit.

Lastly, I'll say it's rather easy to mistakenly not reference a test that you intended to run. You wouldn't have written the test if you didn't want it to run, so it's a bit annoying Zig does not have a better way to discover tests in a project than refAllDecls. It should be much harder to make this mistake.

Adding gas to the fire

I frequently find in my projects that zig build test doesn't enumerate which tests were ran by default:

$ zig build test-mach
All 5 tests passed.

I've had other suggest I need to use this Bash incantation (what is the incantation on Windows, by the way?) to get enumerated output:

$ zig build test-mach 2>&1|cat
1/5 test_0... OK
2/5 test.ref... OK
3/5 test.invalid... OK
4/5 test.path... OK
5/5 test.font_builder... OK
All 5 tests passed.

While in other projects I find it sometimes tells me which tests it ran, but leaves out some of them. Did it run the other 3 tests?! e.g.:

% zig build test
Test [2/4] test.simple test2... 1337
All 4 tests passed.

You might've picked up by now that this behavior is dependent upon whether the test writes to stderr; I find this magical change in behavior baffling and surprising. It still trips me up to this day. I don't want the behavior of zig test to change based on whether I am writing to stderr. I don't want the behavior to change based on whether I am piping output somewhere else in Bash. The output should be stable, reliable, and understandable. Tell me consistently which tests were ran.

Where is my output, anyway?

Another frequently annoying thing to me is that the first line you print to stderr ends up.. on the same line as the test runner's output. i.e. std.debug.print("{}\n", .{1337}); ends up here:

% zig build test
Test [2/4] test.simple test2... 1337
All 4 tests passed.

I now out of habbit insert a \n at the start of anything I print in tests, so that it ends up on it's own line. I find this a pretty inconvenient design decision.

Putting that aside, another major foot-gun here is that std.log.debug and std.log.info do not show up in zig build test output at all! This is not the right way to encourage folks to use proper logging levels.

(To zig build test's credit, though, it is very cool that std.log.err actually fails the test if you use it!)

expectEqual parameters are backwards

I know this is an open proposal and the problem is more nuanced than I make it out to be here (there are good type inference reasons for this inversion.) I just want to call out that I find it incredibly confusing today, and that almost all other languages' testing frameworks do not have this inversion.

P.S. std.testing.expectEqualSlices formatting/explanations are not great.

Zig testing could go beyond other languages

Everything I've described so far are mostly paper-cuts. The last thing I want to suggest is that Zig take testing a bit to the next level:

• Benchmark tests like Go has, including a measured allocator for memory usage, etc. would be awesome.
• The ability to run setup/teardown code for multiple tests. e.g. I don't want to have to call glfw.Init in each test I run, but rather just once per testing process before the tests run. Go allows this by defining a TestMain function
• Automatic-updating of tests. In many cases when writing tests you just want to confirm the output of a function (which you believe is correct right now) does not change in the future. OCaml and Go (via autogold) support basically running tests with an -update flag and having it auto-fix expect() values inline for you (at least for simple values.) Finding a Zig-like approach for this would be tricky, but I think worthwhile.

Thanks for reading

Thanks for listening to my Friday morning ramble. I posted here to hopefully keep the audience smaller / just to Zig folks; I just want testing in Zig to be an exceptional experience since it's so critical to new users using the language, it's already pretty good :)