Freezing a CLI contract with 32 red tests

The first thing I saw wasn't code, it was red

On September 21 I pushed commit 6f5ae82 to my internal my-agent repo, and the terminal was still showing the selftest run: 32 failures, all red. My first guess was that I had messed up the argument validation in ingest-blueprint.py. For a moment I seriously considered just deleting the runner, since the implementation didn't exist anyway.

Wrong read. The failures were the feature. ingest-blueprint-selftest.sh was designed so that every test fails at this point: the ingest-blueprint.py stub deliberately prints "NOT IMPLEMENTED" to stderr and exits 1. That is the RED step of TDD, and the 32 red tests are the specification, not evidence of a problem.

Freezing the contract, not just failing

What I froze wasn't code but the CLI contract: the four subcommands check, render, init, and audit, the exit code for every condition, the output fields, down to flag names. Once the contract is pinned by tests, the implementation can only chase targets that are already written down. Silent drift becomes visible the moment it happens, because it shows up as a red test.

The principle is the same one behind contract testing [3], capture capture the agreement between two parties, store it, and hold both sides to it. Here the two parties aren't services. They are me writing the implementation today, and me three weeks from now who has forgotten every detail of the decision.

Fowler notes the most underrated benefit of test-first: it forces you to think about the interface before the implementation details [1]. That matched my experience exactly. The slowest part of this work wasn't writing the check logic. It was deciding what the output format should look like, and that decision is much cheaper to make while everything is still red.

A batch-freeze variant, not canon TDD

To be honest about the trade-off here. Kent Beck's Canon TDD says: take your list of scenarios, turn exactly one item into a concrete test, make it pass, then move to the next [2]. He explicitly warns that converting the whole list at once invites rework. What I did is exactly that: all 32 scenarios became real tests in one batch.

This was a deliberate batch-freeze variant, not textbook TDD. The goal wasn't micro-iteration. It was locking the specification in a separate commit before writing a single line of implementation. The rework risk is real: if test number six forces a design change, some of the other tests need revising too. I took the trade anyway. For a CLI contract, a stable specification matters more to me than iteration rhythm.

Synthetic fixtures and red you can read

The 557-line runner never touches real data. Every test runs in a mktemp directory against synthetic fixtures, never real jobs. Speed is a bonus; the real reason is honesty. No leftover state can make a test pretend to pass, so when a test eventually turns green, it is because my logic is right, not because of stale data from an earlier run. The assertion pattern borrows from bash testing tools: with errexit, every shell line is an assertion, and exiting 0 means pass [4].

One small detail I like: the runner counts STUB separately from FAIL. So the output doesn't read "32 failures" but "32 still stubs", a red state you can read at a glance without guessing which failures are real and which are just work that has not started yet.

Why not point the tests at the real jobs directory and skip the fixtures? Because a selftest that reads live state stops being a selftest the day the state changes underneath it. Fixtures are boring on purpose, and boring is what makes a verdict trustworthy.

Next step is obvious: implement until green, one scenario at a time, exactly the order Beck recommends. The difference is that I start from a complete map. I don't have to guess flag names or output formats while writing code, because they are already written down in the red tests. A screen full of red turned out to be the map, not a problem.

Sources

[1] Martin Fowler, Test Driven Development bliki, updated 2023-12-11
[2] Kent Beck, Canon TDD, 2023-12-11
[3] PactFlow blog, What is contract testing & how is it used?, updated 2023-09-02
[4] bats-core README, Bash Automated Testing System