The test must assert the class, not the case

A chat bot I run follows sports scores. One evening someone asked why there hadn’t been any updates from a team they follow, and the answer turned out to be sitting in the runlog: thirteen consecutive polls, thirteen errors, all reading no game matched, every five minutes for an hour. Two other follows in the same state file were running clean.

The cause was one line. A scoreboard fetch took a league argument with a default, and the follow loop was calling it bare:

games = fetch_scoreboard()          # defaults to one specific league

So every non-default follow was fetching the wrong league’s scoreboard and searching it for its own team. One of the working follows worked by accident — its league happened to be the default.

Easy fix. And this is where the interesting part starts.

One cause, four sites, one already fixed

I grepped for other bare calls before writing the fix, and found four call sites in three modules. One of them had already been fixed, months earlier, with a fan-out helper that queries every league and merges the results. Somebody — me — had understood this bug completely, built the right abstraction for it, applied it to the site in front of them, and moved on.

The three survivors produced a symptom pattern that had been visible from the user’s chair for weeks and that nobody had connected:

• a follow for a team in a non-default league silently matched nothing, 13 out of 13;
• asking for that team’s card by name worked — the named path already fanned out;
• asking card? contextually in the same room did not — the contextual resolver was still bare.

Same room, same team, same minute, two different answers. That split was the only real signal, and it lived in the difference between paths, not in any one path’s logs.

So: understanding the cause was never the problem. Applying it everywhere was. The previous fix proved the cause was understood, and it did not prevent the recurrence at all.

Why the obvious regression test is the bug

The instinct after a fix like this is to write a test that pins the reported behaviour: “the follow loop passes a league argument”. It is a fine test. It passes. It will keep passing.

And it is the bug, written down and made permanent.

The thing that went wrong was fixing one member of a class. A test that covers one member of the class is that same mistake in test form. It creates exactly the feeling of safety that let the first fix stop at one site. Next year, when someone adds a fifth call site, the test stays green.

So the regression test I actually wrote asserts something else:

for mod in (module_a, module_b, module_c):
    source = strip_comments(inspect.getsource(mod))
    assert "fetch_scoreboard()" not in source, \
        f"{mod.__name__} has a bare fetch_scoreboard() call"

No bare call exists anywhere. That assertion would have caught all three survivors on the day the first one was fixed — months before the bug was reported — and it will catch the fifth when someone adds it.

When to reach for a class assertion

The trigger is specific and easy to recognise: the fix is “call the helper”.

Whenever a bug is repaired by routing a call through something that already existed — a wrapper, a sanitiser, a fan-out, a retry policy, a permission check, a path-normalising function — the interesting question is never “is this site fixed?”. It is “how many other sites are there?”. You already have to run the grep to answer that. Make the grep the test.

Yes, this is a source-level assertion, and source-level assertions are usually a smell — they couple tests to formatting, they break on refactors, they test the letter rather than the behaviour. That objection is right in general and wrong here, because of what the invariant is about. The claim “this dangerous default is never taken implicitly” is genuinely a claim about the source text. A behavioural test can assert that a given call is correct. No behavioural test can express “and nowhere else.” The universal quantifier is the thing you need, and the source is the only place it lives.

A few practical notes from using these for a while. Strip comments and strings first, or a docstring showing the bad pattern will fail the test. Enumerate the modules explicitly rather than walking the package, so that a new module has to be consciously added — an auto-discovering version silently stops covering things it can’t import. And phrase the failure message as the instruction, not the observation: “call fetch_scoreboard(league=…) or the fan-out helper” saves the next person a git-blame.

Always prove the test fails first

This is the step people skip, and skipping it converts the whole exercise into theatre.

A class assertion that is silently vacuous is worse than no test at all, because it reads as coverage. A typo in the grep string, a module that isn’t imported, a strip_comments that ate the whole file — every one of those produces a green test that asserts precisely nothing, and a green test is a closed question.

So before committing: restore the unfixed sources, run the test, confirm it fails, and count the failures. When I did that on this fix it reported seven failing assertions across the two bugs I was closing. Without that step, I would have had no way to distinguish “the code is clean” from “the test can’t see the code”.

It is the same discipline as deliberately breaking a guard to check that the guard notices — which, the one time I bothered, found two bugs in the guard.

The rule: if the fix is “call the helper”, the test is “nothing calls it bare” — and prove that test fails before you trust it green.

---

sources: concepts/the-test-must-assert-the-class-not-the-case, concepts/a-plausible-negative-hides-the-wrong-question