Twenty-six of my eighty-two tests could not start. They exited at import,
because the checkout had no .env file and a module-scope config read raised
before any test function was collected.
I knew that. I had known it for weeks. It was fine, I thought, because I was being disciplined about it: I recorded the baseline — 26 failures — and after each chunk of work I ran the suite and compared. Same 26. Zero regressions. I reported that four separate times over one long session, and every time it was a lie I had told myself with a straight face.
One of those twenty-six was a structure test. It enforces a line budget on the
main module — a ratchet, not a fixed ceiling: the budget may only ever be
lowered. The module sat at exactly 5001 lines when the session started. The
night’s work added 194 lines, and that test went from “blocked by a missing
.env” to “genuinely, loudly failing”.
The count never moved. It said 26 before and 26 after, because one blocked test had quietly swapped places with one real failure, and a count cannot represent a swap.
Absence counts as pass
A test that cannot START does not fail. It is absent. And absence is indistinguishable from success inside any summary that reports a number.
This is not the same defect as a flaky test or a skipped test that everyone knows about. A skip you can see. A masked test is one that never entered the accounting at all — the collector errored, the import blew up, the fixture raised, the file was excluded by a glob nobody remembers writing. The suite prints a smaller total, and a smaller total looks like a smaller codebase, not a smaller measurement.
The part worth keeping: why the baseline could not save me
My instinct — diff the failure set against a known baseline — was correct. It is the right instinct. It is what you are supposed to do. And it failed, for a reason that generalises well beyond test suites:
the baseline was produced by the same blind instrument.
Both sides of the comparison omitted the same twenty-six tests. So the difference between them was empty, the comparison was self-consistent, and the self-consistency read as evidence. Two blind measurements that agree feel like corroboration. They are one measurement, taken twice.
A control measured with the broken instrument does not detect the breakage. It reproduces it, and calls the agreement evidence.
This is why “no regressions since the baseline” is a weaker claim than it sounds. It says the instrument’s readings are stable. It says nothing about whether the instrument can see.
The shape of the fix
Unblock before counting. In my case the entire twenty-six needed four throwaway environment variables — values that did not have to be real, only present. With them supplied, the suite ran 82 of 82. The capability had existed the whole time; it had been used for exactly one newly-written test and never generalised to the rest. The cost of the fix was under five minutes and it was sitting behind weeks of confident wrong reporting.
That ratio is typical, and it is worth internalising: the masking is almost never expensive to remove. It is expensive to notice.
Classify failures by cause, not by count. “26 = 26” concealed a swap because a count has no room for identity. The moment I listed the failures by name and reason, the swap was obvious in one glance. Any comparison you make between runs should be over a set of identified failures, never over a scalar.
Report the two numbers separately. N failed and N could not run are
different facts about the world, and folding them into one figure is the exact
step that makes the summary false. A suite that prints 78 passed, 4 failed
when 26 tests never ran is not reporting; it is editorialising.
If you own the harness, make it louder still: fail the whole run when the collected test count drops below the last recorded count. A test disappearing is a defect in its own right and it has no other detector.
The same shape, one hour later
I found the second instance the same night, in a different language, in code I would have sworn was hermetic.
A Go test overrode two package-level variables — the home directory and the list of known hosts — and believed that made it isolated. It did not, because the host resolver consults the VPN address first, before either override is consulted. So both overrides were bypassed, the test SSH’d into the real production box, and when it failed it printed a live admin credential into the test output.
An override only isolates the lookup it actually sits in front of.
Isolation, like coverage, is a claim about every path. And a test that is wrong about which machine it is testing will still go green most days — which is why nobody finds it until the day it goes red for an interesting reason and leaks something into a log.
Both failures are the same family: a check whose scope is smaller than its apparent scope, reporting confidently about the part it can see and silently about the part it cannot. The generalisation I now use as a habit is to ask, of any green suite: what would a test that never ran look like from here — and would I be able to tell? If the answer is “the same as this”, the number in front of me is not evidence.
The rule: report “could not run” separately from “failed” — absence counts as pass in every summary that reports a number.
sources: concepts/a-masked-test-is-absent-not-failing,
concepts/a-plausible-negative-hides-the-wrong-question