I have a pipeline that turns one drawing of a character into a rigged 3D model the game loads as a single file. Fifteen steps, in order: paint five views from the drawing, measure them, make a body mesh and a head mesh, weld, decimate, rig, graft the head on, add eyes, close the neck, install. Each step writes one named artefact, and each step is skipped when its artefact is newer than its inputs, so a re-run only redoes what changed. It had shipped three characters and I trusted it the way you trust a build that has been green for a week.
Then I re-ran a character that had already shipped, to pick up a change in the eye stage. Every step printed its line and exited zero. The runner said “wrote” about the post-processed mesh, said “wrote” about the graft, said “installed”, and the game loaded the character. It was the previous model. Byte-identical in size, correct in every visible way, and not the file the run claimed to have produced. The same day, a full re-run of a different character was a complete no-op that read, line for line, like a success.
Diagram: read the green line first; then the four guided views walk the failure, why it hid, and the fix.
The principle: a green step proves the process, not the artefact
An exit code is a claim about a process: it ran and it did not die. What a pipeline needs at every step is a claim about an artefact: the file it promised exists, and it is not the file that was there before. Those are different claims, and a step can be honest about the first while saying nothing about the second.
Why it hides
Two of the fifteen tools are polite. They never overwrite: if the output name is
taken, they write the same name with -2 appended, print “exists, writing -2
instead”, and exit zero. That is a reasonable design for a tool run by hand,
where clobbering someone’s afternoon is the worst outcome. The runner, though,
verified each step with the only check that seemed to need writing: does the
output file exist afterwards? Yesterday’s file existed. The check passed, the
runner said “wrote”, and the new bytes sat beside it under a name nothing
downstream would read.
The second half is what turned one stale file into a clean-looking run. The skip rule, “a step is up to date when its output is newer than its inputs”, is correct and it is the engine of the cascade. The stale mesh was older than the rig built from it last week, so the rig step skipped as up to date. The rig was older than last week’s graft, so the graft skipped. The install step copied the same file it had copied before. Every skip was legitimate given its input, and the input was the lie. Nothing in the log distinguished this run from a real one, because a no-op and a success print the same lines; the only thing that differed was a timestamp on a file nobody was checking.
There is a smaller version of the same hole one stage earlier, and it is why I recognised the shape at all. The painted views used to be gated by eye: look at the sheet, confirm the hat is there, move on. A slow recolour passes that gate, because every frame looks fine on its own. One character’s generated walk had shipped at a median skin difference of 29 ΔE76 units from its own reference, the worst score in the cast, and a later pass had quietly turned him paler and lighter-haired. I was verifying that the artefact existed and looked plausible, which is the same thing the runner was doing with a file test, done with a human eye instead of a shell builtin.
The shape of the fix
The runner now reads the output’s modification time before it runs a step, and
after the step it refuses to continue if the time has not moved. The refusal is
loud and specific: it names the step, names the file, and says where the -2
copy will be, because the person reading it has just watched a green line and
needs to be told the green line was the problem. That is roughly fifteen lines
in a shell function and one stat per step, and it caught the next stale run
the same afternoon.
The mirror-image case had to be handled at the same time, or the rule would have been a nuisance. One of the mesh tools segfaults in its teardown after it has written a perfectly valid file, so a strict “non-zero means failed” would have thrown away good work. The runner accepts a declared non-zero exit when the named output verifies. Both halves are the same principle: judge the artefact, not the process’s account of itself.
And the gate that used to be a look is now a measurement. Before the sheet goes anywhere near the GPU, its median skin and garment colour are compared against the drawing with the same instrument the 2D game uses to audit its sprites, and a median ΔE76 above 10 stops the run. The check reads the artefact against the reference, not the process’s word that it painted something, and it runs before the expensive stage rather than after.
If you have a pipeline with a skip-if-newer rule, three questions are worth an hour. Does any tool in it refuse to overwrite, and what does it exit with when it does? Does the runner check that an output exists, or that it changed? And is there any gate whose verification is a person looking at the thing, because that gate will pass the one failure that looks plausible.
A step has not succeeded until the artefact it promised exists and has changed.