The chat bot that runs in my house stopped for five days and nobody noticed. It did not crash. Its container exited with status zero and simply was not there any more, and for five days every host it lived on answered every ping, the uptime dashboard on the small cloud instance stayed green, and the family stopped getting sports scores. I found out because someone asked why it had gone quiet.
So I wrote a watchdog. A timer on the media server ran every sixty seconds, checked that every container that should be running was running, and pushed one heartbeat outward naming whatever was down. It ran for weeks. It detected the next failure correctly and it told nobody, because the environment variable holding its push URL had been empty since the day it was installed. It wrote the alarm to the journal instead, which is the exact silence it had been built to end. Around the same time the checker inside the house, the one with nineteen monitors on it, was itself dead for three days.
None of this is a story about bad tooling. The house is a small estate where most services have no public name: a DNS resolver, a home-automation server, a printer console, a code forge and the bot all answer only on the LAN or over a private VPN. The uptime checker that matters runs outside, on a cloud instance, because a checker that shares a hypervisor and a power strip with what it watches goes dark at the interesting moment. And that instance cannot reach into the house at all. The dashboard was green because it had never once asked the question.
Diagram: read it as four steps — what the outside checker can and cannot see, what went wrong, why the green survived it, and the shape of the fix; the four guided views walk them in order.
The principle
If your monitor cannot reach a service, the service has to reach the monitor. The check then stops being “is it up?” and becomes “did the beat arrive?”, and the absence of a beat is the alarm. The signal you cannot receive is the one you must send.
Why the failure hides
A dashboard renders the last thing it learned. That is the whole mechanism, and it is the same one under every green in this story. A poll that cannot reach its target learns nothing, and nothing has no colour of its own, so the tile keeps the colour it had. Two monitors were once created with a timeout of zero, meaning none; when their target began hanging, the checks hung with it, produced no further results, and showed UP for as long as anyone looked. A check that has stopped checking is indistinguishable from one that keeps passing.
Webhooks have the same shape from the other side. The code forge tells the bot about finished runs and failed mirror syncs by sending an event, and the bot turns each into a card in a chat room that serves as the log. If the forge itself dies, no mirror fails loudly and no event is sent. Every signal it would have produced simply stops, and a stopped stream of events looks exactly like a quiet day. A webhook cannot report the failure of the thing that would have sent it.
Even the signals that do arrive can be green without being true. The forge’s health endpoint answered “pass” on the day it was installed, while the instance was serving its setup wizard and every API route returned 404. Liveness and correctness are different questions, and one probe answers one of them. And a monitor that cannot go red is worse than none: two checks against bare domain roots that serve a placeholder passed every minute and measured nothing anyone would act on, which reads as coverage.
The shape of the fix
The whole fix was one shell script, one systemd timer and an afternoon, plus a handful of rules that came out of getting it wrong first.
First, for every service, write down who can reach it. Anything the outside checker cannot reach becomes a push monitor, and something inside becomes its publisher. Here a script on the DNS box runs every two minutes and makes four probes: it asks the resolver for a known name and compares the answer to the expected address, it curls the home-automation server and the printer console, and it asks the forge’s health endpoint. Each result is pushed to the checker with a ten-second timeout.
Second, one token per service. A hypervisor heartbeat once carried a single status for the host and its guests together, so a guest stopped on purpose made the host read as dead. Two facts in one field is one fact fewer than you need, and the beat should carry a message that names what is down, so one alert holds the whole picture instead of thirteen pages.
Third, probe by address, never through the thing you are watching. The script could resolve the other services’ names through the resolver, and if it did, a resolver outage would take all four checks dark at once and hide which one actually failed.
Fourth, make the window longer than the publisher’s period, with slack. The windows were 120 seconds for a while, against a timer that fires two minutes after the previous run finished, which is always a little longer than 120 seconds. One day of that produced 983 beats marked up and 980 marked pending, and a log that was a wall of “no heartbeat in the time window”. Noise like that trains you to stop reading, which is a slower version of the original failure. The windows are 300 seconds now.
Fifth, give every sender you depend on an absence-based half. The forge has a push monitor of its own, fed by the same script, and it is the only instrument in the set whose silence is the alarm. It could never have been a plain HTTP check, because the forge has no public name by design; the push is not a workaround, it is the only honest shape.
Sixth, arm a push monitor only when its publisher exists. Two were created before anything published to them and sat grey for weeks, which looked like a slow failure and was a monitor waiting for a sender. And keep the alert channel outside the thing it reports on, so that the message about the house going dark is not sent from inside the house.
The remaining gap is named rather than hidden. A mirror credential was rejected once this week and the credential store erased it, so pushes to the forge stopped quietly. What caught it was a red line printed into a terminal by the receiving hook, not a monitor. The last successful mirror time is the next beat to send.
If the checker cannot reach it, it must reach the checker, and then silence is the alarm.