Monitoring a house the internet cannot reach

Monitoring a house the internet cannot reach An architecture diagram generated by Archify. in-house watcher · timer · every 2 min · 4 probes · inside the house · pushes, never polled in-house watcher timer · every 2 min · 4 probes pushes, never polled container watchdog · media server · every 60 s · inside the house · names what is down container watchdog media server · every 60 s names what is down private services · DNS · home automation · printer · inside the house · no public name private services DNS · home automation · printer no public name code forge · repos · runs · issues · inside the house · LAN + VPN only code forge repos · runs · issues LAN + VPN only published services · chat server · blog · media · inside the house published services chat server · blog · media house bot · webhook receiver · ledgers · inside the house · health-checked every 60 s house bot webhook receiver · ledgers health-checked every 60 s ops room · the log, as chat · inside the house ops room the log, as chat public uptime monitor · small cloud instance · 16 monitors · public internet · outside the house public uptime monitor small cloud instance · 16 monitors outside the house alert channel · chat + phone push · public internet · outlives what it reports alert channel chat + phone push outlives what it reports probes by address healthz · liveness only push heartbeats · 4 tokens · 300 s window push · every 60 s · names what is down keyword · 60 s webhooks cards down · recovered inside the house public internet

What green hid

  • • The bot exited cleanly and sat five days; every host answered ping the whole time
  • • The watchdog fired every 60 s, detected it, and wrote to the journal — its push URL was empty
  • • Two checks with no timeout hung on a hanging target and kept showing UP

The fix, costed

  • • One shell script and one timer, an afternoon: four probes, four tokens, pushed every 2 min
  • • Probes by address, never through the resolver being watched
  • • Windows longer than the period: 120 s on a 2-min timer gave 983 up / 980 pending in a day

Rules that came with it

  • • Every sender you depend on gets an absence-based half — the forge's beat is the one whose silence is the alarm
  • • A push monitor is armed by its publisher, not its creation; two sat grey for weeks
  • • The alert channel lives outside the thing it reports on