The on-call engineer I couldn't hireproject

Every network-attached device you own is always talking: what’s happening, what’s gone wrong, and a whole lot of noise. The (often ignored) process of “listening” is called monitoring. Monitoring, however, only tells you that something happened, never whether it matters. My homelab’s logs (from the NAS, my network gear, my servers, and every container therein) all flow to one place, an open-source log platform called Graylog, and rules watch the stream for trouble: failed logins, crashing services, filling disks. All of that is a solved problem. The unsolved part was that every “alert” still terminated in the same expensive component: me, reading logs at some inconvenient hour (or worse, never), deciding whether to care.

The alert that broke me was an SSH login at an odd time of night. “Am I under attack?” Investigating it took about four commands: who logged in, from where, and what they ran. The answer was a machine account doing exactly its job (phew). Four commands and 10 minutes of my evening to conclude “not a state-sponsored attack on my dog photos,” and as I closed the laptop it occurred to me that running a handful of read-only commands and reasoning about their output is precisely the kind of judgment an AI model exercises well.

So the pipeline grew an investigator. When an alert fires now, an agent receives the event with its surrounding log context, opens an SSH session into the server under a read-only account, and works the problem: up to 50 commands and 5 minutes of wall clock, with stall detection so it gives up honestly instead of thrashing (no #tokenmaxxing in this economy). It ends with a verdict (benign, action required, or stalled) and posts a single Discord message containing the evidence and its reasoning. Dead ends are first-class citizens: the account has no sudo on purpose, so when the agent can’t see something, it reports the gap rather than guessing, and a weekly digest collects the gaps so they become fixes instead of blind spots.

The investigator runs on a different machine, never the homelab server itself. If the server is the thing on fire, the fire marshal should not be standing in the building. A hung Docker daemon, a bad 4am container update, a storage cascade; in each case the second machine keeps its footing and its ability to reason about the first. The router in front keeps a fallback for the reverse failure: if the investigator is unreachable, it degrades to a simpler analyze-and-post path. An alert is allowed to arrive degraded. It is not allowed to vanish.

Daily use then filed its bug reports. The first was volume: the pipeline originally posted three messages per alert (an initial analysis, an “investigating” note, then the verdict), at 10-15 alerts on a normal day. That is how you turn an alert channel into background noise and train yourself to ignore the one that matters. Now there is exactly one message per alert, posted by the investigator, with everything in it. Here’s a recent verdict, exactly as it lands in the channel:

The second was repetition. A misbehaving integration once fired 13 alerts in twelve hours, eight of which the agent investigated to the same “benign” conclusion. So it learned to back off: after three consecutive benign verdicts on the same signature within two hours, that signature goes quiet for six, with a periodic roll-up so the silence stays visible, and any non-benign verdict ends the quiet immediately.

Diagnosis was the easy half of on-call. Letting the agent fix what it finds is the harder half, because it raises the stakes on trust, so the design work shifts from prompts to guardrails: it will propose, I will approve, and nothing will execute without a human’s fingerprint on it. That’s the harness I’m drafting now.