Patchdayproject

For years, my homelab’s software updated itself at 4am. A tool called Watchtower checked every Docker container for a newer image while the house slept and swapped whatever it found, which is why my morning-troubleshooting rule has long been “if something broke overnight, check Watchtower first.” Around 20 services lived on that autopilot. Another 15, ones where a surprise upgrade could hurt (the family photo library, my log alerting stack), were version-pinned and fed into a recurring to-do list item I reviewed by hand, eventually, sometimes. Two failure modes, neatly split: the floating services mutated silently in the dark, and the pinned ones aged toward unpatched vulnerabilities while the review item rolled over another week.

Patch management is the unglamorous chore underneath every self-hosted system (every service is software, every software has versions, and somebody has to decide when to move), and I had effectively split the job between a robot with no judgment and a human with no time. What I wanted was the middle: something that watches versions constantly, explains what changed, and then waits for me. The forcing function was ultimately the broad consumer rollout of the latest class of frontier models (Fable 5, Sol 5.6) that put the capacity to exploit software vulnerabilities into most everyone’s hands.

The build splits the job in three. Discovery went to Renovate, because knowing what “newer” means across four different container registries is a solved problem I had no interest in re-solving. Judgment lives in a weekly Discord digest: every pending update, its changelog AI-summarized, each with a Skip button, and one “install” button for whatever survives my pruning. Execution belongs to a small container named patchday, the only service in the update pipeline holding write access, which applies exactly what the digest approved. Majors never get a button; they’re flagged for a sit-down. And services with database migrations (Immich, Graylog, Paperless, Ghost) carry an honest warning in every digest: rollback restores the image, not the schema. The rollback I’d promised myself quietly excluded migrations, the half that can’t be undone. My backup paranoia serves me well here.

The disarming part came first, though. Phase 0 pinned all 34 floating tags to the exact versions already running, verified by image digest so the “safety” deploy itself changed nothing. Only then did the autopilot start losing altitude. That ordering is the point of the whole project: patchday exists to retire Watchtower, so the net amount of unattended mutation in the house decreases, not increases, even as the new system gains a write path.

Day one of live fire found four bugs, none of them in the judgment. The digest scheduler died silently on its first unhandled exception and the bot looked perfectly healthy while never posting again; now, every loop iteration wears its own try/except, and failures are loud. My GitHub token leaked into the log pipeline and had to be rotated, an hour I owe to grepping my own logs out of paranoia. A Unicode arrow character in version strings broke the changelog parser. And Discord’s 2000-character message cap truncated the first real digest mid-service. Shipping trust is mostly shipping plumbing.

The best test arrived uninvited. A MongoDB major upgrade, researched and deliberately attempted, refused to start: version 8 is incompatible with the bleeding-edge 7.x Linux kernel my homelab runs, a detail no app-level compatibility matrix mentioned. I rolled back (untouched data, five minutes), and then taught the pipeline the lesson: Renovate now carries a version cap for mongo until the upstream fix ships, with a dated reminder to lift it. The update system now remembers what it’s not allowed to install, and why.

The autopilot is grounded now. A forced-failure drill (deliberately break an update, then confirm the rollback path in anger) passed, and Watchtower is retired, a container I was oddly sentimental about deleting. Updates happen when I tap, the tap knows what it’s installing, and 4am is quiet.