The test container gave itself away
Both investigators looked at the same broken container and wrote the same verdict: a chaos-engineering test, behaving as designed, benign. One was a frontier cloud model. The other was the 27B Qwen model running on the GPU in my homelab. The container had been up five hours in an unhealthy state, was printing a fatal error every three seconds, carried a failing streak of 21, and was configured never to restart itself. Both models gathered those facts and set them aside, because the compose service was named chaos-victim.
I have been aggressively pursuing the boundaries of local AI, and most of what I used to send to frontier models now works just as well on my own hardware. The holdout was the pair that watches my homelab’s logs: a router that decides whether a blip deserves attention, and an investigator that SSHes in and works the problem. In early September both still ran on a paid cloud API. Comparing them honestly against local models takes real faults, and my genuine, non-benign alerts arrive in the single digits a month. Months of running both would have bought me maybe twenty cases to score.
A chaos container buys that time back. It injects the worst things my homelab is plausibly going to do, on demand and repeatably, in a throwaway container that shares the real alert pipeline with everything else. The quarterly drills had already shown the detection layer notices an injected fault; this drill asks what the investigators make of one. The first batch proved the plumbing. Every injection raised its alert, the cloud model investigated it, and the local model ran a shadow investigation beside it that never posts anywhere.
The trial that ran right after the held fault shows how the investigators got to their verdicts. Its fault lasted 75 seconds, so by the time they looked, the container really was healthy again, and both verdicts say so. Then both reach for the giveaways: the service name, and the run of benign verdicts on this same container, the held fault’s among them, that every investigation is shown as recent history.
I wasn’t surprised that the frontier model identified the chaos container for what it was. I had assumed it was leaning on what it knows about me. It wasn’t: both models run on the same prompt, which describes the host, the rules and the method, and nothing about who built what. I did not expect the local model to get there too. It did, and that left the trial unable to tell me how either investigator would handle the same fault in an ordinary service.
There was a side effect I liked less. Three acquittals in a row is the shape my triage layer reads as a service that genuinely flaps, so my own drill put the real alert pipeline into its six-hour suppression state. I had it cleared that evening.
If a name can decide a verdict, the repair is to stop handing over the name, along with every other clue I could find. The container, the service, the image and the network took a generic name with no obvious matches on GitHub. The command it runs, the restart policy, the startup line and the failure signature all read as an ordinary, slightly poorly written service. Its control routes sit behind a token the investigator is never handed, because the command filter permits curl and a curious model could otherwise just ask it. A drill’s own verdicts no longer count toward the suppression streak or reach the next investigation’s history.
The disguise did cost me the thing I was proudest of. The old victim fired its faults in sequence and produced a result in 20 minutes; the new one has to respect the pipeline’s own throttles and let its fingerprints age out of the investigator’s memory, so a batch now eats the better part of a weekend.
The next batch tested four cases: enough for an initial comparison, not a production decision. Both models escalated the held fault this time, on the evidence. They agreed on three of the four; on one benign case the local model escalated and the cloud model let it go. Reading the transcripts also found the disguise leaking somewhere I hadn’t inventoried. The held fault raised a second alert, and on those investigations both models opened the compose file, where a comment explained the whole arrangement and the build path said the rest. That comment is gone.
The two seats have since gone different ways. The local router holds the routing seat in production, and a quarter of the alerts it decides to skip get a second look from the cloud model. The investigator seat stayed with the cloud model, because the local model still misses a fault that has healed by the time it looks, and two prompt changes didn’t close that gap.