The upgrade my own referee refusedproject

At 6am the overnight run looked perfect. My email classifier’s fine-tune had chewed through 1,970 training examples in about 8 hours on the homelab GPU, and the loss curve (the number that’s supposed to shrink as a model learns) had collapsed to nearly zero before the first epoch finished. Reassuring at a glance, suspicious in hindsight; a curve that good that early is how memorization looks, not learning. I read it as a clean run and a candidate ready for evaluation.

Some context from the last mailroom post: my email agent improves weekly from my corrections, and when enough of them bank up, it earns a retrain, a new LoRA adapter fine-tuned on its own mail history plus everything I’ve red-penned. The first adapter shipped and made the classifier measurably safer. This was retrain number two, unlocked by a 304-correction sweep, and everything about it looked like progress.

The first alarm was almost comic. The training container’s built-in validation check scored the new adapter zero out of 60. It also scored the current production model zero out of 60, the model that labels my real mail correctly all day long. Turns out that rebuilding the training image had silently pulled a new major version of Hugging Face’s Transformers library, which broke the exact output formatting the checker expects, and every answer, right or wrong, scored as garbage. The score was advisory anyway: the checker printed its number and the job stayed green either way, which is all “passing” had ever meant here. I’d hit this genus of bug before (a gate measuring format while claiming to measure accuracy), and here it was again in a new coat. Just further evidence that version pinning matters.

The second alarm was quieter and worse: the training job had written its output over the live adapter’s files on disk. Production was still fine, because the serving engine holds its copy in memory, but the system was one routine restart away from serving an unvalidated model with no decision ever being made. I restored the incumbent, and candidates now live under their own names, loaded side by side with the production adapter; the serving engine happily hosts both, so evaluation never again shares a filename with what’s live.

Which cleared the stage for the real referee: a full-corpus replay, 745 real emails, candidate versus incumbent. Promotion required two things: no new dangerous misses on mail the incumbent already handles correctly, and overall agreement with my own verdicts at least as high as the incumbent’s. The candidate fixed 19 of the incumbent’s 32 dangerous misses (the doctor’s-message-filed-as-junk category, the whole reason to retrain), which on its face looks like halving the problem. It also invented 4 new dangerous misses of its own, and agreement fell from 480 of the 745 emails to 413. Sharper exactly where I’d corrected it, noisier everywhere else. My leading suspect is the diet: the training set was mostly the old model’s own verdicts with my corrections as a minority voice, a mix that would predict exactly this shape, though one run can’t convict it. Both bars failed. The candidate was deleted before breakfast.

Before I built the full-corpus replay, I would have shipped it. The quick signals all said yes: beautiful loss curve, clean logs, and a validation check that stayed green right up until I noticed it couldn’t do anything else. The referee that said “no” was the boring, expensive one, built precisely because the first adapter’s launch taught me that every grader needs a grader.

So as I write this, production still runs last month’s adapter, unglamorously and correctly. The null result went into the decision log as a result, not a failure, and the next retrain gets a different diet: corrections weighted up, the model’s self-labeled mail thinned out, and the same referee waiting at the end. The best thing my training pipeline produced this week was a no.