The Bitter Lesson, applied to my own harnessessay

One of the house rules I wrote for my homelab’s alert router existed to save four seconds: it disabled the model’s built-in reasoning mode, because the router only routes, and I wanted the verdict fast. Then a drill the router kept failing (a class of faults that look benign on the surface) refused to yield to round after round of my rule-tweaking. What finally fixed it was deleting my rule and switching the reasoning back on. The optimization was the bug, and I had written it.

I had a name for that failure by then, because I’d just read the essay that predicted it. In 2019 the reinforcement-learning pioneer Richard Sutton published a one-page essay called The Bitter Lesson. Its famous claim is that 70 years of AI research keep teaching the same thing: general methods riding on more computation beat human-crafted knowledge, every time, and the researchers who built their own expertise into their systems kept losing to the ones who didn’t. Turns out, hand-built knowledge does worse than waste effort as methods improve; it complicates systems “in ways that make them less suited to taking advantage of” what’s coming. His receipts run from computer chess to speech recognition, and the LLM era has been re-teaching the same lesson on a faster clock. As someone who takes pride in having graduated from prompt engineering to harness engineering, that knocked the wind out of me. I only found the essay because Daniel Miessler’s prompts to run when a new pinnacle model drops, an excellent set of upgrade-day exercises, pointed me at it. So I spent an evening (read: weekend) auditing every piece of scaffolding in my fleet. I expected to be vindicated (the audit was my idea, after all). I was not.

Regular readers know the fleet: one agent triages my email, another investigates alerts from my homelab, a third writes my morning brief. Every one is wrapped in structure I wrote by hand (prompts, worked examples, filter rules, little helper scripts), and I had always filed that structure under “diligence,” and if I’m honest, a bit of “expertise.”

The audit needed a rubric, and this is the piece I’d suggest to anyone running agents on models that continually improve underneath them. Every piece of scaffolding encodes exactly one of five things:

1. Values and authority: what the agent is allowed to do, what needs my approval
2. Facts about my world: who I am, how the network is laid out, where the backups live
3. Verification: the answer keys and replay tests that check the agent’s work
4. Capability prosthetics: compensating for what today’s model can’t do
5. Cost crutches: compensating for what I can’t (or would rather not) afford

The first three age well; verification actually appreciates, because every more capable model that ships gets adopted as fast as my tests can vet it. The last two melt. Worse than melting, in fact, per Sutton: they actively block the better model from doing the job its own way.

The rubric is also where I part ways with the essay, deliberately. Sutton’s claim is about how to think, and on that ground compute keeps winning the argument. It says nothing about what to want. No amount of computation teaches a model my risk tolerance, or which changes need my approval, or that the family’s photos are irreplaceable; values, facts about my world, and the tests that check the work are not expertise smuggled into the system, they’re the job description. The dividing line the audit enforces is exactly that: scaffolding that tells the system what I want survives every upgrade, and scaffolding that tells it how to be smart is a bet against the next model.

My audit found the second kind everywhere, posing as house rules. The style guide for my agents carried workarounds for my current 14-billion-parameter Qwen local model written as timeless wisdom: hand-picked worked examples in every prompt, the reasoning rule from the opening, a double-sampling trick to paper over judgment I didn’t trust. My alert pipeline had accumulated nearly two dozen hand-curated exclusion rules, each one me encoding “what noise looks like” one pattern at a time, precisely the hand-built feature engineering the essay warns against. Even my email classifier’s retraining program (fine-tuning my preferences into its weights) is knowledge installation with a shelf life. None of these were mistakes when I built them. All of them are bets against the next model.

I got unexpected confirmation the same week. Anthropic’s Claude Code team wrote how they removed over 80% of their product’s own system prompt for the newest models, with no measurable loss on their coding evaluations: the hand-tuned instructions and worked examples that helped last year’s model had begun to constrain this year’s. The same audit, run by a vendor with the clearest possible view of the frontier, landed on the same verdict.

The fix I landed on is not resolve; resolve is how the scaffolding piled up in the first place. Every capability prosthetic in the fleet now carries a tag, [MELT: <the assumption it rests on>], and every model upgrade now includes a melt check: rerun a frozen set of real cases with the new model bare versus scaffolded, and whatever the bare model matches, delete that day. Deletion becomes a consequence of upgrading instead of an act of remembering. The ritual is Miessler’s framing paying off: upgrade day as the moment to point the new model at your own harness and its stale assumptions.

What convinced me wasn’t the essay; it was noticing my own systems had already proven it twice. An overnight fine-tune of my email classifier, a hand-curated training mix, lost head to head to the very model it was supposed to improve (that story deserves a post of its own, and now it has one). And the router drill from the opening fell to un-writing a rule, not to writing another one. The bitter part of the lesson was never that the machine wins; it’s that I thought it wise to challenge it at all.