Reading files back from my backups
The first run of my new backup check reported a config mirror as missing or empty, which is not a phrase I ever want to read next to the word backup. It spurred a momentary panic.
The files were there. That mirror carries absolute symlinks whose targets exist on the host and point at nothing inside a container, so the check that walks it followed one into empty space. A catch-all handler printed the one explanation it had been given, “missing or empty,” and a single dangling link condemned 207 files that were sitting right there. The bug was in the drill, not the backup.
The drill itself was overdue. When I moved the family’s data off a Synology appliance onto direct-attached storage, I found out how much I had taken for granted in the appliance’s ecosystem: automatic backups, verification, snapshotting, all of it mine to rebuild from scratch. I went into the transition wide-eyed and did the due diligence. What I never did afterward was a deliberate fire drill. The appliance had earned my trust through over a decade of use; the new storage had no such history, and its backup jobs had been reporting success since the move without anyone testing the claim.
What I built is a sampled read-back check, not a restore. Once a month it pulls a random sample of files back out of each kind of backup and checks them against the source, and nothing in it imports a database or brings a service up. Something as critical as data recovery needs a pass-or-fail answer rather than an interpretation, so I was happy to use AI tools to build the bench, but there is no summarizing model anywhere in it.
| Backup | What the drill checks | What a pass means |
|---|---|---|
| Snapshots | random files from the newest, a middle and the oldest snapshot, hashed (a content fingerprint) and compared with the live file where it hasn’t changed since | those files read back intact; an empty snapshot listing fails |
| Mirrors | random files sampled from the live copy, which the mirror must have | the mirror holds what the source holds, for that sample |
| Database dumps | the newest dumps decompressed end to end, then searched for the dump’s header and its completion trailer | the dump inside the file finished |
| Offsite copy | files pulled back down and hashed | those files came back intact; a missing one fails |
The sampling is seeded and the seed is printed on the card, so a failing run can be repeated against the same selection instead of argued about.
A pass is bounded by what the drill can reach. The walk skips symlinks now, and a folder the drill’s account cannot read is skipped without a word, which is a known gap in the checker rather than a property of the backups. Media and archive snapshots get a retention check, not file reads.
The fix for the first failure was small: skip links and report the real exception instead of a guessed one. The rerun on the printed seed passed, with 78 checks verified across the four legs.
A review later that morning found one more problem, also in the drill rather than the backups. The mirror leg had been sampling the mirror itself, where a copy job that quietly stopped including a path could never fail. It samples the source now; both cards above predate that change.
The first scheduled run, on October 5, was also the first with the corrected mirror leg, and that leg did its job: 15 files picked from the live side were all in the mirror and matched, and six more were read end to end from the mirror itself. The run still reported FAIL, and again the fault was the drill’s. A Docker engine update the day before had most likely changed who owns the drill’s scratch space, so the offsite leg couldn’t create the folder it pulls files into and crashed before pulling a single one. After the fix, a rerun on the same seed the next morning passed with 81 checks verified, including 26 files pulled back down from the offsite copy, each matching the source.