A database that answered every query and was still corrupt

On Tuesday one tenant's cell database filled its disk. The write-ahead log had not been checkpointed since the last restore, because the backup replicator had silently stopped confirming uploads a week earlier. The agents went down together.

We restored from the bucket. The restore replayed a frame whose page bytes no longer matched its checksum, and we let it through to get the tenant back online. Every agent answered again. Every query we ran came back fine.

Then we ran PRAGMA quick_check. The database was malformed.

Where the damage was

Only one table was broken: the full text index over step text. Every real table, experiences, steps, signals, artifacts, harness entries, read cleanly. The index is derived data, so we renamed the broken table out of the way, created a fresh one and filled it from the source table in 4,000 row batches, retrying when an agent held the write lock. The count matched the source to the row. quick_check answered ok six minutes later.

A single long statement over the proxy dies at about 150 seconds. Batches are the only honest way to do this kind of repair on a live cell.

What we changed

The database lives on a volume now. A stopped machine used to wipe its disk and force a full restore from the bucket, 25 to 40 minutes for this tenant. New cells get a 20 GB volume at birth; existing cells move one at a time with a script that stops the old machine, creates the new one on the volume, waits for it to answer health, and only then deletes the old one.

Database machines never suspend. The platform had suspended this one mid restore. Its connection to the bucket died on resume and the restore waited on a dead read forever.

The runner stays up when its database is down. It used to exit and loop. Now it reports why on its health endpoint, refuses work with a clear reason, and reconnects with backoff.

The backup chain is verified every ten minutes. The replicator opens a new generation on every checkpoint and links it to its parent through a 16 byte object. The verifier walks that chain back to the last snapshot and checks every batch of frames is consecutive. The hole that would have warned us a week early now shows up on the tenant's page the same morning.

Nothing here is exotic. It is the difference between a database that answers and a database you can trust, and we only learned it by asking the second question.