Tracking down the 16-year-old WAL-reset SQLite bug • Hacker News Board

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#71

Post by dilyevsky » Wed, Aug 12, 2026, 6:01 PM UTC

jeffbee wrote: post #20

Block device upfuckery layers are powerful against databases. Years ago some colleagues wrote one that provides most of the hazards described by "Parity Lost and Parity Regained"[1] to test FoundationDB, which immediately uncovered several flaws in a project that described itself as well-tested. It's easy to do this with all the probing features that Linux (and others) provide today. 1: https://www.usenix.org/legacy/…

With opus 4.7-ish to fable 5, immediately after release - before they locked it down, it was shockingly easy to find crash bugs in a lot of very heavily used DBs and other software.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#72

Post by Neywiny » Wed, Aug 12, 2026, 6:01 PM UTC

wanderr wrote: post #68

This was a great technical writeup and very interesting to read, but it's not clear to me why once the suspected source of the bug was identified, they seemingly didn't build a automated way to trigger the condition? It seems like that could have cut down on the uncertainty of whether the fix worked over a painfully long period of time.

The sqlite dev team did. It's in the article.

> It could exist that long because it was rare—so rare, the SQLite developers had to add code to deliberately trigger it in their testing environments.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#73

Post by asveikau » Wed, Aug 12, 2026, 6:05 PM UTC

> SQLite corruption is possible, but it’s highly unusual and not something you should encounter in normal operation

If there's a hardware failure, for example a flaky SD card, it's not out of the question. A mobile app with a lot of usage will see it.

(Yes, I know this appears to be a server use case.)

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#74

Post by jbs789 » Wed, Aug 12, 2026, 6:07 PM UTC

As a simple user of SQLite, I think this level of debugging is incredible and appreciate being a beneficiary of the ecosystem and hard work of others. Thank you!

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#75

Post by jeffbee » Wed, Aug 12, 2026, 6:08 PM UTC

dilyevsky wrote: post #71

jeffbee wrote: post #20

Block device upfuckery layers are powerful against databases. Years ago some colleagues wrote one that provides most of the hazards described by "Parity Lost and Parity Regained"[1] to test FoundationDB, which immediately uncovered several flaws in a project that described itself as well-tested. It's easy to do this with all the probing features that Linux (and others) provide today. 1: https://www.usenix.org/legacy/…

With opus 4.7-ish to fable 5, immediately after release - before they locked it down, it was shockingly easy to find crash bugs in a lot of very heavily used DBs and other software.

I'll accept crashing in preference to a database acknowledging transaction that weren't durably committed.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#76

Post by danpalmer » Wed, Aug 12, 2026, 6:09 PM UTC

Glad this got found and fixed, but I continue to be astounded at the amount of work people put into making SQLite do things that would be much simpler with other systems.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#77

Post by gwking » Wed, Aug 12, 2026, 6:22 PM UTC

I wonder if this also affected litestream disproportionately, because litestream also inserts itself into the checkpoint process.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#78

Post by devmor » Wed, Aug 12, 2026, 6:25 PM UTC

asveikau wrote: post #73

> SQLite corruption is possible, but it’s highly unusual and not something you should encounter in normal operation If there's a hardware failure, for example a flaky SD card, it's not out of the question. A mobile app with a lot of usage will see it. (Yes, I know this appears to be a server use case.)

I have encountered this exactly once, and it was in fact a flaky SD card - running a small web service off of a raspberry pi, with the SQLite DB stored on the SD.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#79

Post by Spivak » Wed, Aug 12, 2026, 6:26 PM UTC

spockz wrote: post #31

riknos314 wrote: post #10

> Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window. Gotta love single points of failure...

The shard was already a way to make it not a single point of failure.

This is a great example of outages looking different from the perspective of the operator vs the user. Because there's many shards the blast radius of failure is contained to a small subset of users but for those users it's an outage. The way it's designed you can't lose any shards without impacting users. Compare to say Elasticsearch where it's possible to lose nodes and lose shards without the user noticing. One approach isn't universally better than the other.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#80

Post by Corrado » Wed, Aug 12, 2026, 6:29 PM UTC

AtlasBarfed wrote: post #64

binhex wrote: post #3

Earlier quoted context omitted.

Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.

I wish they'd buy the carcass of Keybase from Zoom. It seems very similar to Tailscale: immense utility from a free-tier product for the general public, which leads to trust and a large enterprise market.

Wow, that’s a great idea! I loved keybase back in the day and really thought it solved a real world problem in a unique and interesting way. Then Zoom bought it (for some reason) and left it to rot.