Was curious so we checked and yep, Antithesis finds this bug in about 15 minutes. Will post a repro/writeup here soon.
Discussion here: https://news.ycombinator.com/item?id=49277799
Tracking down the 16-year-old WAL-reset SQLite bug
111–120 of 263 posts
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#112Earlier quoted context omitted.
With opus 4.7-ish to fable 5, immediately after release - before they locked it down, it was shockingly easy to find crash bugs in a lot of very heavily used DBs and other software.
I'll accept crashing in preference to a database acknowledging transaction that weren't durably committed.
For sure! I haven't had the opportunity to burn many tokens on this but given how quickly it was able to fuzz it into crashing i'm sure it can manage.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#113> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.
Not only are they funding open source, they are actively allowing users to host their own control plane via headscale which is a libre implementation of the tailscale control protocol and developed by an engineer who works for Tailscale by day. This instantly made me trust and like them, even if at first I was cautious because I naturally mistrust anything that gets a lot of hype. I've been running headscale on NixOS, hosting my own tailnet since last year and everything just works out of the box. Tailscale lives up to the hype! <3
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#114Very nice article, and I appreciate SQLite's explanation of the bug too. And how extremely cool Tailscale appears to have been about it (paying for the VFS shim, etc.). I'd have liked to have heard more about the decision to checkpoint so frequently that put them on this path though. Presumably that's to keep the WAL tiny for very fast recovery. Trying to mitigate some of the deleterious effects of inserting a DBMS i…
Agreed, I was also wondering about this. Maybe the aggressive checkpointing was for preventing WAL-overflow?
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#115> This investigation is a useful reminder: running boring technology in a non-standard way is a risk. The common paths and standard configurations are incredibly well-tested and reliable. Most people use SQLite in a standard configuration and never face this sort of issue. Everything we were doing was a public, documented, supported configuration—but by taking manual control of the checkpointing process and running at our own aggressive pace, we stepped off the well-trodden operational path.
I feel like they missed a key takeaway from their own argument here, they should not be running a non-standard configuration :)
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#116I lost my Android SMS DB way back in the day (20+ years? dang I'm getting old) because the SMS app's fix when opening the DB detected any issue was to delete it and start fresh.
lol good old ostrich algorithm
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#117> This investigation is a useful reminder: running boring technology in a non-standard way is a risk. The common paths and standard configurations are incredibly well-tested and reliable. Most people use SQLite in a standard configuration and never face this sort of issue. Everything we were doing was a public, documented, supported configuration—but by taking manual control of the checkpointing process and running a…
This reminds me why RFC 2119 includes the description it has regarding SHOULD.
Like this is a great practical takeaway, sure, but eh.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#118Earlier quoted context omitted.
Probably the most refreshing thing I've read in a while. Glad to support them moving forward if this is indeed their modus operandi.
Tailscale is the best. It's infinitely better than Hamachi, ZeroTier, etc. My only gripe is that they have some really weird SSO requirements like GitHub, etc. and then that provider becomes a permanent part of your identity.
How are the requirements weird? They support any OIDC endpoint whether your own or a vendor’s, and, while uninteresting to HN folks, they also now support passkeys for having no SSO provider at all.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#119Glad this got found and fixed, but I continue to be astounded at the amount of work people put into making SQLite do things that would be much simpler with other systems.
Yeah it’s a pity that wasn’t addressed in the article. This is a little like “we shot ourselves in the foot and then performed surgery on our foot, and everything is resolved now.”
That assumes that they do actually believe it's a mistake. They didn't explain the reasons they've gone for this architecture in much/any detail. I'd be interested in hearing them talk more about that in the future.
Perhaps you or I would make a different decision based on the aims that lead them there. But there isn't enough information to say whether or not their decision was a mistake, even if it has lead to a peculiar bug. It may be perfectly legitimate and we just don't know some of the constraints they had.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#120Maybe it's just me, but the explanations of the cause don't align. One clue was that during corruption incidents, our metrics showed that SQLite would report copying more pages from the WAL file than were actually available. If there are 10 pages in the WAL file and 20 pages get copied to the database, something is clearly wrong. vs it thinks some of the pages have been copied from the WAL into the main database file…
Those seem consistent to me. Some pages weren't written to the WAL (yet?), but something else referenced them or otherwise indicated they existed, so then the other process tried to read them, resulting in the issue in the first quote.