As others have said: great article! I did find myself wanting them to get to the point, but once they started describing the bug and the fix, it was very satisfying. I'm very happy there are companies out there on the frontiers of functionality not only funding fixes and debugging measures, but taking the time to write up the details so we can all benefit. Tailscale just moved up in my priorities list. Was going to h…
Tracking down the 16-year-old WAL-reset SQLite bug
161–170 of 263 posts
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#162Re: Tracking down the 16-year-old WAL-reset SQLite bug
#163Glad this got found and fixed, but I continue to be astounded at the amount of work people put into making SQLite do things that would be much simpler with other systems.
Tailscale likely deals with a lot of security-sensitive traffic, given the nature of the service. I'm guessing one of the requirements they were given was to have a tiny blast radius in case encryption keys got leaked, and that meant isolating each customer's tailnet (meta)data to it's own sqlite db rather than letting everyone share a postgres cluster.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#164Re: Tracking down the 16-year-old WAL-reset SQLite bug
#165Re: Tracking down the 16-year-old WAL-reset SQLite bug
#166Earlier quoted context omitted.
Interesting. I don't think this kind of technique could realistically have caught the bug in advance, though; you're not going to model the entire codebase in TLA+, so the investigation in that post modeled just the parts that turned out in hindsight to be relevant.
> you're not going to model the entire codebase in TLA+ Not by hand in the olden days but with AI to help with it, why not?
This is importantly different from formal code verification, in which the specification that needs to match the programmers' intentions is small enough to review, while the bulk of the LLM's output is proof tactics and such, which don't need human review because the verifier guarantees* that they're right.
I suppose you could just tell Fable "hey try using TLA+ to look for bugs in this code" and see whether it finds any.
* Uh, usually. https://leodemoura.github.io/blog/2026-8-1-postmortem-for-ke...
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#167Earlier quoted context omitted.
Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.
I wish they'd buy the carcass of Keybase from Zoom. It seems very similar to Tailscale: immense utility from a free-tier product for the general public, which leads to trust and a large enterprise market.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#168Earlier quoted context omitted.
> what if this bug is still there but the timing window is now one CPU instruction? I think a model checker or something similar that can exhaustively cover a search space would suffice, though at that point I think the boundary between "test" and "formal verification" becomes somewhat fuzzy.
So, a proof, not a test.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#169Earlier quoted context omitted.
Yeah, I have my Tailscale tied to my Apple account, which just feels weird. I can add a Passkey account to my Tailnet and make it manager, etc., so that’s what I’ve done. The owner is my Apple account, but I actually do everything admin-wise with a Passkey account.
Oh I had no idea they supported Apple. Maybe they didn't back when I signed up? I'm seemingly stuck with GitHub forever now though.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#170Well written post, really enjoyed reading it. > A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used. This line led me to believe that the writer and checkpointing logic lived on the same database connection, so I was curious to find out how the data race occurred. However, the bug details on the SQ…