Tracking down the 16-year-old WAL-reset SQLite bug • Hacker News Board

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#61

Post by procflora » Wed, Aug 12, 2026, 5:33 PM UTC

Very nice article, and I appreciate SQLite's explanation of the bug too. And how extremely cool Tailscale appears to have been about it (paying for the VFS shim, etc.).

I'd have liked to have heard more about the decision to checkpoint so frequently that put them on this path though. Presumably that's to keep the WAL tiny for very fast recovery. Trying to mitigate some of the deleterious effects of inserting a DBMS into your network layer, I suppose? Tricky stuff. Wonder how that compares to typical etcd snapshot frequencies too.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#62

Post by garganzol » Wed, Aug 12, 2026, 5:33 PM UTC

binhex wrote: post #3

simonw wrote: post #2

> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.

Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.

In real life, an imaginary "leadership" quickly evaporates upon the first hurdle of any kind until proven otherwise by real actions. If I was in your shoes, I would be very careful with blanket statements like these.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#63

Post by ameliaquining » Wed, Aug 12, 2026, 5:34 PM UTC

andai wrote: post #35

SQLite: 92 million lines of tests Dijkstra: Tests can only prove the presence of bugs, never their absence!

I admit to curiosity as to whether static analysis could have caught this. E.g., Rust's type system (yeah yeah I know) catches all data races, unless they originate in unsafe code, which this one might or might not have; a hypothetical Rust SQLite would probably need a lot of unsafe (https://github.com/tursodatabase/turso has 556 unsafe blocks in the core), and I don't have a sense of whether the particular part that contained this bug would be included in that.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#64

Post by AtlasBarfed » Wed, Aug 12, 2026, 5:40 PM UTC

binhex wrote: post #3

simonw wrote: post #2

> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.

Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.

I wish they'd buy the carcass of Keybase from Zoom.

It seems very similar to Tailscale: immense utility from a free-tier product for the general public, which leads to trust and a large enterprise market.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#65

Post by tptacek » Wed, Aug 12, 2026, 5:41 PM UTC

riknos314 wrote: post #10

> Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window. Gotta love single points of failure...

This is maybe one of the purest examples of the Bell Curve Meme in software engineering. The things you would do to the system Tailscale operates to eliminate all single points of failure (generally, and in the specific case where, where the "single point of failure" applies only to a small cohort of customers) would make the system less resilient, and increase failures.

Generally, you do complex distributed systems without on-paper single-points-of-failure anywhere when you absolutely have to, because those systems don't have transient failures. That's not mesh networks like Tailscale at all.

As always: https://how.complexsystems.fail/

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#66

Post by jdjdjebeb » Wed, Aug 12, 2026, 5:44 PM UTC

felixg3 wrote: post #43

LoganDark wrote: post #28

Earlier quoted context omitted.

Tailscale is the best. It's infinitely better than Hamachi, ZeroTier, etc. My only gripe is that they have some really weird SSO requirements like GitHub, etc. and then that provider becomes a permanent part of your identity.

I think their reasoning on not being an identify provider but acting solely downstream is very clever.

Y? What's wrong with providing username/password authentication

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#67

Post by graemep » Wed, Aug 12, 2026, 5:48 PM UTC

saghm wrote: post #59

LoganDark wrote: post #28

Earlier quoted context omitted.

Tailscale is the best. It's infinitely better than Hamachi, ZeroTier, etc. My only gripe is that they have some really weird SSO requirements like GitHub, etc. and then that provider becomes a permanent part of your identity.

I've been a fan of Tailscale since encountering it for the first time at a previous job at a small startup. Someone asked if anyone had a Linux machine when we were all testing out something, and I mentioned I had a personal Linux desktop but wasn't sure how to connect it to the VPN for access, and it turned out that the solution was just literally running two commands in the terminal after installing tailscale from…

With a desktop its usually possible from the network setting GUI? Worked like that last time I needed to use a VPN for access to a corporate network.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#68

Post by wanderr » Wed, Aug 12, 2026, 5:53 PM UTC

This was a great technical writeup and very interesting to read, but it's not clear to me why once the suspected source of the bug was identified, they seemingly didn't build a automated way to trigger the condition? It seems like that could have cut down on the uncertainty of whether the fix worked over a painfully long period of time.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#69

Post by LoganDark » Wed, Aug 12, 2026, 5:59 PM UTC

jdjdjebeb wrote: post #66

felixg3 wrote: post #43

Earlier quoted context omitted.

I think their reasoning on not being an identify provider but acting solely downstream is very clever.

Y? What's wrong with providing username/password authentication

You become responsible for keeping those credentials secure, among a whole host of other problems. (Abuse like credential stuffing (more), data breaches (more), account farms (more), and so on.)

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#70

Post by LoganDark » Wed, Aug 12, 2026, 5:59 PM UTC

artimaeis wrote: post #56

LoganDark wrote: post #28

Earlier quoted context omitted.

Tailscale is the best. It's infinitely better than Hamachi, ZeroTier, etc. My only gripe is that they have some really weird SSO requirements like GitHub, etc. and then that provider becomes a permanent part of your identity.

Just FYI - with Tailscale you can switch SSO providers by putting in a support ticket. I did it last year and it was a breeze.

Oh, thank you! I might do that.