Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking down the 16-year-old WAL-reset SQLite bug

tailscale.com

Post reply on HN
Print view View original

181–190 of 263 posts

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#181

Post by ashton314 »

aw1621107 wrote: post #149
inigyou wrote: post #131

Earlier quoted context omitted.

no it can't. what if this bug is still there but the timing window is now one CPU instruction?

> what if this bug is still there but the timing window is now one CPU instruction? I think a model checker or something similar that can exhaustively cover a search space would suffice, though at that point I think the boundary between "test" and "formal verification" becomes somewhat fuzzy.

> exhaustively cover a search space

This is the gotcha: in stateful environments you'll have state-space explosion so big you'll have to come up with some compromising measure of "equivalent". You're right that that would constitute a proof by exhaustion of correctness. Unfortunately in practice it's entirely impractical.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#183

Post by bradfitz »

xyst wrote: post #143
buggymcbugfix wrote: post #113

Earlier quoted context omitted.

Not only are they funding open source, they are actively allowing users to host their own control plane via headscale which is a libre implementation of the tailscale control protocol and developed by an engineer who works for Tailscale by day. This instantly made me trust and like them, even if at first I was cautious because I naturally mistrust anything that gets a lot of hype. I've been running headscale on NixOS…

The only nit on my end is that you have to configure a few settings on client and server to avoid sending diagnostics to TS. For macOS, you need to build the open source tailscale client. Or create file in esoteric file location to opt out using App Store. For iOS I don’t even think it’s possible to opt out.

Oh? I thought we added that option on iOS recently. I'll go check.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#184

Post by waterTanuki »

inigyou wrote: post #163
waterTanuki wrote: post #141

Earlier quoted context omitted.

Tailscale likely deals with a lot of security-sensitive traffic, given the nature of the service. I'm guessing one of the requirements they were given was to have a tiny blast radius in case encryption keys got leaked, and that meant isolating each customer's tailnet (meta)data to it's own sqlite db rather than letting everyone share a postgres cluster.

How about having devices generate and not share their private keys, instead?

That's already how it works at the data-plane layer. I'm talking about the control-plane layer where tailscale needs to maintain a server that gets its state from somewhere (they need to know where to route your data, what your permissions ACLs are, device names, etc.). If everyone shared the same db cluster accessed over the network there is a real risk of an accident leaking the encryption keys (of the database itself, not your tailnet) to the db they would have to maintain, and leaking semi-sensitive info like these device names and IP addresses. Using sqlite means each tailnet's metadata is isolated to the container running it.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#186

Post by rcleveng »

packetlost wrote: post #15
binhex wrote: post #3

Earlier quoted context omitted.

Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.

Avery is one of the few people I have enough respect for to look up to.

Avery is amazing, reading his writings was one of the highlights of my time at Google. There were many talented and articulate people there, the level of discourse used to be amazing.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#187

Post by fragmede »

garganzol wrote: post #62
binhex wrote: post #3

Earlier quoted context omitted.

Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.

In real life, an imaginary "leadership" quickly evaporates upon the first hurdle of any kind until proven otherwise by real actions. If I was in your shoes, I would be very careful with blanket statements like these.

This would be the "proven otherwise by real actions" part. They've been around since 2017 and have had plenty of time to hit various hurdles. Past performance is no guarantee of future performance and it is within the realm of possibility for them to go of change course like Elon did. But who are you? Are your Internet comments some proenoucement of truth that can never ever be taken back? So you said something that, years later, turns out to be wrong. So what?

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#188

Post by computerfriend »

ameliaquining wrote: post #63
andai wrote: post #35

SQLite: 92 million lines of tests Dijkstra: Tests can only prove the presence of bugs, never their absence!

I admit to curiosity as to whether static analysis could have caught this. E.g., Rust's type system (yeah yeah I know) catches all data races, unless they originate in unsafe code, which this one might or might not have; a hypothetical Rust SQLite would probably need a lot of unsafe ( https://github.com/tursodatabase/turso has 556 unsafe blocks in the core), and I don't have a sense of whether the particular part tha…

I wouldn't think so, but Go also has a data race detector: https://go.dev/doc/articles/race_detector.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#189

Post by chillaranand »

buggymcbugfix wrote: post #113
simonw wrote: post #2

> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.

Not only are they funding open source, they are actively allowing users to host their own control plane via headscale which is a libre implementation of the tailscale control protocol and developed by an engineer who works for Tailscale by day. This instantly made me trust and like them, even if at first I was cautious because I naturally mistrust anything that gets a lot of hype. I've been running headscale on NixOS…

Important features like App Connectors won't work with HeadScale.

Unfortunately, HeadScale is not a drop in replacement for TailScale. Most people don't realize this until they start self hosting.

Due to this, I had to migrate from Tailscale to NetBird, which is completely open source.

https://avilpage.com/2026/06/moving-from-tailscale-to-netbir...

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#190

Post by ec109685 »

dboreham wrote: post #32
ec109685 wrote: post #9

While technically true as written, it seems to downplay the significance: > The bug is a data race with tight timing constraints. It is unlikely to occur in common use. A large customer did experience this corruption, so it's important for people with tailscale's setup update immediately. > The developers have never been able to reproduce the bug organically and had to add special testing logic to SQLite that deliber…

Quick note that data corruption bugs that are impossible to reproduce are not uncommon (perhaps they're the norm). So some amount of head scratching trying to figure out a plausible scenario by which the system could get into the state represented by the smoking remains is often required. Then you attempt to force it into the supposed bad state by modifying code paths accordingly. So the approach used in this case is…

I agree it's tricky, but Tailscale had downtime by this bug and the SQLite Changelog downplayed it.
Post reply on HN