Live data from Hacker News

Tracking down the 16-year-old WAL-reset SQLite bug

tailscale.com

91–100 of 263 posts

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#91
post #87

Was curious so we checked and yep, Antithesis finds this bug in about 15 minutes. Will post a repro/writeup here soon.

I'm equal parts intrigued and skeptical- I guess if the prompt doesn't lead on there is a bug there then I'm impressed.

Stay tuned!

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#94
post #67
post #59

Earlier quoted context omitted.

I've been a fan of Tailscale since encountering it for the first time at a previous job at a small startup. Someone asked if anyone had a Linux machine when we were all testing out something, and I mentioned I had a personal Linux desktop but wasn't sure how to connect it to the VPN for access, and it turned out that the solution was just literally running two commands in the terminal after installing tailscale from…

With a desktop its usually possible from the network setting GUI? Worked like that last time I needed to use a VPN for access to a corporate network.

Some of the NetworkManager VPN plugins have weird side-effects. Like the one for OpenVPN force-disables split tunnel configurations and there's no way to turn that off, meaning that all traffic routes through the VPN even if you'd rather only send stuff for certain destination addresses through.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#95

Glad this got found and fixed, but I continue to be astounded at the amount of work people put into making SQLite do things that would be much simpler with other systems.

Yeah it’s a pity that wasn’t addressed in the article.

This is a little like “we shot ourselves in the foot and then performed surgery on our foot, and everything is resolved now.”

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#96

> Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window. Gotta love single points of failure...

Well it's SPoF for a shard not the whole thing so in practice that makes this considerably better. Secondly, it's not a datapath SPoF so presumably all tailscaled processes would continue running and exchanging traffic no problem - you just won't be able to launch any new ones.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#98
post #79
post #31

Earlier quoted context omitted.

The shard was already a way to make it not a single point of failure.

This is a great example of outages looking different from the perspective of the operator vs the user. Because there's many shards the blast radius of failure is contained to a small subset of users but for those users it's an outage. The way it's designed you can't lose any shards without impacting users. Compare to say Elasticsearch where it's possible to lose nodes and lose shards without the user noticing. One ap…

Of course. On the other hand, is it fair to call it a single point of failure if it being down only affects a shard? Maybe within the shard indeed. Or for a single customer indeed. Not sure if we have terminology to indicate one or the other.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#99
post #35

SQLite: 92 million lines of tests Dijkstra: Tests can only prove the presence of bugs, never their absence!

I admit to curiosity as to whether static analysis could have caught this. E.g., Rust's type system (yeah yeah I know) catches all data races, unless they originate in unsafe code, which this one might or might not have; a hypothetical Rust SQLite would probably need a lot of unsafe ( https://github.com/tursodatabase/turso has 556 unsafe blocks in the core), and I don't have a sense of whether the particular part tha…

>Rust's type system (yeah yeah I know)

What do you know?

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#100
post #83

Earlier quoted context omitted.

I admit to curiosity as to whether static analysis could have caught this. E.g., Rust's type system (yeah yeah I know) catches all data races, unless they originate in unsafe code, which this one might or might not have; a hypothetical Rust SQLite would probably need a lot of unsafe ( https://github.com/tursodatabase/turso has 556 unsafe blocks in the core), and I don't have a sense of whether the particular part tha…

There was a post recently using TLA to model this issue and (re)discover the bug: https://news.ycombinator.com/item?id=48730953 .

Interesting. I don't think this kind of technique could realistically have caught the bug in advance, though; you're not going to model the entire codebase in TLA+, so the investigation in that post modeled just the parts that turned out in hindsight to be relevant.
Post reply on HN