Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking down the 16-year-old WAL-reset SQLite bug

tailscale.com

Post reply on HN
Print view View original

141–150 of 263 posts

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#141

Post by waterTanuki »

danpalmer wrote: post #76

Glad this got found and fixed, but I continue to be astounded at the amount of work people put into making SQLite do things that would be much simpler with other systems.

Tailscale likely deals with a lot of security-sensitive traffic, given the nature of the service. I'm guessing one of the requirements they were given was to have a tiny blast radius in case encryption keys got leaked, and that meant isolating each customer's tailnet (meta)data to it's own sqlite db rather than letting everyone share a postgres cluster.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#142

Post by [deleted] »

calmingsolitude wrote: post #8

Well written post, really enjoyed reading it. > A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used. This line led me to believe that the writer and checkpointing logic lived on the same database connection, so I was curious to find out how the data race occurred. However, the bug details on the SQ…

[deleted]

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#143

Post by xyst »

buggymcbugfix wrote: post #113
simonw wrote: post #2

> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.

Not only are they funding open source, they are actively allowing users to host their own control plane via headscale which is a libre implementation of the tailscale control protocol and developed by an engineer who works for Tailscale by day. This instantly made me trust and like them, even if at first I was cautious because I naturally mistrust anything that gets a lot of hype. I've been running headscale on NixOS…

The only nit on my end is that you have to configure a few settings on client and server to avoid sending diagnostics to TS.

For macOS, you need to build the open source tailscale client. Or create file in esoteric file location to opt out using App Store.

For iOS I don’t even think it’s possible to opt out.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#144

Post by xyst »

binhex wrote: post #3
simonw wrote: post #2

> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.

Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.

At least for now. All it takes is one greedy executive decision to stop supporting headscale. Given it’s an American company, it’s entirely possible.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#146

Post by q3k »

inigyou wrote: post #129

This reaffirms my belief that SQLite is not well suited for systems with significant concurrency. It replaces fopen, not postgres. Although this corruption is a rare bug and sqlite is usually extremely stable, it's usually not worth it from a performance and features standpoint either. Here they were trying to do a backup by forcing a checkpoint and then copying the file. Systems like postgres let you do online conti…

Yeah, I also feel 'just use SQLite [no matter what]' is just the pendulum swinging hard after the 'just use mongodb [no matter what]' of yesteryear.

It's so sometimes just performative. I remember when Tailscale had a similar performative approach with 'just use a JSON file on disk'. Then etcd. Then SQLite. Like sure, you can keep picking the absolite mininum technology for your needs and then change it every couple of years... Or you could just immediately go with a solid Postgres (or Postgres-like setup, eg. yugabyte) and save yourself a bunch of faffing about with weird solutions and migrating between them. But I guess that doesn't drive engagement on your blog.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#147

Post by [deleted] »

q3k wrote: post #146
inigyou wrote: post #129

This reaffirms my belief that SQLite is not well suited for systems with significant concurrency. It replaces fopen, not postgres. Although this corruption is a rare bug and sqlite is usually extremely stable, it's usually not worth it from a performance and features standpoint either. Here they were trying to do a backup by forcing a checkpoint and then copying the file. Systems like postgres let you do online conti…

Yeah, I also feel 'just use SQLite [no matter what]' is just the pendulum swinging hard after the 'just use mongodb [no matter what]' of yesteryear. It's so sometimes just performative. I remember when Tailscale had a similar performative approach with 'just use a JSON file on disk'. Then etcd. Then SQLite. Like sure, you can keep picking the absolite mininum technology for your needs and then change it every couple…

[deleted]

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#148

Post by [deleted] »

[deleted] wrote: post #147
q3k wrote: post #146

Earlier quoted context omitted.

Yeah, I also feel 'just use SQLite [no matter what]' is just the pendulum swinging hard after the 'just use mongodb [no matter what]' of yesteryear. It's so sometimes just performative. I remember when Tailscale had a similar performative approach with 'just use a JSON file on disk'. Then etcd. Then SQLite. Like sure, you can keep picking the absolite mininum technology for your needs and then change it every couple…

[deleted]

[deleted]

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#149

Post by aw1621107 »

inigyou wrote: post #131
0x457 wrote: post #49

Earlier quoted context omitted.

It can prove absence of specific bugs though.

no it can't. what if this bug is still there but the timing window is now one CPU instruction?

> what if this bug is still there but the timing window is now one CPU instruction?

I think a model checker or something similar that can exhaustively cover a search space would suffice, though at that point I think the boundary between "test" and "formal verification" becomes somewhat fuzzy.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#150

Post by j16sdiz »

fragmede wrote: post #138
ameliaquining wrote: post #100

Earlier quoted context omitted.

Interesting. I don't think this kind of technique could realistically have caught the bug in advance, though; you're not going to model the entire codebase in TLA+, so the investigation in that post modeled just the parts that turned out in hindsight to be relevant.

> you're not going to model the entire codebase in TLA+ Not by hand in the olden days but with AI to help with it, why not?

Because AI have a recall of 80% or 90% at best. While it might be better than some human, it won't hit the 100% TLA+ modeling aims
Post reply on HN