Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking down the 16-year-old WAL-reset SQLite bug

tailscale.com

Post reply on HN
Print view View original

11–20 of 263 posts

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#11

Post by kccqzy »

riknos314 wrote: post #10

> Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window. Gotta love single points of failure...

What are some solutions to avoid database corruption being single points of failure? I can’t think of any off the top of my head. I don’t think people typically consider database corruption to be a kind of failure common enough to design for, unless you have unusual requirements.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#13

Post by Ariarule »

ec109685 wrote: post #9

While technically true as written, it seems to downplay the significance: > The bug is a data race with tight timing constraints. It is unlikely to occur in common use. A large customer did experience this corruption, so it's important for people with tailscale's setup update immediately. > The developers have never been able to reproduce the bug organically and had to add special testing logic to SQLite that deliber…

Odd not to highlight the sentence where they answer the obvious question "Why Tailscale in particular?":

> They also explained why we were more likely to hit the bug than other SQLite users: we take manual control of the checkpointing process, and we checkpoint very aggressively. Even a bug triggered by a rare condition was bound to hit us eventually.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#14

Post by EastSmith »

simonw wrote: post #2

> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.

Started using them like 2 weeks back, happy to see how they work.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#15

Post by packetlost »

binhex wrote: post #3
simonw wrote: post #2

> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.

Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.

Avery is one of the few people I have enough respect for to look up to.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#18

Post by maitrungduc »

calmingsolitude wrote: post #8

Well written post, really enjoyed reading it. > A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used. This line led me to believe that the writer and checkpointing logic lived on the same database connection, so I was curious to find out how the data race occurred. However, the bug details on the SQ…

[flagged]

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#20

Post by jeffbee »

Block device upfuckery layers are powerful against databases. Years ago some colleagues wrote one that provides most of the hazards described by "Parity Lost and Parity Regained"[1] to test FoundationDB, which immediately uncovered several flaws in a project that described itself as well-tested. It's easy to do this with all the probing features that Linux (and others) provide today.

1: https://www.usenix.org/legacy/event/fast08/tech/full_papers/...

Post reply on HN