Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking down the 16-year-old WAL-reset SQLite bug

tailscale.com

Post reply on HN
Print view View original

41–50 of 263 posts

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#41

Post by x0x0 »

saghm wrote: post #6
binhex wrote: post #3

Earlier quoted context omitted.

Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.

Yeah, this part also stuck out to me: > Because this wouldn’t be a quick or easy fix, we reached out to the SQLite developers for a professional support contract. This was a great decision. It gave us direct access to their deep expertise and experience, and we had many detailed technical conversations about our architecture and our incidents. They were willing to pay to get help solving the problem, and then pay aga…

It's common for databases.

This is Percona's business model. They employ core pg/mysql developers and you can buy a support package from them. Same for enterprisedb. Pretty reasonably priced packages (like maybe $10k-ish/core IIRC) get you 24x7 support. I've only had to escalate issues once but inside 10 hours we got a pg core committer to debug some very strange vacuum behavior.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#42

Post by bch »

This was really, really interesting - what a triumphant adventure.

A few (very, very, very pedantic) things that stood out:

> We wanted a way to restore service that didn’t involve rolling back to the last known-good backup (which would lose a lot of data) or repairing the known-corrupted database (which was potentially risky).

(Emphasis mine) - it would be "risky", not "potentially risky" - then the "calculated risk period" starts and it's "potentially problematic".

In the SQLite report[0] (11.2) I wish they downplayed this less - a mention of the rarity, then technical details - I'm friendly with a few of the devs/previous-devs, have the utmost respect for their skill and accomplishments (and by extension, faith that the developers I do not personally interact with are also excellent), appreciation and fondness for the huge accomplishment that is SQLite, and on and on... this is world-class work. Maybe section 11.2 wasn't really aimed at me, or I'm too critical. To be fair to all involved, what a minor quibble for such an interesting problem/fix. I hope my comment isn't a fly in the ointment.

Last bugfix point[1] - ugh. What a sinking feeling that must've been to deploy a fix then be flooded with not-green - and a lesson[2] against smuggling other changes in a changeset "just because we're already here"? Happy it turned out non-catastrophic, but did result in a rare (not remembering other instances of top of head) recall[3] from SQLite. That it was throwing errors at the same time SQLite and Tailscale were testing the other WAL-issue bug must've upset some stomachs for a moment.

[0] https://sqlite.org/wal.html#the_wal_reset_bug

[1] https://tailscale.com/blog/sqlite-wal-reset-bug#fixed-with-a...

[2] Nobody conceptually learned anything here - we're all just reminded of what we know: that sometimes "perfect storms" do actually occur.

[3] https://sqlite.org/releaselog/3_52_0.html

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#43

Post by felixg3 »

LoganDark wrote: post #28
jjordan wrote: post #23

Earlier quoted context omitted.

Probably the most refreshing thing I've read in a while. Glad to support them moving forward if this is indeed their modus operandi.

Tailscale is the best. It's infinitely better than Hamachi, ZeroTier, etc. My only gripe is that they have some really weird SSO requirements like GitHub, etc. and then that provider becomes a permanent part of your identity.

I think their reasoning on not being an identify provider but acting solely downstream is very clever.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#44

Post by arjie »

riknos314 wrote: post #10

> Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window. Gotta love single points of failure...

You don’t need the control plane most of the time. I had a zero downtime headscale upgrade because once the nodes negotiate through the control plane they can talk to each other all the time. The data plane is peer to peer.

It’s problematic because you can’t run connections but it doesn’t stop the world.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#47

Post by freeAgent »

LoganDark wrote: post #28
jjordan wrote: post #23

Earlier quoted context omitted.

Probably the most refreshing thing I've read in a while. Glad to support them moving forward if this is indeed their modus operandi.

Tailscale is the best. It's infinitely better than Hamachi, ZeroTier, etc. My only gripe is that they have some really weird SSO requirements like GitHub, etc. and then that provider becomes a permanent part of your identity.

Yeah, I have my Tailscale tied to my Apple account, which just feels weird. I can add a Passkey account to my Tailnet and make it manager, etc., so that’s what I’ve done. The owner is my Apple account, but I actually do everything admin-wise with a Passkey account.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#48

Post by ball_of_lint »

surgical_fire wrote: post #37
LgWoodenBadger wrote: post #29

Maybe it's just me, but the explanations of the cause don't align. One clue was that during corruption incidents, our metrics showed that SQLite would report copying more pages from the WAL file than were actually available. If there are 10 pages in the WAL file and 20 pages get copied to the database, something is clearly wrong. vs it thinks some of the pages have been copied from the WAL into the main database file…

My interpretation is that they haven't been copied because they didn't exist? If you have 10 pages and it tries to copy 20, either those 10 pages wouldn't really be copied, or bogus data would be written. That's how I read at least. Those things are not mutually exclusive.

Or you could have 10 pages, it actually copies 9, and reports 20 anyways.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#50

Post by LoganDark »

freeAgent wrote: post #47
LoganDark wrote: post #28

Earlier quoted context omitted.

Tailscale is the best. It's infinitely better than Hamachi, ZeroTier, etc. My only gripe is that they have some really weird SSO requirements like GitHub, etc. and then that provider becomes a permanent part of your identity.

Yeah, I have my Tailscale tied to my Apple account, which just feels weird. I can add a Passkey account to my Tailnet and make it manager, etc., so that’s what I’ve done. The owner is my Apple account, but I actually do everything admin-wise with a Passkey account.

Oh I had no idea they supported Apple. Maybe they didn't back when I signed up? I'm seemingly stuck with GitHub forever now though.
Post reply on HN