Similar bug to the one that plagued Codex until 3 months ago.
Tracking down the 16-year-old WAL-reset SQLite bug
21–30 of 263 posts
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#22> Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window. Gotta love single points of failure...
What are some solutions to avoid database corruption being single points of failure? I can’t think of any off the top of my head. I don’t think people typically consider database corruption to be a kind of failure common enough to design for, unless you have unusual requirements.
The general answer to this is Byzantine consensus, which cryptocurrency blockchains are designed to solve. If your nodes are willing to fail a little more politely (e.g. no lying, immediately crashing, etc) you can use something cheaper like raft/paxos.
But yeah, it's a lot cheaper to build a reliable system than it is to be resilient.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#23Earlier quoted context omitted.
Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.
Yeah, this part also stuck out to me: > Because this wouldn’t be a quick or easy fix, we reached out to the SQLite developers for a professional support contract. This was a great decision. It gave us direct access to their deep expertise and experience, and we had many detailed technical conversations about our architecture and our incidents. They were willing to pay to get help solving the problem, and then pay aga…
Probably the most refreshing thing I've read in a while. Glad to support them moving forward if this is indeed their modus operandi.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#24>In our control plane, we take manual control of the checkpoint process so we can run fast and consistent backups.
> running boring technology in a non-standard way is a risk.
It was a good read and reminder that the industry is loosing experts gradually. I am not a DBA and yet I have heard about this behavior at least couple times in the past as something to avoid. Its just one of those things which didnt get a chance to be documented cause experts avoided it and regulars didn't get into
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#25>In our control plane, we take manual control of the checkpoint process so we can run fast and consistent backups. > running boring technology in a non-standard way is a risk. It was a good read and reminder that the industry is loosing experts gradually. I am not a DBA and yet I have heard about this behavior at least couple times in the past as something to avoid. Its just one of those things which didnt get a chan…
In other words there exists a concept of HOT and COLD backups for this reason only.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#26I imagine the SQLite eschews AI generated code, but using it for testing (vulnerability, performance, etc) would seem like an easy win. I know their proprietary testing framework is their secret sauce so we may never know...
Richard Hipp's recent talk at Software Should Work explains that AI agents have been testing SQLite and they've gotten a deluge of new bug reports from the fuzz-like testing they can do. But they do not do this in house; hobbyists and other organizations do this in their own internal agent-driven fuzzing.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#27[flagged]
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#28Earlier quoted context omitted.
Yeah, this part also stuck out to me: > Because this wouldn’t be a quick or easy fix, we reached out to the SQLite developers for a professional support contract. This was a great decision. It gave us direct access to their deep expertise and experience, and we had many detailed technical conversations about our architecture and our incidents. They were willing to pay to get help solving the problem, and then pay aga…
Probably the most refreshing thing I've read in a while. Glad to support them moving forward if this is indeed their modus operandi.
Tailscale is the best. It's infinitely better than Hamachi, ZeroTier, etc. My only gripe is that they have some really weird SSO requirements like GitHub, etc. and then that provider becomes a permanent part of your identity.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#29Maybe it's just me, but the explanations of the cause don't align.
One clue was that during corruption incidents, our metrics showed that SQLite would report copying more pages from the WAL file than were actually available. If there are 10 pages in the WAL file and 20 pages get copied to the database, something is clearly wrong.
vs
it thinks some of the pages have been copied from the WAL into the main database file, but they haven’t. Those pages never get written to the database file, and that data is permanently lost.
The first says "more were copied than existed" but the second says "fewer were copied than should have been."
Like I said, it's probably just me interpreting something incorrectly.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#30What a brutal bug. I'd never entertain a that bug in SQLite could be causing problems in code I wrote.