[flagged]
Tracking down the 16-year-old WAL-reset SQLite bug
251–260 of 263 posts
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#252Earlier quoted context omitted.
Yeah, this part also stuck out to me: > Because this wouldn’t be a quick or easy fix, we reached out to the SQLite developers for a professional support contract. This was a great decision. It gave us direct access to their deep expertise and experience, and we had many detailed technical conversations about our architecture and our incidents. They were willing to pay to get help solving the problem, and then pay aga…
It's common for databases. This is Percona's business model. They employ core pg/mysql developers and you can buy a support package from them. Same for enterprisedb. Pretty reasonably priced packages (like maybe $10k-ish/core IIRC) get you 24x7 support. I've only had to escalate issues once but inside 10 hours we got a pg core committer to debug some very strange vacuum behavior.
To clarify, I'm not saying that support contracts are uncommon. I'm saying that going out of their way to then additionally sponsor development of a specific testing tool on top of the support contract is not something I've heard companies do before, and it's pretty awesome!
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#253Earlier quoted context omitted.
Based on the experience that I (and other coworkers, including on other distros) had, whatever configurations our VPN needed did not seem to work out of the box on network manager.
A lot of this is just weird Linux on the desktop weirdness. On Windows and Mac, VPN config is usually just installing a client, signing in, and then it all basically works. Sometimes you don't even need the client and can set it up in settings.
Those are pretty much all first-party clients though, right? I'm not sure that "some companies don't provide a Linux client for their software" is something I'd characterize as "Linux desktop weirdness". My point here is that Tailscale seems to have actually put time and effort into making a Linux client that's a breeze to use, whereas most of the other VPNs I've used for work have not.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#254Earlier quoted context omitted.
Yeah, this part also stuck out to me: > Because this wouldn’t be a quick or easy fix, we reached out to the SQLite developers for a professional support contract. This was a great decision. It gave us direct access to their deep expertise and experience, and we had many detailed technical conversations about our architecture and our incidents. They were willing to pay to get help solving the problem, and then pay aga…
This isn't exactly new; the other model is to just hire them directly. Unfortunately this has not generally led to open source outside of the enterprise world getting similar funding, but it is what it is.
What other examples of companies paying for something specific to be developed outside of their ongoing paid support contract can you point me to? I'm open to the possibility that this is more common than I realize, but I've yet to hear of another example of sponsoring development separately from support contracts.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#255Earlier quoted context omitted.
Important features like App Connectors won't work with HeadScale. Unfortunately, HeadScale is not a drop in replacement for TailScale. Most people don't realize this until they start self hosting. Due to this, I had to migrate from Tailscale to NetBird, which is completely open source. https://avilpage.com/2026/06/moving-from-tailscale-to-netbir...
Is this because HS hasn't implemented it yet (which is on the community), or because Tailscale does something to prevent them from working specifically?
Is it on the community, when TS have said that they won't open source their control plane because HS is good enough / easier to run?
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#256Earlier quoted context omitted.
Yeah, I also feel 'just use SQLite [no matter what]' is just the pendulum swinging hard after the 'just use mongodb [no matter what]' of yesteryear. It's so sometimes just performative. I remember when Tailscale had a similar performative approach with 'just use a JSON file on disk'. Then etcd. Then SQLite. Like sure, you can keep picking the absolite mininum technology for your needs and then change it every couple…
Postgres has also had pretty serious bugs, eg fsyncgate
There's also the more fundamental issue that Postgres - unlike something like Yugabyte - does not use a distributed consensus algorithm for writes and can lose committed writes during network partitions.
But of course, neither does Tailscale's weird DIY contraption.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#257Well written post, really enjoyed reading it. > A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used. This line led me to believe that the writer and checkpointing logic lived on the same database connection, so I was curious to find out how the data race occurred. However, the bug details on the SQ…
Database corruption is due to 2+ peer connects with 744 file permission entering header rwxr -.- WAL write new content into secondary header tag: inter-element whitespace.
Bug details:
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#258Earlier quoted context omitted.
I agree it's tricky, but Tailscale had downtime by this bug and the SQLite Changelog downplayed it.
How is "Fix the WAL-reset database corruption bug" downplaying this? https://www.sqlite.org/changes.html
> The bug is a data race with tight timing constraints. It is unlikely to occur in common use. The developers have never been able to reproduce the bug organically and had to add special testing logic to SQLite that deliberately triggers the circumstances of the the bug in order to verify that the issue has been fixed.
I read that and it feels like some theoretical thing versus causing impactful outages for workloads shaped like tailscale’s.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#259SQLite: 92 million lines of tests Dijkstra: Tests can only prove the presence of bugs, never their absence!
Obligatory UNIX fortune: "Computer Science can be thought of as the post-Turing decline in formal methods."
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#260Earlier quoted context omitted.
I wish they'd buy the carcass of Keybase from Zoom. It seems very similar to Tailscale: immense utility from a free-tier product for the general public, which leads to trust and a large enterprise market.
Why would enterprises need Keybase's product?
We used it for dozens of things back at my old job. Send each other passwords. Informal credentials store fo one off projects and non production. Sensitive project documentation