Tracking down the 16-year-old WAL-reset SQLite bug • Hacker News Board

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#231

Post by l72 » Thu, Aug 13, 2026, 1:33 PM UTC

miki123211 wrote: post #224

semiquaver wrote: post #86

Earlier quoted context omitted.

Being an identity provider for anything important is the freaking worst. Exposes you to a million problems. You need human support for login problems and lost MFA tokens, and you are an attack magnet.

Which is why you want magic links. Don't be the identity provider, have the email host be the identity provider (which it is anyway if you have a forgot password prompt). Agreed 100% that nobody should still be using passwords in 2026 though.

As a user, I really don't care for magic links. The whole, start the log-in process, switch context, wait for email (sometimes up to a minute), click on it, have it open a new tab in a different window than where I started is just a pain. I feel like I spend half my day logging in to services these days.

The only time I like magic links is for services where I am "not really a user". For example, an appointment reminder for my doctor where I need to validate my insurance. Great, send me an email 24-hours before with the reminder and a magic link, as I don't want to think about an account there.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#232

Post by sandeepkd » Thu, Aug 13, 2026, 1:55 PM UTC

inigyou wrote: post #133

sandeepkd wrote: post #24

>In our control plane, we take manual control of the checkpoint process so we can run fast and consistent backups. > running boring technology in a non-standard way is a risk. It was a good read and reminder that the industry is loosing experts gradually. I am not a DBA and yet I have heard about this behavior at least couple times in the past as something to avoid. Its just one of those things which didnt get a chan…

I've never heard of any reliability reason you shouldn't run checkpoints whenever you want - only performance reasons. Can you elaborate? Backing up sqlite by copying the file (e.g. rsync) while it's open is a surefire way to eventually get corruption caused by a race condition, but it seems like tailscale wasn't doing that. They were probably using the proper sqlite backup API. But you don't need checkpoints for con…

My generalist reading of this (which is applicable in lot of cases) is that a dedicated flow is handling the house keeping part of the job, aka resetting the checkpoint after writing the WAL to disk. These one off house keepers are common in a lot of softwares and they expect to work alone, they are tested to work alone. They always have a set of ritual, rules and order in which they do all the things. Now if some other thread takes off some of those jobs then they break this routine for the house keeping job and this new thread/person may not always know what else has to be done before and after this one particular job for the sake of completeness.

On second question of why the tailscale developers did it, its possibly for the same reason why they invested this much into debugging this issue. Some one believed the current behavior did not fit into their architecture, they want to be more performant and take control over things. A big part of me considers this is a required exercise to try, grow and learn. The only thing they could have for improvement would be to have these old hands on architect kind of folks on their team who might have hinted/pointed them to the problem a long before. Challenge/chances are that these older folks would have even stopped them from going in this direction in the design phase itself.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#233

Post by serf » Thu, Aug 13, 2026, 1:56 PM UTC

miki123211 wrote: post #222

chillaranand wrote: post #189

Earlier quoted context omitted.

Important features like App Connectors won't work with HeadScale. Unfortunately, HeadScale is not a drop in replacement for TailScale. Most people don't realize this until they start self hosting. Due to this, I had to migrate from Tailscale to NetBird, which is completely open source. https://avilpage.com/2026/06/moving-from-tailscale-to-netbir...

Is this because HS hasn't implemented it yet (which is on the community), or because Tailscale does something to prevent them from working specifically?

it's a proposed and planned feature by the looks of it :

https://github.com/juanfont/headscale/issues/1651

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#234

Post by IX-103 » Thu, Aug 13, 2026, 2:23 PM UTC

aw1621107 wrote: post #220

inigyou wrote: post #205

Earlier quoted context omitted.

You can't even exhaustively test every 64 bit integer value.

Just because you can't exhaustively test one thing doesn't mean you can't exhaustively test anything.

Not being able to exhaustively test simple problem absolutely means you can't exhaustively test massively more complex problem. For a monotonically increasing function f, which I think we can agree a function mapping from the size of the input set to the runtime of an exhaustive execution of all elements in the set qualifies as, if |small| < |large| then f(small) < f(large). Therefore if f(small) is too big then f(large) is definitely too big.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#235

Post by 2afTq » Thu, Aug 13, 2026, 2:23 PM UTC

simonw wrote: post #2

> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.

New? The sqlite VSF shim is much older. Nice spin.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#236

Post by raggi » Thu, Aug 13, 2026, 2:35 PM UTC

inigyou wrote: post #136

raggi wrote: post #122

Earlier quoted context omitted.

We have very good reasons for our checkpointing model, related to our backup + disaster recover strategy, along with resource cost. It might be worth writing about one day, so I'll not give away all the details, but in very short form, we organize a backup strategy that has minimal pause time, avoids doubling the page cache cost of the database, and enables extremely fast byte-copy restores in disaster recovery.

okay but explain why you are using sqlite and copying the file to S3 instead of using any client/server DB and its online backup feature?

SQLite has an online backup API as well, but it is slower and requires a significant additional page cache cost.

The team chose SQLite early on (there are some blog posts about this) and then we vertically scaled against the SQLite architecture. There are subtle ways you come to depend on the proximity/latency when you scale with local storage that mean switching requires a lot of non-obvious work - it’s probably the largest hazard for embracing SQLite in a growing saas - but at the same time you can push the vertical scale pretty far, which has great margins.

Had we scaled a different architecture of database there’s little reason to believe it would have been plain sailing as seems to be implied here.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#237

Post by chuckadams » Thu, Aug 13, 2026, 2:50 PM UTC

petcat wrote: post #228

bhaak wrote: post #209

Earlier quoted context omitted.

> Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however. https://sqlite.org/faq.html#q5 One writer, multiple readers is a specifically supported way of using SQLite. Why should you be worried if it is used as designed?

> Why should you be worried if it is used as designed? Well this whole article is about a company discovering a catastrophic corruption bug even though they were using it as designed. I think the lesson is that if you're ever actually worried about concurrency then just don't use sqlite. We can see here that concurrency is hard and the bugs are old and deep.

I guarantee that any hand-rolled replacement will have more and worse bugs.

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#238

Post by simonw » Thu, Aug 13, 2026, 3:00 PM UTC

2afTq wrote: post #235

simonw wrote: post #2

> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.

New? The sqlite VSF shim is much older. Nice spin.

Spin?

Looks to me like this is the tool in question, added in January: https://github.com/sqlite/sqlite/commits/master/ext/misc/tms...

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#239

Post by Ariarule » Thu, Aug 13, 2026, 4:52 PM UTC

ec109685 wrote: post #190

dboreham wrote: post #32

Earlier quoted context omitted.

Quick note that data corruption bugs that are impossible to reproduce are not uncommon (perhaps they're the norm). So some amount of head scratching trying to figure out a plausible scenario by which the system could get into the state represented by the smoking remains is often required. Then you attempt to force it into the supposed bad state by modifying code paths accordingly. So the approach used in this case is…

I agree it's tricky, but Tailscale had downtime by this bug and the SQLite Changelog downplayed it.

How is "Fix the WAL-reset database corruption bug" downplaying this? https://www.sqlite.org/changes.html

Re: Tracking down the 16-year-old WAL-reset SQLite bug

#240

Post by andai » Thu, Aug 13, 2026, 4:56 PM UTC

layer8 wrote: post #219

otterley wrote: post #46

Earlier quoted context omitted.

Everyone knows that tests don't prevent all bugs. But they are very good at preventing known bugs from recurring in the future.

Everyone doesn’t seem to know that, because tests are often cited as a way to ensure that AI-generated code is correct.

I was very excited about formal proofs, which are now very cheap to produce, in service of validating AI generated code.

But I had a funny experience recently where an agent implemented an entire feature completely wrong (exactly backwards, actually, in a way that defeated the purpose, introduced security issues etc.).

It happily supplied tests for the new functionality, and all the tests passed.

What I realized was, even formal verification wouldn't have helped here -- it would have just written a mathematical proof that the incorrect functionality was correctly implemented!

So there's a gap here, where first, the human's intention needs to be formally specified (by the human, or at least the human needs to be able and willing to verify it), and then the slopswarm can hack away at it...