SQLite: 92 million lines of tests Dijkstra: Tests can only prove the presence of bugs, never their absence!
https://news.ycombinator.com/item?id=18442941
Jesus Christ. I have this comment saved, but it gets me every time.
Tracking down the 16-year-old WAL-reset SQLite bug
241–250 of 263 posts
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#242[flagged]
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#243Earlier quoted context omitted.
Just because you can't exhaustively test one thing doesn't mean you can't exhaustively test anything.
Not being able to exhaustively test simple problem absolutely means you can't exhaustively test massively more complex problem . For a monotonically increasing function f, which I think we can agree a function mapping from the size of the input set to the runtime of an exhaustive execution of all elements in the set qualifies as, if |small| < |large| then f(small) < f(large). Therefore if f(small) is too big then f(l…
> Not being able to exhaustively test simple problem absolutely means you can't exhaustively test massively more complex problem.
"Simple"/"complex" doesn't necessarily imply "small"/"large", respectively. Testing whether the Collatz conjecture holds is "simple", but its search space is (potentially) infinitely large. Testing correctness of concurrent data structures is (probably) a more "complex" problem, and yet it's still possible to perform exhaustive tests of their functionality [0].
In the end, the practicality of exhausting the 64-bit integer search space is irrelevant since it says nothing about the size of the write + checkpoint search space. I could just as easily say that it's easy to exhaust the 16-bit integer search space, but that doesn't mean exhaustively testing for this particular bug is easy.
[0]: http://demsky.eecs.uci.edu/publications/c11modelcheck.pdf
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#244For this particular category of bug, SQLite's existing testing methodology is demonstrably outclassed by modern deterministic concurrency testing. https://antithesis.com/blog/2026/wal-reset-bug/
If you already know the bug is caused by a race between writing and checkpointing then it's easy. I'm unsure how the linked article makes your point at all though.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#245Earlier quoted context omitted.
Being an identity provider for anything important is the freaking worst. Exposes you to a million problems. You need human support for login problems and lost MFA tokens, and you are an attack magnet.
Which is why you want magic links. Don't be the identity provider, have the email host be the identity provider (which it is anyway if you have a forgot password prompt). Agreed 100% that nobody should still be using passwords in 2026 though.
I detest magic links. I have probably 9-10 accounts with a certain service that uses magic links (I mostly use them to get an API key or download a file every few months and never visit directly otherwise) because
1. the only login method they support is magic link and login with Google/Facebook. There's no password field so my password manager can't fill out my login details for me.
2. I don't have a Facebook account, and I don't keep my Google account signed in. Even if I did, I don't want them using these as my primary email address. (I use a catch-all and don't give out the primary address; I give each service it's own address)
3. A bunch of newsletters and forums (hackaday, etc) mention them by name every few weeks, often in the subject field, making it hard to search for. They don't send me any newsletters themselves, and I don't participate in their forums so there's nothing from them directly. They can't do anything about this, but it makes things harder.
4. If you enter a new email address that they've never seen before, the process exactly the same until after you click the magic link you received. They don't tell you they've never seen it before until you verify the address.
5. As such, if I can't recall the exact address I used to sign in, I automatically create a new account and don't find out until after I click the magic link and I'm asked to enter a new username or use the existing username they've already created. Proceed to settings and create a new API key for a new account instead of just adding one to my existing account.
This is an extreme case, yes, but I have similar issues with several services that only support magic links and GitHub login, and it's hard to configure services to use a new catch-all address when they harvest my address from GitHub login and won't let me change it. Magic links are a great tool, but they should _never_ be the only option. They should supplement existing login methods. Password managers exist for a reason. Several services that I have since stopped using for this reason have in the past decided that they should start sending emails to the addresses they harvested from login with other service. Two services have gotten my magic link address stuck in their system because it's not associated with an account (the account has been deleted) and thus I can't manage subscriptions.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#246Earlier quoted context omitted.
Everyone doesn’t seem to know that, because tests are often cited as a way to ensure that AI-generated code is correct.
I was very excited about formal proofs, which are now very cheap to produce, in service of validating AI generated code. But I had a funny experience recently where an agent implemented an entire feature completely wrong (exactly backwards, actually, in a way that defeated the purpose, introduced security issues etc.). It happily supplied tests for the new functionality, and all the tests passed. What I realized was,…
> first, the human's intention needs to be formally specified
This never changed with AI; in fact, I think it made this need more visible than it ever had been before. You can't get away with not being able to describe in detail what you want. As with working with humans, any ambiguities will be interpreted, and not always in the way you hoped.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#247Earlier quoted context omitted.
Everyone doesn’t seem to know that, because tests are often cited as a way to ensure that AI-generated code is correct.
I was very excited about formal proofs, which are now very cheap to produce, in service of validating AI generated code. But I had a funny experience recently where an agent implemented an entire feature completely wrong (exactly backwards, actually, in a way that defeated the purpose, introduced security issues etc.). It happily supplied tests for the new functionality, and all the tests passed. What I realized was,…
> the human's intention needs to be formally specified
Yes, that has always been the case with formal proofs. Even a mathematical proof with Lean is useless if the proven formal statement doesn’t match what the mathematician meant. And that’s why humans working with formal languages won’t be going away. Natural language is imprecise by nature, even when we have precise ideas in mind. In that case, it’s often better to specify what we mean in a formal language than in natural language.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#248Everybody here knows that SQLite is not production-ready and does not scale.
This is just proof that it can't be used in real-world applications.
SCNR
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#249This was a great technical writeup and very interesting to read, but it's not clear to me why once the suspected source of the bug was identified, they seemingly didn't build a automated way to trigger the condition? It seems like that could have cut down on the uncertainty of whether the fix worked over a painfully long period of time.
The sqlite dev team did. It's in the article. > It could exist that long because it was rare—so rare, the SQLite developers had to add code to deliberately trigger it in their testing environments.
Yes of course I mean the tailscale team. Having an independent way to repro the bug besides waiting for it to happen in prod seems pretty basic eng best practice.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#250Earlier quoted context omitted.
Yeah it’s a pity that wasn’t addressed in the article. This is a little like “we shot ourselves in the foot and then performed surgery on our foot, and everything is resolved now.”
That assumes that they do actually believe it's a mistake. They didn't explain the reasons they've gone for this architecture in much/any detail. I'd be interested in hearing them talk more about that in the future. Perhaps you or I would make a different decision based on the aims that lead them there. But there isn't enough information to say whether or not their decision was a mistake, even if it has lead to a pec…
Right, the point is it's a kind of elephant in the room - they even allude to it not being a common use case - but they say nothing about the rationale. That leads me to suspect it was a kind of "seemed like a good idea at the time" situation.
They also touch on this when they say that it worked for them for a long time.
This is a pretty classic symptom of a system that was designed a certain way early on and then runs into issues as the system grows.
One can argue that this was due to a bug, but it's a bug that they shouldn't really have had to deal with - a consequence that the design opened them up to.