Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Breaking the WAL

antithesis.com

Post reply on HN
Print view View original

21–30 of 59 posts

Re: Breaking the WAL

#21

Post by carlsverre »

MPSimmons wrote: post #15
carlsverre wrote: post #9

Hi! I'm the author and the person who was on a road trip when I decided to do this experiment back in July. I'm super excited to finally get the blog post out, and even better, alongside a post describing the root cause process that went into the project. You can read about Tailscale's story here: https://tailscale.com/blog/sqlite-wal-reset-bug I'm only sad that I didn't put SQLite under test earlier in the year, or…

How difficult would it have been to isolate that problem if you didn't already know the SQLite subsystem it was in? This feels, to someone relatively ignorant of the SQLite / Tailscale / Antithesis architectures as a "hindsight is 20/20" kind of thing, but I'm open to learning more.

Great question! The general approach we take with transactional systems like this is to put reachability statements throughout the complex stateful machinery and then stress-test them in Antithesis.

The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!).

Said differently, this is exactly the approach we take with all stateful transactional systems. I am only sad that I didn't do this experiment months earlier, as it would have saved Tailscale and the SQLite team a lot of time.

[1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...

Re: Breaking the WAL

#22

Post by biorach »

gamegoblin wrote: post #14
minimaltom wrote: post #12

I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite. I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test. The closest I mention of the prompt I could find was: > Then I asked it to write a simple workload which exercised the WAL insert and ch…

Their founder guy says[1] they found some other bugs while doing this that they will report soon, so if that is true, seems more plausibly like a pretty generic thing. Looking forward to seeing the other bugs they found. 1. https://news.ycombinator.com/item?id=49278351

looks like the bug fixes are in individual commits here: https://github.com/sqlite/sqlite/compare/master...antithesis...

I'd be interested in seeing what the SQLite people have to say about them if/when they are submitted upstream

Re: Breaking the WAL

#23

Post by carlsverre »

minimaltom wrote: post #20
carlsverre wrote: post #18

Earlier quoted context omitted.

Hey, author here! Good question. The prompt was to use the Antithesis skills[1] to write a workload that exercises SQLite's insertion and checkpoint machinery across multiple processes using a specific version of SQLite (3.51.2). The agent who did the work was aware of the bug and knew I wanted to see if Antithesis could reproduce it. However, I specifically designed the workload the same way we approach testing othe…

Thanks for clarifying! It would be really interesting if Antithesis finds the bug when: 1. The specific bug isnt mentioned 2. (If youre game) a model with a knowledge-cutoff date before the report is used

This is a solid idea. However, something that is maybe similar to this idea is this: https://antithesis.com/blog/2026/finding-bugs-in-raft-implem...

Which is: "Given the knowledge of a kind of bug that can happen in one Raft implementation, can a similar workload find similar bugs in other Raft implementations?" Turns out it can.

Re: Breaking the WAL

#24

Post by carlsverre »

biorach wrote: post #22
gamegoblin wrote: post #14

Earlier quoted context omitted.

Their founder guy says[1] they found some other bugs while doing this that they will report soon, so if that is true, seems more plausibly like a pretty generic thing. Looking forward to seeing the other bugs they found. 1. https://news.ycombinator.com/item?id=49278351

looks like the bug fixes are in individual commits here: https://github.com/sqlite/sqlite/compare/master...antithesis... I'd be interested in seeing what the SQLite people have to say about them if/when they are submitted upstream

Those are the commits from upstream that resolved the bug. Notice that most of them were committed by Dr. Richard Hipp himself.

Re: Breaking the WAL

#25

Post by carlsverre »

returningfory2 wrote: post #19
carlsverre wrote: post #6

Earlier quoted context omitted.

Carl (author) here. This is a poor reading of the blog post. Notably, I built a general-purpose workload that simply exercises the write/checkpoint machinery. The exact same workload we use to break all kinds of transactional systems. This generic workload found the bug without any special knowledge of the bug, due to how Antithesis works.

I think the issue is that you knew there was a bug to be found in the write and checkpoint interactions, which then determined which workload you built. My understanding is that this workload is very uncommon: the Tailscale blog says they used a custom unusual configuration to have many checkpoints like this. So without knowing the bug, it seems unlikely one would build this workload and then find the bug. If that ma…

I can see how the post comes across that way. I may need to edit it to somehow be clearer about how I approached the workload development. I'll refer you to my other replies to comments for more information on my approach:

https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278521

Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.

Re: Breaking the WAL

#26

Post by Mawr »

carlsverre wrote: post #6
grebc wrote: post #4

[flagged]

Carl (author) here. This is a poor reading of the blog post. Notably, I built a general-purpose workload that simply exercises the write/checkpoint machinery. The exact same workload we use to break all kinds of transactional systems. This generic workload found the bug without any special knowledge of the bug, due to how Antithesis works.

Okay, so use the same methodology to find a new, unknown bug? That'd be actually impressive.

As-is, this is just 20/20 hindsight with concerns about leading the AI on through the prompt hand-waved away. Come on.

Re: Breaking the WAL

#27

Post by uhohherewegoaga »

as a long time lurker who usually enjoys antithesis posts, this was a pretty existentially sad read. we all know that company blog posts here always tend to be ads at their core, but historically afaicr antithesis generally toes the line well of keeping the "thing we sell is really good" as a side dish to the "i (author) did very interesting deep thinking and/or hard work to accomplish something cool" entree.

this one falls very flat on that front imo. the "antithesis finds rare bugs" is the overwhelming majority of the actual content here, where as the usual meat of the blog, the hard work, the deep thinking, really distills down to "typed this into claude mobile, typed that into claude mobile". yes, its a statement to the power of the technologies of today that's all it takes to find this bug, but man it's a sad signal for those of us looking for compelling technical reads instead of b2b sales pitches.

Re: Breaking the WAL

#28

Post by Mawr »

carlsverre wrote: post #21
MPSimmons wrote: post #15

Earlier quoted context omitted.

How difficult would it have been to isolate that problem if you didn't already know the SQLite subsystem it was in? This feels, to someone relatively ignorant of the SQLite / Tailscale / Antithesis architectures as a "hindsight is 20/20" kind of thing, but I'm open to learning more.

Great question! The general approach we take with transactional systems like this is to put reachability statements throughout the complex stateful machinery and then stress-test them in Antithesis. The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This e…

> This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!).

Suuure. So why exactly haven't you found this bug already a long time ago? I mean it is trivially obvious that this is where bugs in SQLite would be, right?

SQLite is open source and is one of the most popular databases, surely testing it to find bugs would also be obvious?

Re: Breaking the WAL

#29

Post by biorach »

carlsverre wrote: post #24
biorach wrote: post #22

Earlier quoted context omitted.

looks like the bug fixes are in individual commits here: https://github.com/sqlite/sqlite/compare/master...antithesis... I'd be interested in seeing what the SQLite people have to say about them if/when they are submitted upstream

Those are the commits from upstream that resolved the bug. Notice that most of them were committed by Dr. Richard Hipp himself.

oh! damn! I did not notice that. That does answer my question.

Re: Breaking the WAL

#30

Post by minimaltom »

carlsverre wrote: post #23
minimaltom wrote: post #20

Earlier quoted context omitted.

Thanks for clarifying! It would be really interesting if Antithesis finds the bug when: 1. The specific bug isnt mentioned 2. (If youre game) a model with a knowledge-cutoff date before the report is used

This is a solid idea. However, something that is maybe similar to this idea is this: https://antithesis.com/blog/2026/finding-bugs-in-raft-implem... Which is: "Given the knowledge of a kind of bug that can happen in one Raft implementation, can a similar workload find similar bugs in other Raft implementations?" Turns out it can.

Yeah 100%! And I'm sorry if I sound a little more critical and less eager, its just thats theres a world of difference between a priori finding the bug, and reproducing it, and the impression of the article (from my read) was the former.

But please keep writing, I know its super hard to put yourself out there and make content!

Post reply on HN