Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Breaking the WAL

antithesis.com

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Breaking the WAL

#31

Post by wwilson »

Mawr wrote: post #28
carlsverre wrote: post #21

Earlier quoted context omitted.

Great question! The general approach we take with transactional systems like this is to put reachability statements throughout the complex stateful machinery and then stress-test them in Antithesis. The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This e…

> This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!). Suuure. So why exactly haven't you found this bug already a long time ago? I mean it is trivially obvious that this is where bugs in SQLite would be, right? SQLite is open source and is one of the most popular databases, surely testing it to find bugs would also be obvious?

Brutally honest answer: it only recently became easy to get random systems under test with Antithesis without a bunch of manual/expert effort. Expect us to be reporting a lot of bugs against open source projects in the near future.

Re: Breaking the WAL

#32

Post by carlsverre »

Mawr wrote: post #28
carlsverre wrote: post #21

Earlier quoted context omitted.

Great question! The general approach we take with transactional systems like this is to put reachability statements throughout the complex stateful machinery and then stress-test them in Antithesis. The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This e…

> This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!). Suuure. So why exactly haven't you found this bug already a long time ago? I mean it is trivially obvious that this is where bugs in SQLite would be, right? SQLite is open source and is one of the most popular databases, surely testing it to find bugs would also be obvious?

We put many things to the test, but it costs money, and until very recently we haven't had a way to do this without a lot of human oversight. Now that we have https://github.com/antithesishq/antithesis-skills and LLMs are getting pretty reliably good at writing workloads, we are looking at ways to open up our technology to the OSS community. We have already started down that path with Etcd last year (https://etcd.io/blog/2025/autonomus_testing_with_antithesis/) and are working with various OSS groups to put more systems under test in a responsible way.

I'm not gonna lie, though, you're hitting a sore point. I'm absolutely kicking myself for not putting SQLite under test when I joined. The first few projects I put under test after joining were all projects that had a few properties: 1. The project had a path to a commercial deal 2. The project likely had interesting bugs to find

TBH, I didn't expect that one of the most widely used and well-tested pieces of software in existence would have a bug our fuzzer could find in 15 mins. (To be clear, this is not a hit on SQLite, just that our fuzzer is stupid good at finding race conditions. SQLite remains one of my favorite pieces of software ever.)

With this in mind, are there other similar projects that I'm not thinking of that I should put under test? Before you ask, I'm already scaling up my SQLite harness to cover more of its state space and run against master. Maybe I'll find something novel in SQLite after all! Never too late to check.

Re: Breaking the WAL

#33

Post by carlsverre »

minimaltom wrote: post #30
carlsverre wrote: post #23

Earlier quoted context omitted.

This is a solid idea. However, something that is maybe similar to this idea is this: https://antithesis.com/blog/2026/finding-bugs-in-raft-implem... Which is: "Given the knowledge of a kind of bug that can happen in one Raft implementation, can a similar workload find similar bugs in other Raft implementations?" Turns out it can.

Yeah 100%! And I'm sorry if I sound a little more critical and less eager, its just thats theres a world of difference between a priori finding the bug, and reproducing it, and the impression of the article (from my read) was the former. But please keep writing, I know its super hard to put yourself out there and make content!

Thank you! I appreciate the note of support! I'd love to write more blog posts in the future.

Re: Breaking the WAL

#34

Post by skybrian »

carlsverre wrote: post #9

Hi! I'm the author and the person who was on a road trip when I decided to do this experiment back in July. I'm super excited to finally get the blog post out, and even better, alongside a post describing the root cause process that went into the project. You can read about Tailscale's story here: https://tailscale.com/blog/sqlite-wal-reset-bug I'm only sad that I didn't put SQLite under test earlier in the year, or…

Nice article, but could you get someone to fix the blog layout? Having the author card pinned to the window rather than being able to scroll past it is distracting.

Re: Breaking the WAL

#35

Post by skybrian »

carlsverre wrote: post #32
Mawr wrote: post #28

Earlier quoted context omitted.

> This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!). Suuure. So why exactly haven't you found this bug already a long time ago? I mean it is trivially obvious that this is where bugs in SQLite would be, right? SQLite is open source and is one of the most popular databases, surely testing it to find bugs would also be obvious?

We put many things to the test, but it costs money, and until very recently we haven't had a way to do this without a lot of human oversight. Now that we have https://github.com/antithesishq/antithesis-skills and LLMs are getting pretty reliably good at writing workloads, we are looking at ways to open up our technology to the OSS community. We have already started down that path with Etcd last year ( https://etcd.io…

Other embedded databases like Dolt? DuckDB? git?

Re: Breaking the WAL

#36

Post by carlsverre »

skybrian wrote: post #35
carlsverre wrote: post #32

Earlier quoted context omitted.

We put many things to the test, but it costs money, and until very recently we haven't had a way to do this without a lot of human oversight. Now that we have https://github.com/antithesishq/antithesis-skills and LLMs are getting pretty reliably good at writing workloads, we are looking at ways to open up our technology to the OSS community. We have already started down that path with Etcd last year ( https://etcd.io…

Other embedded databases like Dolt? DuckDB? git?

We've started testing git and DuckDB! Haven't played much with Dolt, though. We have some initial findings on Git that we are validating and will hopefully be making public soon!

Re: Breaking the WAL

#37

Post by karagenit »

carlsverre wrote: post #25
returningfory2 wrote: post #19

Earlier quoted context omitted.

I think the issue is that you knew there was a bug to be found in the write and checkpoint interactions, which then determined which workload you built. My understanding is that this workload is very uncommon: the Tailscale blog says they used a custom unusual configuration to have many checkpoints like this. So without knowing the bug, it seems unlikely one would build this workload and then find the bug. If that ma…

I can see how the post comes across that way. I may need to edit it to somehow be clearer about how I approached the workload development. I'll refer you to my other replies to comments for more information on my approach: https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278521 Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.

I’m sorry you’re feeling ganged up on, I thought it was a nice article and cool you were able to repro the bug with your tools.

Like some others have mentioned, one of my earliest thoughts was “how much of a hint was the LLM given about the bug?” I think if the prompt used was stated clearly/verbatim near the beginning of the article, that would probably dispel a good amount of the criticism.

Re: Breaking the WAL

#39

Post by awwaiid »

carlsverre wrote: post #32
Mawr wrote: post #28

Earlier quoted context omitted.

> This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!). Suuure. So why exactly haven't you found this bug already a long time ago? I mean it is trivially obvious that this is where bugs in SQLite would be, right? SQLite is open source and is one of the most popular databases, surely testing it to find bugs would also be obvious?

We put many things to the test, but it costs money, and until very recently we haven't had a way to do this without a lot of human oversight. Now that we have https://github.com/antithesishq/antithesis-skills and LLMs are getting pretty reliably good at writing workloads, we are looking at ways to open up our technology to the OSS community. We have already started down that path with Etcd last year ( https://etcd.io…

Matrix, specifically synapse/rustMatrixSDK :)
Post reply on HN