Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Breaking the WAL

antithesis.com

Post reply on HN
Print view View original

41–50 of 59 posts

Re: Breaking the WAL

#41

Post by wps »

carlsverre wrote: post #32
Mawr wrote: post #28

Earlier quoted context omitted.

> This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!). Suuure. So why exactly haven't you found this bug already a long time ago? I mean it is trivially obvious that this is where bugs in SQLite would be, right? SQLite is open source and is one of the most popular databases, surely testing it to find bugs would also be obvious?

We put many things to the test, but it costs money, and until very recently we haven't had a way to do this without a lot of human oversight. Now that we have https://github.com/antithesishq/antithesis-skills and LLMs are getting pretty reliably good at writing workloads, we are looking at ways to open up our technology to the OSS community. We have already started down that path with Etcd last year ( https://etcd.io…

Test out Fossil SCM, it depends on SQLite and is also written in c89.

Re: Breaking the WAL

#42

Post by [deleted] »

wwilson wrote: post #3

We wanted to publish this ASAP since people are talking about this bug today. In a follow-up, we will show how our automated causality analysis could have saved Tailscale and SQLite from 6 months of root cause analysis. Stay tuned!

[deleted]

Re: Breaking the WAL

#44

Post by NortySpock »

wwilson wrote: post #31
Mawr wrote: post #28

Earlier quoted context omitted.

> This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!). Suuure. So why exactly haven't you found this bug already a long time ago? I mean it is trivially obvious that this is where bugs in SQLite would be, right? SQLite is open source and is one of the most popular databases, surely testing it to find bugs would also be obvious?

Brutally honest answer: it only recently became easy to get random systems under test with Antithesis without a bunch of manual/expert effort. Expect us to be reporting a lot of bugs against open source projects in the near future.

I've been jealous of the work antithesis is doing and support you in spirit; the world needs deterministic, tested software.

If you want a challenging open source deterministic game to test, may I suggest Beyond All Reason, powered by the Spring / Recoil engine? It's an RTS game that's been in various incarnations of development for 20+ years. (Disclosure: I'm an occasional code contributor to the project).

Re: Breaking the WAL

#45

Post by mook »

biorach wrote: post #22
gamegoblin wrote: post #14

Earlier quoted context omitted.

Their founder guy says[1] they found some other bugs while doing this that they will report soon, so if that is true, seems more plausibly like a pretty generic thing. Looking forward to seeing the other bugs they found. 1. https://news.ycombinator.com/item?id=49278351

looks like the bug fixes are in individual commits here: https://github.com/sqlite/sqlite/compare/master...antithesis... I'd be interested in seeing what the SQLite people have to say about them if/when they are submitted upstream

Hmm, shouldn't the diff start at the tag? That is, it seems like it should be https://github.com/sqlite/sqlite/compare/version-3.51.2...an...

Which says they made two commits to make instrumentation work. Also, no fixes (which seems reasonable to me; need to figure out what's wrong first)

Presumably any bugs found are being sent upstream to fix.

Re: Breaking the WAL

#46

Post by carlsverre »

NortySpock wrote: post #44
wwilson wrote: post #31

Earlier quoted context omitted.

Brutally honest answer: it only recently became easy to get random systems under test with Antithesis without a bunch of manual/expert effort. Expect us to be reporting a lot of bugs against open source projects in the near future.

I've been jealous of the work antithesis is doing and support you in spirit; the world needs deterministic, tested software. If you want a challenging open source deterministic game to test, may I suggest Beyond All Reason, powered by the Spring / Recoil engine? It's an RTS game that's been in various incarnations of development for 20+ years. (Disclosure: I'm an occasional code contributor to the project).

Oh man, Beyond All Reason would be a killer demo. We are actually testing another RTS as we speak, but I won't spoil it for you :) Look forward to a really epic post coming soon.

Re: Breaking the WAL

#47

Post by nemothekid »

I understand the author feels he missed an opportunity to catch a bug in Sqlite - but this still feels like catching something in hindsight. Just knowing the bug the is in Sqlite's concurrency engine is a huge win.

But reading over tailscale's blogpost, I'm not sure how I would connect the dots to "this issue I'm seeing in production" to "load this up in antithesis". It seemed half the battle was figuring out if the bug was in Sqlite, Tailscale, or Linux, and in that situation if you gave Antithesis all the context, would it reliably find the bug?

Re: Breaking the WAL

#48

Post by jauntywundrkind »

carlsverre wrote: post #9

Hi! I'm the author and the person who was on a road trip when I decided to do this experiment back in July. I'm super excited to finally get the blog post out, and even better, alongside a post describing the root cause process that went into the project. You can read about Tailscale's story here: https://tailscale.com/blog/sqlite-wal-reset-bug I'm only sad that I didn't put SQLite under test earlier in the year, or…

you asserted that Antithesis would have found it right away, but it seems like from your write-up that you first set up instrumentation on the source, then told it to reproduce the WAL bug.

i don't feel like i gained a strong appreciation for what the actual Antithesis capabilities are or why you had confidence this was anything the Antithesis tools would have helped find? i want to know more about what kind of analysis or capabilities Antithesis has and how it could have helped find the issue. what's clear is that Antithesis could confirm an already found issue. and even there I don't feel like I gained any connection or knowledge or understanding of what the tooling was doing or how it would help. i'd really like more technical meat to this post to understand the value proposition.

Re: Breaking the WAL

#49

Post by moomin »

I feel like we’re in a weird P=NP age, where after someone finds an issue, someone points out a cheaper way to find the same issue, which inevitably involves knowing the issue is there in the first place and it’s approximate location.

Re: Breaking the WAL

#50

Post by quadrifoliate »

carlsverre wrote: post #25
returningfory2 wrote: post #19

Earlier quoted context omitted.

I think the issue is that you knew there was a bug to be found in the write and checkpoint interactions, which then determined which workload you built. My understanding is that this workload is very uncommon: the Tailscale blog says they used a custom unusual configuration to have many checkpoints like this. So without knowing the bug, it seems unlikely one would build this workload and then find the bug. If that ma…

I can see how the post comes across that way. I may need to edit it to somehow be clearer about how I approached the workload development. I'll refer you to my other replies to comments for more information on my approach: https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278521 Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.

I do feel like you're being ganged up on as well. IMO this is a very cool thing that Antithesis is able to find and reproduce this bug in this specific subsystem with a relatively generic workload.

I guess what would have been an even more cool thing is "we ran some more general testing with Antithesis, and it found five other bugs". Have you thought along those lines or explored something like that? There have to be other, similar bugs lurking in SQLite :)

Post reply on HN