[deleted]
Breaking the WAL
11–20 of 59 posts
Re: Breaking the WAL
#12I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite.
I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test.
The closest I mention of the prompt I could find was:
> Then I asked it to write a simple workload which exercised the WAL insert and checkpoint code. Notably, this is a completely generic workload.
With a skeptical lens, unclear.
Re: Breaking the WAL
#13[deleted]
Re: Breaking the WAL
#14I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite. I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test. The closest I mention of the prompt I could find was: > Then I asked it to write a simple workload which exercised the WAL insert and ch…
Their founder guy says[1] they found some other bugs while doing this that they will report soon, so if that is true, seems more plausibly like a pretty generic thing.
Looking forward to seeing the other bugs they found.
Re: Breaking the WAL
#15Hi! I'm the author and the person who was on a road trip when I decided to do this experiment back in July. I'm super excited to finally get the blog post out, and even better, alongside a post describing the root cause process that went into the project. You can read about Tailscale's story here: https://tailscale.com/blog/sqlite-wal-reset-bug I'm only sad that I didn't put SQLite under test earlier in the year, or…
How difficult would it have been to isolate that problem if you didn't already know the SQLite subsystem it was in? This feels, to someone relatively ignorant of the SQLite / Tailscale / Antithesis architectures as a "hindsight is 20/20" kind of thing, but I'm open to learning more.
Re: Breaking the WAL
#16Earlier quoted context omitted.
I read your ad. It states you replicated the bug once the SQLite team fixed it, and published it. Not sure what’s difficult about replicating behaviour when it’s spelled out for you.
this is a deliberate mischaracterisation of the content of the article
You’re mischaracterising an ad, that’s also funnily enough full of it.
Re: Breaking the WAL
#17I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite. I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test. The closest I mention of the prompt I could find was: > Then I asked it to write a simple workload which exercised the WAL insert and ch…
[deleted]
Re: Breaking the WAL
#18I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite. I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test. The closest I mention of the prompt I could find was: > Then I asked it to write a simple workload which exercised the WAL insert and ch…
Hey, author here! Good question. The prompt was to use the Antithesis skills[1] to write a workload that exercises SQLite's insertion and checkpoint machinery across multiple processes using a specific version of SQLite (3.51.2). The agent who did the work was aware of the bug and knew I wanted to see if Antithesis could reproduce it. However, I specifically designed the workload the same way we approach testing other WAL checkpoint-style systems: by putting them under stress. The workload itself does not know about the bug.
My goal, which I believe I accomplished, was to determine if Antithesis would have found this bug if we had put the SQLite WAL under test.
Re: Breaking the WAL
#19[flagged]
Carl (author) here. This is a poor reading of the blog post. Notably, I built a general-purpose workload that simply exercises the write/checkpoint machinery. The exact same workload we use to break all kinds of transactional systems. This generic workload found the bug without any special knowledge of the bug, due to how Antithesis works.
I think the issue is that you knew there was a bug to be found in the write and checkpoint interactions, which then determined which workload you built.
My understanding is that this workload is very uncommon: the Tailscale blog says they used a custom unusual configuration to have many checkpoints like this. So without knowing the bug, it seems unlikely one would build this workload and then find the bug. If that makes sense.
Edit: just want to say that you being able to repro it is awesome, but that the overall claim seems a little overstated to me.
Re: Breaking the WAL
#20I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite. I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test. The closest I mention of the prompt I could find was: > Then I asked it to write a simple workload which exercised the WAL insert and ch…
Hey, author here! Good question. The prompt was to use the Antithesis skills[1] to write a workload that exercises SQLite's insertion and checkpoint machinery across multiple processes using a specific version of SQLite (3.51.2). The agent who did the work was aware of the bug and knew I wanted to see if Antithesis could reproduce it. However, I specifically designed the workload the same way we approach testing othe…
Thanks for clarifying! It would be really interesting if Antithesis finds the bug when:
1. The specific bug isnt mentioned 2. (If youre game) a model with a knowledge-cutoff date before the report is used