Tracking down the 16-year-old WAL-reset SQLite bug
tailscale.com
Tracking down the 16-year-old WAL-reset SQLite bug
1–10 of 263 posts
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#2> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future.
Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#3> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.
Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#4> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.
Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.
Their CEO is a very nice and personable guy too. Has given me and others advice on random topics of his interest with no nonsense plenty of times.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#5Which SQLite driver for Go does Tailscale use?
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#6> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.
Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.
Yeah, this part also stuck out to me:
> Because this wouldn’t be a quick or easy fix, we reached out to the SQLite developers for a professional support contract. This was a great decision. It gave us direct access to their deep expertise and experience, and we had many detailed technical conversations about our architecture and our incidents.
They were willing to pay to get help solving the problem, and then pay again to make sure that the problem is easier to avoid in the future! That kind of long-term thinking seems pretty rare nowadays...
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#7Great read. So glad they took the time to tell this story. (And glad they, as a for profit corporation, took out a support contract with SQLite. I hope they continue to do so even though this problem is resolved.)
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#8Well written post, really enjoyed reading it.
> A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used.
This line led me to believe that the writer and checkpointing logic lived on the same database connection, so I was curious to find out how the data race occurred. However, the bug details on the SQLite page[0] outline that it can only ever occur if there are multiple connections open, so the writer and the checkpointer must have been on different threads.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#9While technically true as written, it seems to downplay the significance:
> The bug is a data race with tight timing constraints. It is unlikely to occur in common use.
A large customer did experience this corruption, so it's important for people with tailscale's setup update immediately.
> The developers have never been able to reproduce the bug organically and had to add special testing logic to SQLite that deliberately triggers the circumstances of the the bug in order to verify that the issue has been fixed.
Re: Tracking down the 16-year-old WAL-reset SQLite bug
#10> Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window.
Gotta love single points of failure...