Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: TigerBeetle 0.16.11

jepsen.io

Post reply on HN
Print view View original

61–70 of 88 posts

Re: Jepsen: TigerBeetle 0.16.11

#61

Post by jorangreef »

dralley wrote: post #33
jorangreef wrote: post #7

Earlier quoted context omitted.

Joran, creator and CEO from TigerBeetle here! At a national level, we’re working with the Gates Foundation to integrate TigerBeetle into their non-profit central bank switch that will be powering Rwanda’s National Digital Payments System 2.0 later this year [1]. At an enterprise level, TigerBeetle already powers customers processing 100M+ transactions per month in production, and we recently signed our first $2B fint…

Have you had a difficult time convincing customers to use a product written in a pre-1.0 programming language?

Zig's pre-1.0 status also refers more to API stability. The language and tooling already has more quality, at least in my own experience, than if we had picked C, which was the only other choice available to us when we made the decision to invest in Zig's trajectory back in 2020, given we needed to do static allocation and that any sort of global allocator was out of the question.

But, no. On the commercial side, I don't think we've had one conversation with a prospect or CTO or engineering team where they were concerned that we picked a systems language for the next thirty years. And while Zig is a beautiful, perfect replacement for C, I think the real reason the question has never come up, is that our customers come to us instead of us to them. We're not trying to convince anyone. They're already appreciating the extensive end-to-end testing we do on everything we ship.

However, I should emphasize again, that given all the assertions, fuzzing and DST we do, Zig's quality can't be overstated. It holds up.

Re: Jepsen: TigerBeetle 0.16.11

#62

Post by matklad »

eevmanu wrote: post #47

I have a question that I hope is not misinterpreted, as I'm asking purely out of a desire to learn. I am new to distributed systems and fascinated by deterministic simulation testing. After reading the Jepsen report on TigerBeetle, the related blog post, and briefly reviewing the Antithesis integration code on GitHub workflow, I'm trying to better understand the testing scope. My core question is: could these bugs de…

To add to what aphyr says, you generally need three components for generative testing of distributed systems:

1. Some sort of environment, which can run the system. The simplest environment is to spin up a real cluster of machines, but ideally you want something fancier, to improve performance, control over responses of external APIs, determinism, reproducibility, etc. 2. Some sort of load generator, which makes the system in the environment do interesting thing 3. Some sort of auditor, which observes the behavior of the system under load and decides whether the system behaves according to the specification.

Antithesis mostly tackles problem #1, providing a deterministic simulation environment as a virtual machine. The same problem is talked by jepsen (by using real machines, but injecting faults at the OS level), and by TigerBeetle's own VOPR (which is co-designed with the database, and for that reason can run the whole cluster on just a single thread). There there approaches are complimentary and are good at different things.

For this bug, the critical part was #2, #3 --- writing workload verifier and auditor that actually can trigger the bug. Here, it was aphyr's 1600 lines of TigerBeetle-specfic Clojure code that triggred and detected the bug (and then we patched _our_ equivalent to also trigger it. Really, what's buggy here is not the database, but the VOPR. Database having bugs is par of course, you can't just avoid bugs through the sheer force of will. So you need testing strategy that can trigger most bugs, and any bug that slips through is pointing to the deficiency in the workload generator.)

Re: Jepsen: TigerBeetle 0.16.11

#63

Post by jorangreef »

boris wrote: post #48
jorangreef wrote: post #27

Earlier quoted context omitted.

The DBMS is Apache 2.0 and our customers pay us (well) for everything else to run, integrate, migrate, operate and support that. For more on our open source thinking and how this is orthogonal to business model (and product!), see our interview with the Changelog: https://m.youtube.com/watch?v=Yr8Y2EYnxJs

I watched that but I don't see it as convincing. Let's take the AWS example brought up in the talk. The "compete on the interface, not (open source) implementation" idea I think misses (at least) the following points: 1. AWS will take your initial and ongoing investment in the implementation but they don't have to share theirs with you. Specifically, they will take your improvements but their own improvements (say so…

To be clear, we have no problem if all the hyperscalers decide to offer TigerBeetle as their flagship OLTP database. That builds trust and is a good thing for the ecosystem as a whole.

We also don't expect (or need) anyone to contribute improvements upstream to us. That's open source!

Finally, open source is not the same thing as product. There are thousands of companies around the world who make high quality products that people pay for. TigerBeetle is no different.

Re: Jepsen: TigerBeetle 0.16.11

#64

Post by jorangreef »

eevmanu wrote: post #47

I have a question that I hope is not misinterpreted, as I'm asking purely out of a desire to learn. I am new to distributed systems and fascinated by deterministic simulation testing. After reading the Jepsen report on TigerBeetle, the related blog post, and briefly reviewing the Antithesis integration code on GitHub workflow, I'm trying to better understand the testing scope. My core question is: could these bugs de…

(Note also that 90% of our deterministic simulation testing is done primarily by the VOPR, TigerBeetle's own deterministic simulator, which we built inhouse, and which runs on a fleet of 1,000 dedicated CPU cores 24/7. We also use Antithesis, but as a second layer of DST.)

To understand why the query engine bug slipped through, see: https://tigerbeetle.com/blog/2025-06-06-fuzzer-blind-spots-m...

Re: Jepsen: TigerBeetle 0.16.11

#65

Post by aphyr »

matklad wrote: post #62
eevmanu wrote: post #47

I have a question that I hope is not misinterpreted, as I'm asking purely out of a desire to learn. I am new to distributed systems and fascinated by deterministic simulation testing. After reading the Jepsen report on TigerBeetle, the related blog post, and briefly reviewing the Antithesis integration code on GitHub workflow, I'm trying to better understand the testing scope. My core question is: could these bugs de…

To add to what aphyr says, you generally need three components for generative testing of distributed systems: 1. Some sort of environment, which can run the system. The simplest environment is to spin up a real cluster of machines, but ideally you want something fancier, to improve performance, control over responses of external APIs, determinism, reproducibility, etc. 2. Some sort of load generator, which makes the…

And honestly--designing a generator for a system like this is hard. Really hard. I struggled for weeks to get something that didn't just fail 99% of requests trivially, and it's an (ahem) giant pile of probabilistic hacks. So I wouldn't be too hard on the various TB test generators here!

https://github.com/jepsen-io/tigerbeetle/blob/main/src/jepse...

Re: Jepsen: TigerBeetle 0.16.11

#66

Post by anarazel »

jorangreef wrote: post #16
nindalf wrote: post #3

Very impressed with this report. Whenever I read TigerBeetle's claims on reliability and scalability, I'd think "ok, let's wait for the Jepsen report". This report found a number of issues, which might be a cause for concern. But I think it's a positive because they didn't just fix the issues, they've expanded their internal test suite to catch similar bugs in future. With such an approach to engineering I feel like…

Thanks! Yes, we have around 6,000+ assertions in TigerBeetle. A few of these were overtight, hence some of the crashes. But those were the assertions doing their job, alerting us that we needed to adjust our mental model, which we did. Otherwise, apart from a small correctness bug in an internal testing feature we added (only in our Java client and only for Jepsen to facilitate the audit) there was only one correctne…

> There are known scenarios in the literature that will cause Postgres to lose data, which TigerBeetle can detect and recover from.

What are you referencing here?

Re: Jepsen: TigerBeetle 0.16.11

#67

Post by jorangreef »

anarazel wrote: post #66
jorangreef wrote: post #16

Earlier quoted context omitted.

Thanks! Yes, we have around 6,000+ assertions in TigerBeetle. A few of these were overtight, hence some of the crashes. But those were the assertions doing their job, alerting us that we needed to adjust our mental model, which we did. Otherwise, apart from a small correctness bug in an internal testing feature we added (only in our Java client and only for Jepsen to facilitate the audit) there was only one correctne…

> There are known scenarios in the literature that will cause Postgres to lose data, which TigerBeetle can detect and recover from. What are you referencing here?

The scenarios described in our QCon London talk linked above.

This surveys the excellent storage fault research from UW-Madison, and in particular:

  “Can Applications Recover from fsync Failures?”

  “Protocol-Aware Recovery for Consensus-Based Storage”
Finally, I'd recommend watching “Consensus and the Art of Durability”, our talk from SD24 in NYC last year:

https://www.youtube.com/watch?v=tRgvaqpQPwE

Re: Jepsen: TigerBeetle 0.16.11

#68

Post by kristoff_it »

boris wrote: post #58
kristoff_it wrote: post #56

Earlier quoted context omitted.

> AWS will take your initial and ongoing investment in the implementation but they don't have to share theirs with you. Specifically, they will take your improvements but their own improvements (say some performance optimizations) they can keep to themselves. It's good business sense if it allows them to further differentiate their "improved" offering from your "vanilla" service. In practice all I've seen from AWS is…

> In other words, hypothetical improvements by AWS (and other Clouds) are extremely overrated. Interesting, in a recent thread (I think it was about Redis going back open source) an AWS employer was bragging about substantial concurrency optimizations they implemented in Valkey. At the time I thought it could have been a great differentiator to keep proprietary but perhaps they decide to sacrifice it to help make sur…

That's a special case for sure, given the new fight for supremacy between the two forks, that said you can see in all those threads antirez bickering with the AWS people over exactly who introduced what.

Re: Jepsen: TigerBeetle 0.16.11

#69

Post by jorangreef »

FlyingSnake wrote: post #45

Love the wonderfully detailed report. Getting it tested and signed off by Jepsen is such a huge endorsement for TigerBeetle. It’s not even reached v1.0 and I can’t wait to see it hit new milestone in the future. Special kudos to the founders who are sharing great insights in this thread.

Yes, Kyle did an incredible job and I also love the detail he put into the report. I kept saying to myself: “this is like a work of art”, the craftsmanship and precision.

Appreciate your kind words too, and look forward also to sharing something new in our talks at SD25 in Amsterdam soon!

Re: Jepsen: TigerBeetle 0.16.11

#70

Post by kiitos »

jorangreef wrote: post #67
anarazel wrote: post #66

Earlier quoted context omitted.

> There are known scenarios in the literature that will cause Postgres to lose data, which TigerBeetle can detect and recover from. What are you referencing here?

The scenarios described in our QCon London talk linked above. This surveys the excellent storage fault research from UW-Madison, and in particular: “Can Applications Recover from fsync Failures?” “Protocol-Aware Recovery for Consensus-Based Storage” Finally, I'd recommend watching “Consensus and the Art of Durability”, our talk from SD24 in NYC last year: https://www.youtube.com/watch?v=tRgvaqpQPwE

    [disks are] somewhere between non-byzentine fault tolerance and
    Byzantine fault tolerance ... you expect the disk to be almost 
    an active adversary ...
    ...
    so you start to see just a single disk as a distributed system
My goodness, not at all! If you can't trust the interface to a local disk then you're lost just at a fundamental level. And even ignoring that, a disk is an implementation detail of a node in a distributed system, whatever properties that disk may have to that local node are irrelevant in the context of the broader system, and are the responsibility of the local node to manage before communicating anything with other nodes in that broader system.

Combined with https://www.youtube.com/watch?v=tRgvaqpQPwE it seems like the author/presenter is conflating local/disk-related properties/details with distributed/system-based requirements/guarantees. If consensus requires a node to have durably persisted some bit of state before it sends a particular message to other nodes in the distributed system, then it doesn't matter how that persistence is implemented, it only matters how that persistence is observable, disks and FS caches and etc. aren't requirements, they're just one of many possible implementation choices.

Post reply on HN