Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: TigerBeetle 0.16.11

jepsen.io

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Jepsen: TigerBeetle 0.16.11

#51

Post by matklad »

matklad wrote: post #31
jrpelkonen wrote: post #28

Earlier quoted context omitted.

Hi Joran, I have followed TigerBeetle with interest for a while, and thank you for your inspirational work and informative presentations. However, you have stated in several occasions that the lack of memory safety in Zig is not a concern since you don't dynamically allocate memory post startup. However, one of the defects uncovered here (#2435) was caused by dereferencing an uninitialized pointer. I find this pretty…

Note that that's a bug in the client, in the Zig-java FFI code, which is inherently unsafe. We'd likely made an a similar bug in Rust. Which is, yeah, one of the bigger technical challenges for us --- we ship language-native libraries for Go,node,Java,C#,Python and Rust, and, like in the Tolstoi novel, each one is peculiar in its own way. What's worse, they aren't directly covered by our deterministic simulator. That…

Oh, important clarification from andrewrk(https://lobste.rs/c/tf6jng), which I totally missed myself: this isn't actually a dereference of uninitialized pointer, it's a defer of a pointer which is explicitly set to a specific, invalid value.

Re: Jepsen: TigerBeetle 0.16.11

#52

Post by AndyKelley »

jrpelkonen wrote: post #28
jorangreef wrote: post #16

Earlier quoted context omitted.

Thanks! Yes, we have around 6,000+ assertions in TigerBeetle. A few of these were overtight, hence some of the crashes. But those were the assertions doing their job, alerting us that we needed to adjust our mental model, which we did. Otherwise, apart from a small correctness bug in an internal testing feature we added (only in our Java client and only for Jepsen to facilitate the audit) there was only one correctne…

Hi Joran, I have followed TigerBeetle with interest for a while, and thank you for your inspirational work and informative presentations. However, you have stated in several occasions that the lack of memory safety in Zig is not a concern since you don't dynamically allocate memory post startup. However, one of the defects uncovered here (#2435) was caused by dereferencing an uninitialized pointer. I find this pretty…

TigerBeetle uses ReleaseSafe optimization mode, which means that the pointer was in fact initialized to 0xaaaaaaaaaaaaaaaa. Since nothing is mapped to this address, it reliably causes a segfault. This is equivalent to an assertion failure.

Re: Jepsen: TigerBeetle 0.16.11

#53

Post by eevmanu »

aphyr wrote: post #50
eevmanu wrote: post #47

I have a question that I hope is not misinterpreted, as I'm asking purely out of a desire to learn. I am new to distributed systems and fascinated by deterministic simulation testing. After reading the Jepsen report on TigerBeetle, the related blog post, and briefly reviewing the Antithesis integration code on GitHub workflow, I'm trying to better understand the testing scope. My core question is: could these bugs de…

Yeah, TigerBeetle's blog post goes into more detail here, but in short, the tests that were running in Antithesis (which were remarkably thorough) didn't happen to generate the precise combination of intersecting queries and out-of-order values that were necessary to find the index bug, whereas the Jepsen generator did hit that combination. There are almost certainly blind spots in the Jepsen test generators too--tha…

Thanks for your answer aphyr and for this amazing analysis

Re: Jepsen: TigerBeetle 0.16.11

#54

Post by jrpelkonen »

AndyKelley wrote: post #52
jrpelkonen wrote: post #28

Earlier quoted context omitted.

Hi Joran, I have followed TigerBeetle with interest for a while, and thank you for your inspirational work and informative presentations. However, you have stated in several occasions that the lack of memory safety in Zig is not a concern since you don't dynamically allocate memory post startup. However, one of the defects uncovered here (#2435) was caused by dereferencing an uninitialized pointer. I find this pretty…

TigerBeetle uses ReleaseSafe optimization mode, which means that the pointer was in fact initialized to 0xaaaaaaaaaaaaaaaa. Since nothing is mapped to this address, it reliably causes a segfault. This is equivalent to an assertion failure.

That’s good to hear! Thanks for the clarification.

Re: Jepsen: TigerBeetle 0.16.11

#55

Post by jrpelkonen »

matklad wrote: post #51
matklad wrote: post #31

Earlier quoted context omitted.

Note that that's a bug in the client, in the Zig-java FFI code, which is inherently unsafe. We'd likely made an a similar bug in Rust. Which is, yeah, one of the bigger technical challenges for us --- we ship language-native libraries for Go,node,Java,C#,Python and Rust, and, like in the Tolstoi novel, each one is peculiar in its own way. What's worse, they aren't directly covered by our deterministic simulator. That…

Oh, important clarification from andrewrk( https://lobste.rs/c/tf6jng ), which I totally missed myself: this isn't actually a dereference of uninitialized pointer, it's a defer of a pointer which is explicitly set to a specific, invalid value.

This is indeed an important point, the way I originally understood the bug was that the memory was not initialized at all. Thanks for the clarification

Re: Jepsen: TigerBeetle 0.16.11

#56

Post by kristoff_it »

boris wrote: post #48
jorangreef wrote: post #27

Earlier quoted context omitted.

The DBMS is Apache 2.0 and our customers pay us (well) for everything else to run, integrate, migrate, operate and support that. For more on our open source thinking and how this is orthogonal to business model (and product!), see our interview with the Changelog: https://m.youtube.com/watch?v=Yr8Y2EYnxJs

I watched that but I don't see it as convincing. Let's take the AWS example brought up in the talk. The "compete on the interface, not (open source) implementation" idea I think misses (at least) the following points: 1. AWS will take your initial and ongoing investment in the implementation but they don't have to share theirs with you. Specifically, they will take your improvements but their own improvements (say so…

> AWS will take your initial and ongoing investment in the implementation but they don't have to share theirs with you. Specifically, they will take your improvements but their own improvements (say some performance optimizations) they can keep to themselves. It's good business sense if it allows them to further differentiate their "improved" offering from your "vanilla" service.

In practice all I've seen from AWS is just to add integrations with their internal orchestrators and not much else. Back when I was at Redis Labs, AWS added TLS support to Redis and was dying to get that upstreamed (so that they wouldn't have to maintain the patch), except that as far as I understood nobody upstream wanted that code. In other words, hypothetical improvements by AWS (and other Clouds) are extremely overrated. When it comes to tigerbeetle, I would put the chance that they introduce bugs and vulnerabilities much higher than the possibility they add any meaningful improvement over what the actual experts (the tigrebeetle team) have already done.

> Do we really want to go through all of this with another vendor just for one extra service.

That's a great point, and in fact I've seen AWS purposefully offer insane (in Europe maybe we would say anti-competitive) discounts precisely to prevent Redis Labs from gaining market share. I'm sure they will try the same with TB once it becomes mainstream enough. What TB has that Redis doesn't have is the fact that it's a database designed for truly mission-critical stuff (i.e. counting the money) and maybe customers will be willing to go through the extra motions to ensure they get the best service they can (assuming TB will be able to provide that).

Re: Jepsen: TigerBeetle 0.16.11

#57

Post by ManBeardPc »

TigerBeetle is something I’m interested in. I see there is no C or Zig client listed in the clients documentation. Thought these would be the first ones to exist given it is written in Zig. Do they exist or maybe still WIP?

Re: Jepsen: TigerBeetle 0.16.11

#58

Post by boris »

kristoff_it wrote: post #56
boris wrote: post #48

Earlier quoted context omitted.

I watched that but I don't see it as convincing. Let's take the AWS example brought up in the talk. The "compete on the interface, not (open source) implementation" idea I think misses (at least) the following points: 1. AWS will take your initial and ongoing investment in the implementation but they don't have to share theirs with you. Specifically, they will take your improvements but their own improvements (say so…

> AWS will take your initial and ongoing investment in the implementation but they don't have to share theirs with you. Specifically, they will take your improvements but their own improvements (say some performance optimizations) they can keep to themselves. It's good business sense if it allows them to further differentiate their "improved" offering from your "vanilla" service. In practice all I've seen from AWS is…

> In other words, hypothetical improvements by AWS (and other Clouds) are extremely overrated.

Interesting, in a recent thread (I think it was about Redis going back open source) an AWS employer was bragging about substantial concurrency optimizations they implemented in Valkey. At the time I thought it could have been a great differentiator to keep proprietary but perhaps they decide to sacrifice it to help make sure Valkey takes over the Redis midshare.

Re: Jepsen: TigerBeetle 0.16.11

#59

Post by jorangreef »

jitl wrote: post #26

Really happy to see TigerBeetle live up to its claims as verified by aphyr - because it's good to see that when you take the right approach, you get the right results. Question about how people end up using TigerBeetle. There's presumably a lot of external systems and other databases around a TigerBeetle install for everything that isn't an Account or Transfer. What's the typical pattern for those less reliable syste…

Joran from TigerBeetle here! Thanks! Really happy to see the report published too.

The typical pattern in integrating TigerBeetle is to differentiate between control plane (Postgres for general purpose or OLGP) and data plane (TigerBeetle for transaction processing or OLTP).

All your users (names, addresses, passwords etc.) and products (descriptions, prices etc.) then go into OLGP as your "filing cabinet".

And then all the Black Friday transactions these users (or entities) make, to move products from inventory accounts to shopping cart accounts, and from there to checkout and delivery accounts—all these go into OLTP as your "bank vault". TigerBeetle lets you store up to 3 user data identifiers per account or transfer to link events (between entitites) back to your OLGP database which describes these entities.

This architecture [1] gives you a clean "separation of concerns", allowing you to scale and manage the different workloads independently. For example, if you're a bank, it's probably a good idea not to keep all your cash in the filing cabinet with the customer records, but rather to keep the cash in the bank vault, since the information has different performance/compliance/retention characteristics.

This pattern makes sense because users change their name or email address (OLGP) far less frequently than they transact (OLTP).

Finally, to preserve consistency, on the write path, you treat TigerBeetle as the OLTP data plane as your "system of record". When a "move to shopping cart" or "checkout" transaction comes in, you first write all your data dependencies to OLGP if any (and say S3 if you have related blob data) and then finally you commit your transaction by writing to TigerBeetle. On the read path, you query your system of record first, preserving strict serializability.

Does that make sense? Let me know if there's anything here we can drill into further!

[1] https://docs.tigerbeetle.com/coding/system-architecture/

Post reply on HN