Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: TigerBeetle 0.16.11

jepsen.io

Post reply on HN
Print view View original

81–88 of 88 posts

Re: Jepsen: TigerBeetle 0.16.11

#81

Post by schubart »

jorangreef wrote: post #16
nindalf wrote: post #3

Very impressed with this report. Whenever I read TigerBeetle's claims on reliability and scalability, I'd think "ok, let's wait for the Jepsen report". This report found a number of issues, which might be a cause for concern. But I think it's a positive because they didn't just fix the issues, they've expanded their internal test suite to catch similar bugs in future. With such an approach to engineering I feel like…

Thanks! Yes, we have around 6,000+ assertions in TigerBeetle. A few of these were overtight, hence some of the crashes. But those were the assertions doing their job, alerting us that we needed to adjust our mental model, which we did. Otherwise, apart from a small correctness bug in an internal testing feature we added (only in our Java client and only for Jepsen to facilitate the audit) there was only one correctne…

> we have around 6,000+ assertions in TigerBeetle.

Are they enabled in production? Are there some expensive ones that aren’t?

Re: Jepsen: TigerBeetle 0.16.11

#82

Post by jorangreef »

schubart wrote: post #81
jorangreef wrote: post #16

Earlier quoted context omitted.

Thanks! Yes, we have around 6,000+ assertions in TigerBeetle. A few of these were overtight, hence some of the crashes. But those were the assertions doing their job, alerting us that we needed to adjust our mental model, which we did. Otherwise, apart from a small correctness bug in an internal testing feature we added (only in our Java client and only for Jepsen to facilitate the audit) there was only one correctne…

> we have around 6,000+ assertions in TigerBeetle. Are they enabled in production? Are there some expensive ones that aren’t?

Yes, we drive with the seat belts on.

It’s not expensive.

Because we batch, this naturally separates the control plane from the data plane, amortizing assertions against the (larger) buffers now flowing through the data plane.

We do also have some intensive online verification checks, and these are gated behind a comptime flag.

Finally, we compile Zig with ReleaseSafe and further have all Zig’s own assertions enabled. For example, checked arithmetic for bounds overflow, which is not something you see enabled by default in safe builds for most languages, but which is critically important for safety.

The reason why all this is so important, is because if your program does something wrong in production, with people’s money, you want to know about it immediately and shutdown safely.

In other words, production is where you most need the safety, not in development (although you obviously want them there too to find bugs faster). But again, it’s the bugs that make it to production that we’re trying to catch with assertions.

Re: Jepsen: TigerBeetle 0.16.11

#84

Post by schubart »

jorangreef wrote: post #82
schubart wrote: post #81

Earlier quoted context omitted.

> we have around 6,000+ assertions in TigerBeetle. Are they enabled in production? Are there some expensive ones that aren’t?

Yes, we drive with the seat belts on. It’s not expensive. Because we batch, this naturally separates the control plane from the data plane, amortizing assertions against the (larger) buffers now flowing through the data plane. We do also have some intensive online verification checks, and these are gated behind a comptime flag. Finally, we compile Zig with ReleaseSafe and further have all Zig’s own assertions enabled…

Thanks for your reply!

> it’s the bugs that make it to production that we’re trying to catch with assertions.

Nicely put, I think I’ll steal this!

Re: Jepsen: TigerBeetle 0.16.11

#85

Post by jorangreef »

schubart wrote: post #84
jorangreef wrote: post #82

Earlier quoted context omitted.

Yes, we drive with the seat belts on. It’s not expensive. Because we batch, this naturally separates the control plane from the data plane, amortizing assertions against the (larger) buffers now flowing through the data plane. We do also have some intensive online verification checks, and these are gated behind a comptime flag. Finally, we compile Zig with ReleaseSafe and further have all Zig’s own assertions enabled…

Thanks for your reply! > it’s the bugs that make it to production that we’re trying to catch with assertions. Nicely put, I think I’ll steal this!

Great to hear, I'll be using it in future too!

Re: Jepsen: TigerBeetle 0.16.11

#86

Post by aphyr »

kiitos wrote: post #80
jorangreef wrote: post #79

Earlier quoted context omitted.

> I’ve read that paper To be clear, “Durability and the Art of Consensus” is not a paper. > tl;dr: nodes don't necessarily persist state to local disk In the consensus literature, this is sometimes referred to as “diskless crash recovery”. For example, see work by Dan Ports. > Local recovery actions, or any other kinds of node-specific details, have no relevance or influence on the information communicated thru the t…

PAR isn't any kind of panacea or golden rule for nodes in a distributed system, it describes properties of nodes that meet very narrowly-defined requirements, which are in no way universal, and which are in no way requirements for those nodes to participate in the distributed system. More broadly, there's no concept of "crash recovery" at the system level, which has any meaningful utility. Nodes are either there or t…

If I might add on to what you and Joran are both saying, after some time working with TigerBeetle, I found it useful to think of Protocol-Aware Recovery as similar to TAPIR (https://syslab.cs.washington.edu/papers/tapir-tr14.pdf). Normally we build distributed systems on top of clean abstraction layers, like "Nodes are pure state machines that do not corrupt or forget state", or "the transaction protocol assumes each key is backed by a sequentially-consistent system like a Paxos state machine". TAPIR and PAR show a path for building a more efficient, or more capable, system, by breaking the boundaries between those layers and coupling them together.

Re: Jepsen: TigerBeetle 0.16.11

#87

Post by kiitos »

matklad wrote: post #51
matklad wrote: post #31

Earlier quoted context omitted.

Note that that's a bug in the client, in the Zig-java FFI code, which is inherently unsafe. We'd likely made an a similar bug in Rust. Which is, yeah, one of the bigger technical challenges for us --- we ship language-native libraries for Go,node,Java,C#,Python and Rust, and, like in the Tolstoi novel, each one is peculiar in its own way. What's worse, they aren't directly covered by our deterministic simulator. That…

Oh, important clarification from andrewrk( https://lobste.rs/c/tf6jng ), which I totally missed myself: this isn't actually a dereference of uninitialized pointer, it's a defer of a pointer which is explicitly set to a specific, invalid value.

well, per the zig spec, any program that relies on that "explicitly set [and] specific" value of 0xAA isn't valid, so it's absolutely a bug

Re: Jepsen: TigerBeetle 0.16.11

#88

Post by rbatiati »

ManBeardPc wrote: post #57

TigerBeetle is something I’m interested in. I see there is no C or Zig client listed in the clients documentation. Thought these would be the first ones to exist given it is written in Zig. Do they exist or maybe still WIP?

Hi there! We do have a C client, the libtb_client. It's written in Zig and exposes an FFI interface that is consumed by all other TigerBeetle clients.

We don't publish the pre-compiled libs yet, but it will be available soon when we stabilize the API. For now, it can be built locally from source.

Example of using the C client: https://github.com/tigerbeetle/tigerbeetle/blob/main/src/cli...

Post reply on HN