Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jepsen: TigerBeetle 0.16.11

jepsen.io

Post reply on HN
Print view View original

41–50 of 88 posts

Re: Jepsen: TigerBeetle 0.16.11

#41

Post by dumah »

Ygg2 wrote: post #38
saaaaaam wrote: post #29

Earlier quoted context omitted.

That's a slightly redundant criticism though - it doesn't present itself as anything other than a single purpose database designed for financial transactions. That's like saying that rice noodles are no good for making risotto. At the core they are both rice...

People seem to describe it at OLTP, and one of first DBs to come up in OLTP search is MySQL.

OLTP (Online Transaction Processing) is a database paradigm optimized for handling high volumes of short, fast transactions in real-time, typically supporting day-to-day operational activities like order processing, inventory updates, and customer account management where data integrity and quick response times are critical.

Another paradigm is OLAP, in which aggregation of large datasets is the principal concern.

Re: Jepsen: TigerBeetle 0.16.11

#42

Post by thomaspaine »

jorangreef wrote: post #11
diggan wrote: post #8

Earlier quoted context omitted.

> some of the founders of Clear Street, a fairly large brokerage on Wall Street have since invested [3] in TigerBeetle "Invested" in terms of "giving you money" or in terms of "Now uses the database themselves"? I read it as the first, but I think the question is about usage, not investments.

Both. In terms of investing and planning to migrate.

I work on the ledgering system at clear street and as far as I know we have no plans to do this. We evaluated it internally a few years ago and found that the account and transaction model was too different from ours to migrate over.

Re: Jepsen: TigerBeetle 0.16.11

#43

Post by aphyr »

matklad wrote: post #40
ryeats wrote: post #39

I think it is interesting but obvious in hindsight that it is necessary to have the distributed system under test report the time/order things actually happened to enable accurate validation against an external model of the system instead of using wall-clock time.

Note that this works because we have strict serializability. With weaker consistency guarantees, there isn't necessarily a single global consistent timeline. This is an interesting meta pattern where doing something _harder_ actually simplifies the system. Another example is that, because we assume that the disk can fail and need to include repair protocol, we get state-synchronization for a lagging replica "for free…

To build on this--this is something of a novel technique in Jepsen testing! We've done arbitrary state machine verification before, but usually that requires playing forward lots of alternate timelines: one for each possible ordering of concurrent operations. That search (see the Knossos linearizability checker) is an exponential nightmare.

In TigerBeetle, we take advantage of some special properties to make the state machine checking part linear-time. We let TigerBeetle tell us exactly which transactions happen. We can do this because it's a.) strong serializable, b.) immutable (in that we can inspect DB state to determine whether an op took place), and c.) exposes a totally ordered timestamp for every operation. Then we check that that timestamp order is consistent with real-time order, using a linear-time cycle detection approach called Elle. Having established that TigerBeetle's claims about the timestamp order are valid, we can apply those operations to a simulated version of the state machine to check semantic correctness!

I'd like to generalize this to other systems, but it's surprisingly tricky to find all three of those properties in one database. Maybe an avenue for future research!

Re: Jepsen: TigerBeetle 0.16.11

#44

Post by Ygg2 »

dumah wrote: post #41
Ygg2 wrote: post #38

Earlier quoted context omitted.

People seem to describe it at OLTP, and one of first DBs to come up in OLTP search is MySQL.

OLTP (Online Transaction Processing) is a database paradigm optimized for handling high volumes of short, fast transactions in real-time, typically supporting day-to-day operational activities like order processing, inventory updates, and customer account management where data integrity and quick response times are critical. Another paradigm is OLAP, in which aggregation of large datasets is the principal concern.

Yes, I'm aware. It seems now there is a further bifurcation. OLTP is no longer general purpose, but now it's also for only one narrow use-case.

Re: Jepsen: TigerBeetle 0.16.11

#45

Post by FlyingSnake »

Love the wonderfully detailed report. Getting it tested and signed off by Jepsen is such a huge endorsement for TigerBeetle. It’s not even reached v1.0 and I can’t wait to see it hit new milestone in the future.

Special kudos to the founders who are sharing great insights in this thread.

Re: Jepsen: TigerBeetle 0.16.11

#46

Post by jorangreef »

thomaspaine wrote: post #42
jorangreef wrote: post #11

Earlier quoted context omitted.

Both. In terms of investing and planning to migrate.

I work on the ledgering system at clear street and as far as I know we have no plans to do this. We evaluated it internally a few years ago and found that the account and transaction model was too different from ours to migrate over.

Hi Thomas, yes, I was there. However, this is something that Sachin and I subsequently discussed last year (Sachin recently provided the TPS footnote to be used in the report here). However, I understand that roadmap may since have changed, but this is to the best of my knowledge.

Re: Jepsen: TigerBeetle 0.16.11

#47

Post by eevmanu »

I have a question that I hope is not misinterpreted, as I'm asking purely out of a desire to learn. I am new to distributed systems and fascinated by deterministic simulation testing.

After reading the Jepsen report on TigerBeetle, the related blog post, and briefly reviewing the Antithesis integration code on GitHub workflow, I'm trying to better understand the testing scope.

My core question is: could these bugs detected by the Jepsen test suite have also been found by the Antithesis integration?

This question comes from a few assumptions I made, which may be incorrect:

- I thought TigerBeetle was already comprehensively tested by its internal test suite and the Antithesis product.

- I had the impression that the Antithesis test suite was more robust than Jepsen's, so I was surprised that Jepsen found an issue that Antithesis apparently did not.

I'm wondering if my understanding is flawed. For instance:

1. Was the Antithesis test suite not fully capable of detecting this specific class of bug?

2. Was this particular part of the system not yet covered by the Antithesis tests?

3. Am I fundamentally comparing apples and oranges, misunderstanding the different strengths and goals of the Jepsen and Antithesis testing suites?

I would greatly appreciate any insights that could help me understand this better. I want to be clear that my goal is to educate myself on these topics, not to make incorrect assumptions or assign responsibility.

Re: Jepsen: TigerBeetle 0.16.11

#48

Post by boris »

jorangreef wrote: post #27
wiradikusuma wrote: post #25

If memory serves, TigerBeetle is/was not free for production? I can't find the Pricing page, but I kinda remember reading about it somewhere (or it was implied) a while back.

The DBMS is Apache 2.0 and our customers pay us (well) for everything else to run, integrate, migrate, operate and support that. For more on our open source thinking and how this is orthogonal to business model (and product!), see our interview with the Changelog: https://m.youtube.com/watch?v=Yr8Y2EYnxJs

I watched that but I don't see it as convincing. Let's take the AWS example brought up in the talk. The "compete on the interface, not (open source) implementation" idea I think misses (at least) the following points:

1. AWS will take your initial and ongoing investment in the implementation but they don't have to share theirs with you. Specifically, they will take your improvements but their own improvements (say some performance optimizations) they can keep to themselves. It's good business sense if it allows them to further differentiate their "improved" offering from your "vanilla" service.

2. Competing on the the interface in this case really means competing on related services like management, etc. So your thesis is that you will provide a better/cheaper managed service than AWS. Even if that's true (a big if), most of the time the decision which service to use will have little to do with technical merit. I.e. we already use AWS, have SLA painfully negotiated, get volume discounts, etc. Do we really want to go through all of this with another vendor just for one extra service.

Just a couple of thoughts that will hopefully help you sharpen your thesis.

Re: Jepsen: TigerBeetle 0.16.11

#49

Post by sachnk99 »

jorangreef wrote: post #46
thomaspaine wrote: post #42

Earlier quoted context omitted.

I work on the ledgering system at clear street and as far as I know we have no plans to do this. We evaluated it internally a few years ago and found that the account and transaction model was too different from ours to migrate over.

Hi Thomas, yes, I was there. However, this is something that Sachin and I subsequently discussed last year (Sachin recently provided the TPS footnote to be used in the report here). However, I understand that roadmap may since have changed, but this is to the best of my knowledge.

Hi -- Sachin here, one of the founders of Clear Street. To clarify:

- The investment in TigerBeetle was done personally, not through Clear Street.

- I'm no longer actively involved day-to-day as CTO at Clear Street, but while I was, TigerBeetle was a solution we very much had in mind as our volumes were increasing.

That said, roadmaps change, priorities shift, etc. If TigerBeetle existed when we started Clear Street, I very much would have used it, and saved me from many headaches.

Re: Jepsen: TigerBeetle 0.16.11

#50

Post by aphyr »

eevmanu wrote: post #47

I have a question that I hope is not misinterpreted, as I'm asking purely out of a desire to learn. I am new to distributed systems and fascinated by deterministic simulation testing. After reading the Jepsen report on TigerBeetle, the related blog post, and briefly reviewing the Antithesis integration code on GitHub workflow, I'm trying to better understand the testing scope. My core question is: could these bugs de…

Yeah, TigerBeetle's blog post goes into more detail here, but in short, the tests that were running in Antithesis (which were remarkably thorough) didn't happen to generate the precise combination of intersecting queries and out-of-order values that were necessary to find the index bug, whereas the Jepsen generator did hit that combination.

There are almost certainly blind spots in the Jepsen test generators too--that's part of why designing different generators is so helpful!

Post reply on HN