21–30 of 43 posts
Can someone explain why single-threaded execution loop is more aligned with the physical realities of modern hardware ?
Joran from TigerBeetle here! I created TB. Happy to answer questions!
Earlier quoted context omitted.
If one has a TigerBeetle cluster with high inter node latency, are there any easy wins to lower the latency of the whole cluster left? My head hurts when I think of latency in large clusters, so your work this year with latency was inspiring. EDIT: I guess part of the question is about the problems with clusters with >130ms latency and if there are challenges you consider easy.
Hi, Tobi here from TB. Great question! Generally, 130 ms of network latency is challenging, and there often isn't an easy way around it as you're ultimately constrained by the speed of light (e.g. cross region deployments). That said, network latency usually follows a distribution. For example, the median might be 130 ms while p99 is 200 ms. So one important goal is to avoid being affected by the high-latency tail. I…
"By [...] utilizing a single-threaded execution loop, TigerBeetle aligns its software architecture perfectly with the physical realities of modern hardware." Can someone explain why single-threaded execution loop is more aligned with the physical realities of modern hardware ?
It really depends on the problem space. For example, many OLAP workloads (analytical) contain large amounts of parallelizable work, then multi-core execution is absolutely the way to go. That also aligns well with the direction CPU technology is taking, with core counts continuing to increase.
For the transactional workloads we see at TigerBeetle, and in other transactional systems I've worked with, the picture is quite different. We see a lot of read-modify-write operations combined with a power-law distribution of the data.
Take a simple banking example: some accounts, such as those belonging to large online retailers, see much more activity than the average individual account. You might have 80 - 90% of transfers touching a relatively small number of these hot accounts.
Operations on the same account must be serialized to preserve correctness. That means this part of the workload cannot be meaningfully parallelized. In fact, attempting to parallelize it can make performance worse because of lock contention and coordination overhead, something the "Universal Scalability Law" captures quite well (but is also easy to test out yourself with a simple experiment).
Instead, we focus on batched execution. We carefully structure execution to make effective use of CPU caches and efficient algorithms, so that a single batch can be processed extremely efficiently without any coordination. Batch execution also allows to amortize I/O and replication.
That being said, there are areas where we could use multi-threading (e.g. compaction) that are not on the hot execution path.
Joran from TigerBeetle here! I created TB. Happy to answer questions!
When tiger beetle becomes pluggable to different use cases, how should I think about "do I want tiger beetle?"
Joran from TigerBeetle here! I created TB. Happy to answer questions!
Earlier quoted context omitted.
Hi, Tobi here from TB. Great question! Generally, 130 ms of network latency is challenging, and there often isn't an easy way around it as you're ultimately constrained by the speed of light (e.g. cross region deployments). That said, network latency usually follows a distribution. For example, the median might be 130 ms while p99 is 200 ms. So one important goal is to avoid being affected by the high-latency tail. I…
Is this flexible quorum something that deviates from VSR? I thought VSR requires majority of the nodes to form quorum.
The basic idea is pretty simple. In VSR, there are two main phases:
1. Leader election
2. Normal replication / request processing
Before Heidi Howard’s insight, these two phases typically used the same quorum size - for example, 4 out of 6 replicas.
The key observation was that the two phases can actually use different quorum sizes, as long as the relevant quorums still intersect.
With 6 replicas, we could use a quorum of 4 for view change and a quorum of 3 for normal processing, because 4+3>6. This guarantees that every view-change quorum intersects every processing quorum. Therefore, if an operation was committed by a processing quorum, at least one replica participating in the subsequent view change knows about that operation. Combined with the protocol's view-change/log-selection rules, this ensures that committed operations are preserved when the new leader takes over.
If this interests you, Heidi gave a talk about this at systems distributed: https://youtu.be/P0cAG-RM1_c which will be released soon.
Joran from TigerBeetle here! I created TB. Happy to answer questions!
How has AI changed the way that TigerBeetle does software engineering? Given the project’s idiosyncratic language/memory allocation choices, it’s an interesting data point how well the frontier models work for you guys.
We still write, read (and have an independent engineer review) each line of code by hand.
We go faster like that, but, most of all, it’s the guarantee we make to our users, also to continue to invest in our own understanding, because second order that’s valuable for the kind of high performance safety work we do.
Long term, I’m sure LLMs will improve, but right now they’re just not there.
I really wish they turned it into a dependency or database framework where users could define their own business logic to swap out the double entry accounting, while reusing all the system architecture and networking features, consensus etc. Sort of like a new paradigm where opinionated custom databases could be created with arbitrary entry logic built on this stack.
llvm for databases?