TigerBeetle: A Trillion Transactions [video]

nice stuff, multi master replication.

user API, super small.

doubts about how to do streaming backup.

after studying the API and doing some spike architectures I come to the conclusion (I may be wrong):

tiger beetle is awesome to keep the account balance. that's it.

because you pretty much get the transactions affecting and account and IIRC there was not a lot you can do about how to query them or use them.

also I was thinking it would be nice to have something like an account grouping other accounts to answer something like: how much money out user accounts have in this microsecond?

I think that was more or less about itm they have some special fields u128 to store ids to the transaction they represent into your actual system

and IIRC handle multi currency in different books

my conclusion was: I think I don't get it yet. I think I'm missing something. had to write a ruby client for it and build an UI to play with the API and do some transactions and see how it behaved. yet that was my conclusion

would be great to have an official UI client

https://docs.tigerbeetle.com/operating/cdc/

Benchmarking is a complicated problem, but FoundationDB claims 300,000 reads per second on a single core. TigerBeetle claims 100k-500k TPS on... Some kind of hardware?

https://apple.github.io/foundationdb/benchmarking.html

For Halo 4, I architected a system for our telemetry that that had to support subsecond roundtrip user lifetime aggregation. With up to 400ms one-way latencies, that left us with 200ms client + server time to manage lifetime counts. 60ms on the client for batching and network stack delays, and 80ms in the cloud for aggregation after routing delays.

On the client side we leveraged a lot of those TigerStyle ideas (static-allocation, simplified function surface area, explicit limits, strongly asserted). And for the cloud we effectively wrote Diagonal Scaling (Stage 7).

For gamedevs to use this, it had to be as lightweight as possible and introduce no risk of priority inversion or hitching. I wrote a many-reader/many-writer lockless circular queue with performance measured in the microseconds, it was a beast.

I was so proud of our tiny team for pulling off the vision. When we launched, there was a bug in the game: someone left in telemetry in a tight render loop during a cutsceen, so our 20k/s client-side throttles kicked in.

We hit an average of 700k/s on launch day, with peak ingestion throughput of 831k/s (2 weeks to hit a trillion). And we didn't break a sweat.

Partner teams that were trying to provide our reporting capabilities started slowing down, though, haha, and brought it to our attention ("We, uh, can't scale anymore, there's no more servers on the eastcoast data center we can allocate")... so we hit the killswitch on that category of event.

That was another piece I was proud of writing: each instrumentation point did a quick binary mask check for 64 categories and 64 subcategories to see if it should emit... one reason why the instrumentation times were so blazing fast, we had minimal branching based of a hotly-cached variable that would hang around L1 because it was touched so frequently.

Querying that day for insights, though... aggregation queries touching the launch day (that weren't on the per-user hotpath layer that was our primary use-case) would 30x query duration. XD

OH! And I forgot to mention. The tech was impressive enough, that it was quickly adopted as the backbone by every title that Microsoft published, from Gears of War to Solitaire, Forza to Minecraft.

And for comparison: The official, initial Xbox One telemetry stack used pipe-delimited strings, and supported 1-5 transactions/console/second (vs our 20k).