21–30 of 54 posts
I have heard similar talks from Shopify and such back in the day, about their own product, but always love listening to more.
I'll never understand how these cognitive elites live. They're just a completely different kind of human than the rest of us.
I’ve met some exceptional people: top researchers from top universities from several fields, super well paid engineers working on products you probably use, some of the best hackers an advanced persistent threat actor could ask for; they’re just people.
I think if you get a collection of competent, thoughtful people together they would come up with similar solutions to the problems discussed in this talk.
The clickbait title of "billions of dollars a day" is nothing to praise.
This is a good talk. Really gets into the details of how things differ from the classical SaaS or consumer product. I've been doing reliability for most of my career, and have always been able to hide behind, "We're not a bank, if we lose a few requests it doesn't matter". They can't do that. :) One advantage that they have is that the market closes, so they can do maintenance that takes the whole system down, but wh…
there’s a move now towards 24/7 trading. I guess we’ll see how the rigors of the trading environment mesh with zero down time. I’m sure the rollout will be slow and steady.
Earlier quoted context omitted.
I've seen that. I suspect the exchanges will never go for it for this exact reason -- they need downtime for maintenance. But if does go through, it will be a fun challenge to get 100% uptime! I've always said that with infinite money we could get 100% uptime, but no one has infinite money. Trading firms are about as close as I can imagine to infinite money though.
How do you think on-chain exchanges do it? Hyperliquid has 16 employees, not engineers...total employees. It is possible, it isn't going to be possible for many of the legacy exchanges. I work with a major one and, being honest, from day one it was obvious they were incompetent. They employ a huge number of engineers and are unable to deliver basic features at any reasonable pace. Not even remotely close to it either…
Earlier quoted context omitted.
Fastest ones are processing a block every 10ms. It depends what you mean by easy. Even if you are using a slow chain, you still have to compete for finite block space, you still have to work out how to risk/matching fast, etc. With chains built for exchange use, operating them easier, that is why they don't require thousands of engineers. But the actual technical capability of the system is significantly in excess of…
The EOD reconciliation (and corresponding inability to settle a position in milliseconds) is a feature - it allows "obvious erroneous trade" roll-back mechanisms, etc. Very few people want the financial system to be a contractual suicide pact - they want it to be predictable, but when the unpredictable happens - they want the retail and institutional investor to be protected (the HFT players can go beat each other up…
That costs money, indirectly
Earlier quoted context omitted.
I've seen that. I suspect the exchanges will never go for it for this exact reason -- they need downtime for maintenance. But if does go through, it will be a fun challenge to get 100% uptime! I've always said that with infinite money we could get 100% uptime, but no one has infinite money. Trading firms are about as close as I can imagine to infinite money though.
I hated my time as an SRE. But … can’t it be done with some combination of canaries and blue green deployments and extensive testing? Where when things look good you just swap all the traffic to the good stuff keeping the rollback hot etc etc?
I love talks like this so much. Trading isn't something I hardly ever think about as I live/work in a bubble of sorts, and it was a fascinating listen. I have heard similar talks from Shopify and such back in the day, about their own product, but always love listening to more.
Standard SRE operations. Didn’t find anything notable. The clickbait title of "billions of dollars a day" is nothing to praise.
It's fun, because one lost or late packet is an issue immediately red in the monitoring.
I've been SRE too and the most it brought to the table is a concept of error budget.
I can only agree that "billions of dollars" in trades is not much.