Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production engineering when trading billions of dollars a day [video]

youtube.com

Post reply on HN
Print view View original

11–20 of 54 posts

Re: Production engineering when trading billions of dollars a day [video]

#11

Post by gigatexal »

jedberg wrote: post #6
gigatexal wrote: post #5

Earlier quoted context omitted.

I hated my time as an SRE. But … can’t it be done with some combination of canaries and blue green deployments and extensive testing? Where when things look good you just swap all the traffic to the good stuff keeping the rollback hot etc etc?

That's how we got 99.99% at Netflix. And it cost a lot of money. But a canary implies that something may go wrong and you have to roll back. The canary is still production traffic, so some transactions would fail, which isn't allowed for this kind of workload. I image you'd have to use shadow execution, where you roll out a full second copy, run every transaction through both, and compare the results. And then, only…

Folks that keep the lights on 24/7 aka SREs are super heroes that wear capes. Thank you for your service.

I couldn’t do it. I like infra and all but it’s just not my cup of tea. Def true that in a trading pov the trade must be executed. It must settle. It must work. Or capital flight will be huge.

Re: Production engineering when trading billions of dollars a day [video]

#12

Post by dmurray »

TacticalCoder wrote: post #7
gricardo99 wrote: post #3

Earlier quoted context omitted.

there’s a move now towards 24/7 trading. I guess we’ll see how the rigors of the trading environment mesh with zero down time. I’m sure the rollout will be slow and steady.

> there’s a move now towards 24/7 trading. Isn't the plan more like 23/5 like is already the case for several markets? I can't see the standard sessions moving more 9:30am/4pm weekdays to 24/7. I take it they'd still let, at least, one hour off for technical reasons. If I'm not mistaken it's the reason several markets are 23/5 and not 24/5: that one hour of downtime is basically for servers/maintenance right? (maybe…

The 23/7 is not so much for maintenance as to have a defined window for changes to the market to happen.

Every so often a new stock is listed or a stock ticker is changed or a stock is split, etc. There are smaller changes every single day, like to the settlement date of your trade.

It's very convenient to be able to restart all your systems at 5pm, have them all load the updated reference data, and start them again in time for 6pm (or 7pm, or 4am tomorrow...). Even if you trade stocks and options and currencies and futures all over the world, a quirk of the calendar means they're basically all closed between 4 and 5pm Chicago time.

Of course it's possible in principle to build systems where all this is dynamic and you can seamlessly trade with the old configuration at 4:59:59.999 and start trading the new one a millisecond later. But literally everyone has built systems that don't work on this, that rely on being able to chunk the continuous passage of time into discrete days. It would be painful to rearchitect them all now.

Re: Production engineering when trading billions of dollars a day [video]

#13

Post by nippoo »

amluto wrote: post #10
jedberg wrote: post #4

Earlier quoted context omitted.

I've seen that. I suspect the exchanges will never go for it for this exact reason -- they need downtime for maintenance. But if does go through, it will be a fun challenge to get 100% uptime! I've always said that with infinite money we could get 100% uptime, but no one has infinite money. Trading firms are about as close as I can imagine to infinite money though.

An amusing, moderately expensive solution that might actually work would be to have a weekday system and a weekend system. Think of it as a spare D/R system that you intentionally swap twice a week :) If done right, it would be a complete separate system. Separate IP addresses and all.

That's effectively time-based request sharding which seems sensible but you'd still have to reconcile trades and any open positions (etc) across the time boundary where one system stops accepting requests and the other one starts. And keep the databases synchronous (ie have some system to make sure they're in sync at the changeover time) - or have a few minutes/hours of downtime between weekends and weekdays while you copy the whole production database from one system to another. The devil is in the details!

Re: Production engineering when trading billions of dollars a day [video]

#14

Post by skippyboxedhero »

jedberg wrote: post #4
gricardo99 wrote: post #3

Earlier quoted context omitted.

there’s a move now towards 24/7 trading. I guess we’ll see how the rigors of the trading environment mesh with zero down time. I’m sure the rollout will be slow and steady.

I've seen that. I suspect the exchanges will never go for it for this exact reason -- they need downtime for maintenance. But if does go through, it will be a fun challenge to get 100% uptime! I've always said that with infinite money we could get 100% uptime, but no one has infinite money. Trading firms are about as close as I can imagine to infinite money though.

How do you think on-chain exchanges do it? Hyperliquid has 16 employees, not engineers...total employees. It is possible, it isn't going to be possible for many of the legacy exchanges.

I work with a major one and, being honest, from day one it was obvious they were incompetent. They employ a huge number of engineers and are unable to deliver basic features at any reasonable pace. Not even remotely close to it either (as in: you ask them to do something, they say yes, execs say yes, you get a deadline, date comes...deployment difficulties, environment not working, run around goes on and on forever).

I remember the CEO got on a call with us at the start and was slapping himself on the back saying they had no downtime...because they were able to do maintenance when markets shut (and have heard very bad things about how that goes). But it is 24/7 world now, our service is up 24/7 and, of course, this led to massive issues in time due to the very different expectations around delivery/quality. Our execs were impressed, our engineers said this was a bad sign. And, ofc, it transpired that they were total amateurs (to be clear, this is one of the biggest exchanges in the world) and were unable to deliver.

To come back to my original statement: there is a company of 16 people total who is, from the point of view of customers, delivering features faster. It is difficult to understate how insane that is.

Re: Production engineering when trading billions of dollars a day [video]

#15

Post by jedberg »

skippyboxedhero wrote: post #14
jedberg wrote: post #4

Earlier quoted context omitted.

I've seen that. I suspect the exchanges will never go for it for this exact reason -- they need downtime for maintenance. But if does go through, it will be a fun challenge to get 100% uptime! I've always said that with infinite money we could get 100% uptime, but no one has infinite money. Trading firms are about as close as I can imagine to infinite money though.

How do you think on-chain exchanges do it? Hyperliquid has 16 employees, not engineers...total employees. It is possible, it isn't going to be possible for many of the legacy exchanges. I work with a major one and, being honest, from day one it was obvious they were incompetent. They employ a huge number of engineers and are unable to deliver basic features at any reasonable pace. Not even remotely close to it either…

From what I've seen, on-chain transaction times are measured in seconds and minutes, not milliseconds. It's a lot easier when you have time to wait to process a queue.

Re: Production engineering when trading billions of dollars a day [video]

#16

Post by skippyboxedhero »

jedberg wrote: post #15
skippyboxedhero wrote: post #14

Earlier quoted context omitted.

How do you think on-chain exchanges do it? Hyperliquid has 16 employees, not engineers...total employees. It is possible, it isn't going to be possible for many of the legacy exchanges. I work with a major one and, being honest, from day one it was obvious they were incompetent. They employ a huge number of engineers and are unable to deliver basic features at any reasonable pace. Not even remotely close to it either…

From what I've seen, on-chain transaction times are measured in seconds and minutes, not milliseconds. It's a lot easier when you have time to wait to process a queue.

Fastest ones are processing a block every 10ms.

It depends what you mean by easy. Even if you are using a slow chain, you still have to compete for finite block space, you still have to work out how to risk/matching fast, etc.

With chains built for exchange use, operating them easier, that is why they don't require thousands of engineers. But the actual technical capability of the system is significantly in excess of tradfi exchanges. For example, risk function is real-time on-chain as opposed to EoD settlement. This significantly changes the possible feature set. Once you have built it, it is very easy...the question is why big exchanges rely so heavily on eod processes? The answer is: they are bad at engineering.

Re: Production engineering when trading billions of dollars a day [video]

#17

Post by itsthecourier »

skippyboxedhero wrote: post #16
jedberg wrote: post #15

Earlier quoted context omitted.

From what I've seen, on-chain transaction times are measured in seconds and minutes, not milliseconds. It's a lot easier when you have time to wait to process a queue.

Fastest ones are processing a block every 10ms. It depends what you mean by easy. Even if you are using a slow chain, you still have to compete for finite block space, you still have to work out how to risk/matching fast, etc. With chains built for exchange use, operating them easier, that is why they don't require thousands of engineers. But the actual technical capability of the system is significantly in excess of…

so let me get this straight Skippy, you're saying you got better performance and reliability than the LMAX disruptor with Multicast that runs inside many big exchanges?

I have a really hard time believing something decentralized will surpass the the physical limitations of speed of light and low level assembler from C++ optimizations without any GC

also the fact that hyperliquid sequencing of orders is opaque and not opensource, and there is indeed latency in the consensus, I cannot believe yet there are p99 stability in completed transactions

Re: Production engineering when trading billions of dollars a day [video]

#18

Post by foobar10000 »

skippyboxedhero wrote: post #16
jedberg wrote: post #15

Earlier quoted context omitted.

From what I've seen, on-chain transaction times are measured in seconds and minutes, not milliseconds. It's a lot easier when you have time to wait to process a queue.

Fastest ones are processing a block every 10ms. It depends what you mean by easy. Even if you are using a slow chain, you still have to compete for finite block space, you still have to work out how to risk/matching fast, etc. With chains built for exchange use, operating them easier, that is why they don't require thousands of engineers. But the actual technical capability of the system is significantly in excess of…

The EOD reconciliation (and corresponding inability to settle a position in milliseconds) is a feature - it allows "obvious erroneous trade" roll-back mechanisms, etc.

Very few people want the financial system to be a contractual suicide pact - they want it to be predictable, but when the unpredictable happens - they want the retail and institutional investor to be protected (the HFT players can go beat each other up - no one will really cry about them). And unpredictable can be anything from a power event taking out multiple exchanges in the NJ triangle (Sandy hurricane) to a cyber-attack (never happened yet) to a flash-crash driven by algorithms from multiple HFT driving each other nuts (happened at least once).

So, it is not EOD processes as such, but the ability to pause, assess the entire system holistically, and then correct it before it blows up the portfolios of everyone holding a 401k. So even though the exchanges _could_ got to 24/7 trading, I'd be surprised if we just went away from cyclical 24-hr based windows of settlement.

Re: Production engineering when trading billions of dollars a day [video]

#19

Post by justinclift »

jedberg wrote: post #4
gricardo99 wrote: post #3

Earlier quoted context omitted.

there’s a move now towards 24/7 trading. I guess we’ll see how the rigors of the trading environment mesh with zero down time. I’m sure the rollout will be slow and steady.

I've seen that. I suspect the exchanges will never go for it for this exact reason -- they need downtime for maintenance. But if does go through, it will be a fun challenge to get 100% uptime! I've always said that with infinite money we could get 100% uptime, but no one has infinite money. Trading firms are about as close as I can imagine to infinite money though.

Heh, maybe they'll develop a sudden interest in the old Vax VMS clustering approach? ;)

Re: Production engineering when trading billions of dollars a day [video]

#20

Post by infecto »

skippyboxedhero wrote: post #16
jedberg wrote: post #15

Earlier quoted context omitted.

From what I've seen, on-chain transaction times are measured in seconds and minutes, not milliseconds. It's a lot easier when you have time to wait to process a queue.

Fastest ones are processing a block every 10ms. It depends what you mean by easy. Even if you are using a slow chain, you still have to compete for finite block space, you still have to work out how to risk/matching fast, etc. With chains built for exchange use, operating them easier, that is why they don't require thousands of engineers. But the actual technical capability of the system is significantly in excess of…

Hyperliquid while big in crypto is still small compared to mainstream financial markets.

I don’t think you have made a case for anything yet.

Post reply on HN