Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production engineering when trading billions of dollars a day [video]

youtube.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Production engineering when trading billions of dollars a day [video]

#41

Post by skippyboxedhero »

infecto wrote: post #20
skippyboxedhero wrote: post #16

Earlier quoted context omitted.

Fastest ones are processing a block every 10ms. It depends what you mean by easy. Even if you are using a slow chain, you still have to compete for finite block space, you still have to work out how to risk/matching fast, etc. With chains built for exchange use, operating them easier, that is why they don't require thousands of engineers. But the actual technical capability of the system is significantly in excess of…

Hyperliquid while big in crypto is still small compared to mainstream financial markets. I don’t think you have made a case for anything yet.

So when the iphone came out and no-one was using it, it wasn't useful? I cannot conceive a reality where I perceived the current state of things as the only things that could ever exist. I do not believe anyone could have that amount of self-regard, it is impossible.

HL has been tested up to $8bn/day in volume. The gap in resources is several orders of magnitude so if big exchanges were doing 1000x more volume it wouldn't matter because HL is literally running with a handful of engineers vs thousands. In reality, HL is doing 25-40% of CME, for example.

Re: Production engineering when trading billions of dollars a day [video]

#42

Post by skippyboxedhero »

itsthecourier wrote: post #17
skippyboxedhero wrote: post #16

Earlier quoted context omitted.

Fastest ones are processing a block every 10ms. It depends what you mean by easy. Even if you are using a slow chain, you still have to compete for finite block space, you still have to work out how to risk/matching fast, etc. With chains built for exchange use, operating them easier, that is why they don't require thousands of engineers. But the actual technical capability of the system is significantly in excess of…

so let me get this straight Skippy, you're saying you got better performance and reliability than the LMAX disruptor with Multicast that runs inside many big exchanges? I have a really hard time believing something decentralized will surpass the the physical limitations of speed of light and low level assembler from C++ optimizations without any GC also the fact that hyperliquid sequencing of orders is opaque and not…

why do you think LMAX disruptor runs inside many big exchanges? have you read some blogs online? LMAX disruptor does not run at "many big exchanges".

Disruptor is written in Java. LMAX itself is a tiny exchange that largely deals with institutions so doesn't have the same message volume as retail facing.

What language do you think validators run? So much stuff that makes absolutely no sense..

Order sequencing is not opaque, that is one of the primary benefits of DEX that aren't decentralized. SOL has issues with sequencing but there are attempts to fix this (Jane Street is one of the places working on this). This is a point that I made above about block space, and isn't relevant for Hyperliquid.

Re: Production engineering when trading billions of dollars a day [video]

#43

Post by skippyboxedhero »

foobar10000 wrote: post #18
skippyboxedhero wrote: post #16

Earlier quoted context omitted.

Fastest ones are processing a block every 10ms. It depends what you mean by easy. Even if you are using a slow chain, you still have to compete for finite block space, you still have to work out how to risk/matching fast, etc. With chains built for exchange use, operating them easier, that is why they don't require thousands of engineers. But the actual technical capability of the system is significantly in excess of…

The EOD reconciliation (and corresponding inability to settle a position in milliseconds) is a feature - it allows "obvious erroneous trade" roll-back mechanisms, etc. Very few people want the financial system to be a contractual suicide pact - they want it to be predictable, but when the unpredictable happens - they want the retail and institutional investor to be protected (the HFT players can go beat each other up…

Right, but EOD also introduces credit risk on the clearing house/bilateral.

Also, I would say that probabilistic finality is one of the main issues that tradfi has with crypto (which also exists in the case of margined exchanges, for the reasons you mention). Market participants expect trades to be final, the idea that they can be rolled back is extremely unattractive.

The reason you don't need to stop the market in crypto is because you don't have EOD reconcliation. If everything settles immediately and the risk engine can keep up with the market then there is no credit risk (there have also been multiple solutions to this problem in crypto, none of them involved waiting until the end of the day to see what happens when they try to cross everyone). The reason they have market halts is to limit credit risk from the market moving in one direction and winners being unable to recover gains from the losers. It is fair to say crypto DEX haven't solved this with ADLs but they start from a better place and the higher level of competition means that innovation to invent new solutions is actually happening. The reason exchanges have shit tech is because there is no competition.

I feel like your comment is baiting because you surely know what happened at LME with trades getting cancelled because they would have caused LME insiders to lose money. Hunt Brothers caused massive issues for clearing house, HK government had to bail out clearing house...there are massive issues with the current system.

Re: Production engineering when trading billions of dollars a day [video]

#44

Post by antonvs »

gricardo99 wrote: post #3
jedberg wrote: post #2

This is a good talk. Really gets into the details of how things differ from the classical SaaS or consumer product. I've been doing reliability for most of my career, and have always been able to hide behind, "We're not a bank, if we lose a few requests it doesn't matter". They can't do that. :) One advantage that they have is that the market closes, so they can do maintenance that takes the whole system down, but wh…

there’s a move now towards 24/7 trading. I guess we’ll see how the rigors of the trading environment mesh with zero down time. I’m sure the rollout will be slow and steady.

Crypto exchanges have been doing 24/7 trading for well over a decade. Of course they’ve had their own, uh, issues - but generally, reliability of transactions hasn’t been a major one, for the big exchanges.

Re: Production engineering when trading billions of dollars a day [video]

#45

Post by [deleted] »

rvz wrote: post #39
laidoffamazon wrote: post #22

I'll never understand how these cognitive elites live. They're just a completely different kind of human than the rest of us.

Nope. Jane Street engineers just love what they do; and do it for fun in their own time and professionally and they love solving puzzles. They just think for themselves to make money and not what their manager tells them to do unless either the manager or trader believes they will lose money and the game. The only relevant difference to you is that you must have zero morals or ethics in this game and it is actually a…

[deleted]

Re: Production engineering when trading billions of dollars a day [video]

#46

Post by amluto »

nippoo wrote: post #13
amluto wrote: post #10

Earlier quoted context omitted.

An amusing, moderately expensive solution that might actually work would be to have a weekday system and a weekend system. Think of it as a spare D/R system that you intentionally swap twice a week :) If done right, it would be a complete separate system. Separate IP addresses and all.

That's effectively time-based request sharding which seems sensible but you'd still have to reconcile trades and any open positions (etc) across the time boundary where one system stops accepting requests and the other one starts. And keep the databases synchronous (ie have some system to make sure they're in sync at the changeover time) - or have a few minutes/hours of downtime between weekends and weekdays while yo…

For what it’s worth, in some financial markets, there is a sort of natural daily cutover time [0] across which you are often not trading quite the same instrument. For example, the settlement date may roll over, etc. And a lot of Very Serious Finance is already built on the idea that most parties do not instantaneously reconcile anything and don’t depend on real-time trade lists.

I really can imagine a system in which the Monday trading system runs all day and then turns off at a predetermined time. Then it has 15 minutes to produce and disseminate a final list of all transactions, after which it becomes completely unavailable and is ready for maintenance. Any subsequent amendment to Monday’s trading would be done out of band. Open orders at the end of Monday do not carry over immediately to Tuesday, although front ends are welcome to recreate them. Everyone would understand that liquidity would be thin for the first few seconds after the system rolls over.

For added fun, Monday and Tuesday could actually be allowed to overlap in a hypothetical trading system, although the market participants might not love this.

[0] which is not the same for all instruments, and holidays mean that not every instrument rolls over meaningfully every day.

Re: Production engineering when trading billions of dollars a day [video]

#47

Post by laidoffamazon »

LPisGood wrote: post #23
laidoffamazon wrote: post #22

I'll never understand how these cognitive elites live. They're just a completely different kind of human than the rest of us.

What do you mean by “cognitive elites?” I’ve met some exceptional people: top researchers from top universities from several fields, super well paid engineers working on products you probably use, some of the best hackers an advanced persistent threat actor could ask for; they’re just people. I think if you get a collection of competent, thoughtful people together they would come up with similar solutions to the prob…

Cognitive elites are the MIT and Ivy+ grads that make up Jane Street like the speaker.

People like me aren’t getting $5m+ bonuses for being an SRE (I don’t have the pedigree to work at Anthropic either)

Re: Production engineering when trading billions of dollars a day [video]

#48

Post by infecto »

skippyboxedhero wrote: post #41
infecto wrote: post #20

Earlier quoted context omitted.

Hyperliquid while big in crypto is still small compared to mainstream financial markets. I don’t think you have made a case for anything yet.

So when the iphone came out and no-one was using it, it wasn't useful? I cannot conceive a reality where I perceived the current state of things as the only things that could ever exist. I do not believe anyone could have that amount of self-regard, it is impossible. HL has been tested up to $8bn/day in volume. The gap in resources is several orders of magnitude so if big exchanges were doing 1000x more volume it wou…

No idea what you’re trying to prove. You have said a lot of negative things and your examples is hyperliquid which is processing small volume.
Post reply on HN