Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production engineering when trading billions of dollars a day [video]

youtube.com

Post reply on HN
Print view View original

31–40 of 54 posts

Re: Production engineering when trading billions of dollars a day [video]

#31

Post by seb1204 »

LPisGood wrote: post #23
laidoffamazon wrote: post #22

I'll never understand how these cognitive elites live. They're just a completely different kind of human than the rest of us.

What do you mean by “cognitive elites?” I’ve met some exceptional people: top researchers from top universities from several fields, super well paid engineers working on products you probably use, some of the best hackers an advanced persistent threat actor could ask for; they’re just people. I think if you get a collection of competent, thoughtful people together they would come up with similar solutions to the prob…

I am sure they are, my fascination is how they manage to get so much more out of the same 24 hours than me. Sometimes I would just like to know how others manage all the worldly churn that seems to suck my time. starting from cleaning the kitchen, toilets and the home, food prep and washing up, being active, moving the lawn, reading a good book, doing taxes, bringing the vehicle to service, fix that phone for grandma etc. so how do they do it? Is it character, personality, upbringing... What helped put them on this trajectory. So in short I'm curious about their story.

Re: Production engineering when trading billions of dollars a day [video]

#32

Post by willtemperley »

cgio wrote: post #25
gricardo99 wrote: post #3

Earlier quoted context omitted.

there’s a move now towards 24/7 trading. I guess we’ll see how the rigors of the trading environment mesh with zero down time. I’m sure the rollout will be slow and steady.

Only US. Other markets barely have liquidity during daytime and get most liquidity in opening and closing auctions. Maintenance periods are actually a complication. A few more state transitions for the system, but barely used for maintenance. The only value is for upgrades, which would still be scheduled with the market down and systems up, as participants also need to transition codebases for breaking changes, a tes…

Not really US only. LMAX is 24/7 and is a UK company, famous on HN for open sourcing their ring buffer.

Crypto trading has been 24/7 since it began.

Re: Production engineering when trading billions of dollars a day [video]

#33

Post by cgio »

willtemperley wrote: post #32
cgio wrote: post #25

Earlier quoted context omitted.

Only US. Other markets barely have liquidity during daytime and get most liquidity in opening and closing auctions. Maintenance periods are actually a complication. A few more state transitions for the system, but barely used for maintenance. The only value is for upgrades, which would still be scheduled with the market down and systems up, as participants also need to transition codebases for breaking changes, a tes…

Not really US only. LMAX is 24/7 and is a UK company, famous on HN for open sourcing their ring buffer. Crypto trading has been 24/7 since it began.

Ok, I will make it mostly US. Maybe a couple more markets, London and Tokyo? Futures may be a bit broader adoption too. Vast majority won’t move to 24/7. Crypto is a different game for the time being at least. It has its own challenges but also escapes quite a few of traditional exchange complexities.

Re: Production engineering when trading billions of dollars a day [video]

#34

Post by foobar10000 »

woah wrote: post #27
foobar10000 wrote: post #18

Earlier quoted context omitted.

The EOD reconciliation (and corresponding inability to settle a position in milliseconds) is a feature - it allows "obvious erroneous trade" roll-back mechanisms, etc. Very few people want the financial system to be a contractual suicide pact - they want it to be predictable, but when the unpredictable happens - they want the retail and institutional investor to be protected (the HFT players can go beat each other up…

> they want the retail and institutional investor to be protected That costs money, indirectly

But it does allow these investors to participate in the markets without losing their shirts - and the lack of such liquidity would impact the market more so than the cost of the risk mitigation - which as you completely correctly noted is not free - both in first and second order terms.

Re: Production engineering when trading billions of dollars a day [video]

#35

Post by fatata123 »

seb1204 wrote: post #31
LPisGood wrote: post #23

Earlier quoted context omitted.

What do you mean by “cognitive elites?” I’ve met some exceptional people: top researchers from top universities from several fields, super well paid engineers working on products you probably use, some of the best hackers an advanced persistent threat actor could ask for; they’re just people. I think if you get a collection of competent, thoughtful people together they would come up with similar solutions to the prob…

I am sure they are, my fascination is how they manage to get so much more out of the same 24 hours than me. Sometimes I would just like to know how others manage all the worldly churn that seems to suck my time. starting from cleaning the kitchen, toilets and the home, food prep and washing up, being active, moving the lawn, reading a good book, doing taxes, bringing the vehicle to service, fix that phone for grandma…

Get a dishwasher. Don’t rely on a car. There are plenty of life changing effeciency hacks that can free up hours of time a day.

Re: Production engineering when trading billions of dollars a day [video]

#36

Post by hansvm »

seb1204 wrote: post #31
LPisGood wrote: post #23

Earlier quoted context omitted.

What do you mean by “cognitive elites?” I’ve met some exceptional people: top researchers from top universities from several fields, super well paid engineers working on products you probably use, some of the best hackers an advanced persistent threat actor could ask for; they’re just people. I think if you get a collection of competent, thoughtful people together they would come up with similar solutions to the prob…

I am sure they are, my fascination is how they manage to get so much more out of the same 24 hours than me. Sometimes I would just like to know how others manage all the worldly churn that seems to suck my time. starting from cleaning the kitchen, toilets and the home, food prep and washing up, being active, moving the lawn, reading a good book, doing taxes, bringing the vehicle to service, fix that phone for grandma…

> so much more out of the same 24 hours

Do they? Are they doing more work, or is the set of things they've chosen to be good at more stereotypically impressive than the set of things you've chosen to be good at?

> all the worldly churn

Surely all of that adds up to > food prep and washing

I'd be curious to hear more about what this looks like for you. I might have ideas.

Re: Production engineering when trading billions of dollars a day [video]

#37

Post by bostik »

gricardo99 wrote: post #3
jedberg wrote: post #2

This is a good talk. Really gets into the details of how things differ from the classical SaaS or consumer product. I've been doing reliability for most of my career, and have always been able to hide behind, "We're not a bank, if we lose a few requests it doesn't matter". They can't do that. :) One advantage that they have is that the market closes, so they can do maintenance that takes the whole system down, but wh…

there’s a move now towards 24/7 trading. I guess we’ll see how the rigors of the trading environment mesh with zero down time. I’m sure the rollout will be slow and steady.

Sports betting exchanges have been doing that for a very long time. There is never a good time to take the system down for maintenance - event settlements happen every few minutes, and live games with in-play betting are going on somewhere in the world at any given time.

Makes things damn hard indeed, because you have to truly learn asynchronicity, CQRS and complex live migrations. (Incidentally, engineers who have worked on such systems tend to be over-represented in extreme HA businesses.)

Re: Production engineering when trading billions of dollars a day [video]

#38

Post by linkregister »

seb1204 wrote: post #31
LPisGood wrote: post #23

Earlier quoted context omitted.

What do you mean by “cognitive elites?” I’ve met some exceptional people: top researchers from top universities from several fields, super well paid engineers working on products you probably use, some of the best hackers an advanced persistent threat actor could ask for; they’re just people. I think if you get a collection of competent, thoughtful people together they would come up with similar solutions to the prob…

I am sure they are, my fascination is how they manage to get so much more out of the same 24 hours than me. Sometimes I would just like to know how others manage all the worldly churn that seems to suck my time. starting from cleaning the kitchen, toilets and the home, food prep and washing up, being active, moving the lawn, reading a good book, doing taxes, bringing the vehicle to service, fix that phone for grandma…

I assume that Jane Street employees likely use house cleaners, food delivery, laundry services, etc. I wouldn't be the least bit surprised if part of the employee onboarding includes a list of common convenience services like these. Some employers pay for or subsidize such services; I don't know if Jane Street does.

Personal sacrifice is often expected. Fixing the phone for grandma would simply be neglected for many employees in these positions.

Re: Production engineering when trading billions of dollars a day [video]

#39

Post by rvz »

laidoffamazon wrote: post #22

I'll never understand how these cognitive elites live. They're just a completely different kind of human than the rest of us.

Nope. Jane Street engineers just love what they do; and do it for fun in their own time and professionally and they love solving puzzles.

They just think for themselves to make money and not what their manager tells them to do unless either the manager or trader believes they will lose money and the game.

The only relevant difference to you is that you must have zero morals or ethics in this game and it is actually a high stress environment behind the videos and engineering propaganda which is the slender difference between beating or losing to other competitors for their clients.

So really stop worshipping them and fuelling their egos as that is what they (Jane Street) want you to do.

Re: Production engineering when trading billions of dollars a day [video]

#40

Post by alemwjsl »

jedberg wrote: post #2

This is a good talk. Really gets into the details of how things differ from the classical SaaS or consumer product. I've been doing reliability for most of my career, and have always been able to hide behind, "We're not a bank, if we lose a few requests it doesn't matter". They can't do that. :) One advantage that they have is that the market closes, so they can do maintenance that takes the whole system down, but wh…

Not sure what the practical difference is (24/7 vs ~10/5) except for the convenience when planning data migrations if you have regularly planned downtime.

For most code changes being turned off at night isn't much of an advantage, as the new code will need to go live at some point and that point is where the risk is. For systems on 24/7 you simply need a copy of your production environment to test on, a.k.a. staging.

The main thing about 24/7 is needing follow-sun SRE and/or out of hours oncall.

Post reply on HN