Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How exchanges turn order books into distributed logs

quant.engineering

Post reply on HN
Print view View original

21–30 of 73 posts

Re: How exchanges turn order books into distributed logs

#21

Post by 8cvor6j844qw_d6 »

alexpotato wrote: post #7
noitpmeder wrote: post #6

Earlier quoted context omitted.

Curious what your actual role was -- sounds very interesting! Project manager? Dev? Operations specialist? E.g. were you hired into this role, and what were the requisites?

I was what was called "Trade Desk". Many firms have them and they are a hybrid of: - DevOps (e.g. we help, or own, deployments to production) - SRE (e.g. we own the dashboards that monitored trading and would manage outages etc) - Trading Operations (e.g. we would work with exchanges to set up connections, cancel orders etc) My background is: - CompSci/Economics BA - MBA - ~20 years of basically doing the above roles…

May I know if someone with no trading knowledge can get into this field? Or do new hires that you've seen generally have some background knowledge on related to trading, etc.?

I did consider applying for a role in a very similar field, but figured I'll be fighting an uphill battle with no knowledge in trading/stock market/etc.

Re: How exchanges turn order books into distributed logs

#22

Post by WJW »

philipov wrote: post #20
alexpotato wrote: post #4

At a past job (hedge fund), my role was to co-ordinate investigations into why latency may have changed when sending orders. A couple of quants had built a random forest regression model that could take inputs like time of day, exchange, order volume etc and spit out an interval of what latency had historically been in that range. If the latency moved outside that range, an alert would fire and then I would co-ordina…

What kind of founding ethos doesn't allow tracking internal latency? Is their founding ethos "Never Admit Responsibility?"; "Never Leave A Paper Trail?" This company's official ethical foundation is "Don't Get Caught."

From the wiki about IEX: "It was founded in 2012 in order to mitigate the effects of high-frequency trading." I can see how they don't want to track internal latency as part of that, or at least not share those numbers with outsiders. That just encourages high frequency traders again.

Re: How exchanges turn order books into distributed logs

#23

Post by Scubabear68 »

I wish the article had stuck with the technical topic at hand and left out the embellishment. In particular the opening piece talking about what is happening outside the exchange.

What happens outside the exchange really doesn’t matter. The ordering will not happen until it hits the exchange.

And that is why algorithmic traders want their algos in a closet as close to the exchange both physically and also in terms of network hops as possible.

Re: How exchanges turn order books into distributed logs

#24

Post by dboreham »

alexpotato wrote: post #4

At a past job (hedge fund), my role was to co-ordinate investigations into why latency may have changed when sending orders. A couple of quants had built a random forest regression model that could take inputs like time of day, exchange, order volume etc and spit out an interval of what latency had historically been in that range. If the latency moved outside that range, an alert would fire and then I would co-ordina…

> they once swapped out a cable that was a few feet longer than the older cable and that's why the latency increased

That was not why. Possibly the cable made a difference (had an open circuit that made the NICs back down to a lower speed; noisy leading to retransmissions) but it wasn't the length per se.

Re: How exchanges turn order books into distributed logs

#27

Post by TheJoeMan »

8cvor6j844qw_d6 wrote: post #21
alexpotato wrote: post #7

Earlier quoted context omitted.

I was what was called "Trade Desk". Many firms have them and they are a hybrid of: - DevOps (e.g. we help, or own, deployments to production) - SRE (e.g. we own the dashboards that monitored trading and would manage outages etc) - Trading Operations (e.g. we would work with exchanges to set up connections, cancel orders etc) My background is: - CompSci/Economics BA - MBA - ~20 years of basically doing the above roles…

May I know if someone with no trading knowledge can get into this field? Or do new hires that you've seen generally have some background knowledge on related to trading, etc.? I did consider applying for a role in a very similar field, but figured I'll be fighting an uphill battle with no knowledge in trading/stock market/etc.

You know how coders are expected to grind out leetcode interviews? For the finance fields, a common interview topic is what you read in “The Journal” (WSJ). So just stay on top of it for a few weeks, see some trends, etc.

Re: How exchanges turn order books into distributed logs

#29

Post by alexpotato »

blibble wrote: post #13
alexpotato wrote: post #5

> Every modern exchange has a single logical sequencer. No matter how many gateways feed the system, all events flow into one component whose job is to assign the next sequence number. That integer defines the global timeline. A notable edge case here is that if EVERYTHING (e.g. market data AND orders) goes through the sequencer then you can, essentially, Denial of Service to key parts of the trading flow. e.g. one o…

why would market data go through the sequenced stream on an exchange? for an exchange: market data is a projection of the order book, an observer that sits on the stream but doesn't contribute to it and client ports have rate limits

B/c, by design, you want the archived stream of events to include everything.

e.g. a lot of these systems have a "replay" node that can be used by components that just restarted. You want the replay to include ALL of the messages seen so you can rebuild the state at any given point.

(There are, of course, tradeoffs to this so I'm just commenting on the "single sequencer" design philosophy)

Re: How exchanges turn order books into distributed logs

#30

Post by alexpotato »

noitpmeder wrote: post #12
alexpotato wrote: post #7

Earlier quoted context omitted.

I was what was called "Trade Desk". Many firms have them and they are a hybrid of: - DevOps (e.g. we help, or own, deployments to production) - SRE (e.g. we own the dashboards that monitored trading and would manage outages etc) - Trading Operations (e.g. we would work with exchanges to set up connections, cancel orders etc) My background is: - CompSci/Economics BA - MBA - ~20 years of basically doing the above roles…

Thanks for all the info! I'm a front office engineer at a prop firm -- always interesting to get insight into how others do it. We have fairly similar parallels, maybe with the exception of throwing new exchange connections to the dedicated networking group. Always love watching their incident responses from afar (usually while getting impacted desks to put away the pitchforks). Great examples of crisis management, e…

You are very welcome!

It's definitely a job that you don't hear much about but has a lot of interesting positives for people who like technology and trading. Especially if you prefer shorter term, high intensity work vs long term projects (e.g. like developers).

> Always love watching their incident responses from afar

I actually have a thread on that too: https://x.com/alexpotato/status/1227335960788160513?s=20

> (I'm sure joining KCG in August of 2012 was a wild time...)

And not surprisingly, a thread on that as well: https://x.com/alexpotato/status/1501174282969305093?s=20

Post reply on HN