In HFT context (as in the article) measurement is quite easy: you tap incoming and outgoing network fibers and measure this time. Also you can do this in production, as this kind of measurement does not impact latency at all
On latency, measurement, and optimization in algorithmic trading systems
11–20 of 31 posts
Re: On latency, measurement, and optimization in algorithmic trading systems
#12The title is clickbait, unfortunately.
The article states the opposite.
> Writing fast algorithmic trading system code is hard. Measuring it properly is even harder.
Re: On latency, measurement, and optimization in algorithmic trading systems
#13Is your code really fast if you haven't measured it properly? I'd say measuring is hard but a prerequisite for writing fast code, so truly fast code is harder. The number one mistake I see people make is measuring one time and taking the results at face value. If you do nothing else, measure three times and you will at least have a feeling for the variability of your data. If you want to compare two versions of your…
> The distribution is heavily lopsided with a long tail.
You'll see this in any properly active online system. Back in the previous job we had to drill it to teams that mean() was never an acceptable latency measurement. For that reason the telemetry agent we used provided out-of-the-box p50 (median), p90, p95, p99 and max values for every timer measurement window.
The difference between p99 and max was an incredibly useful indicator of poor tail latency cases. After all, every one of those max figures was an occurrence of someone or something experiencing the long wait.
These days, if I had the pleasure of dealing with systems where individual nodes handled thousands of messages per second, I'd add p999 to the mix.
Re: On latency, measurement, and optimization in algorithmic trading systems
#14Yeah, Most people write for the happy path... Few obsess over the runtime behavior under stress.
sadly its usually cheaper to just kick the server once a week than to spend $$$xN dev hours doing the right thing and making it work well
...unless youre faang and can amortize the costs across your gigafleet
Re: On latency, measurement, and optimization in algorithmic trading systems
#15Is your code really fast if you haven't measured it properly? I'd say measuring is hard but a prerequisite for writing fast code, so truly fast code is harder. The number one mistake I see people make is measuring one time and taking the results at face value. If you do nothing else, measure three times and you will at least have a feeling for the variability of your data. If you want to compare two versions of your…
For comparing HFT implementations, the 99th percentile is often more practical than minimum values since it accounts for tail latency while excluding extreme outliers caused by GC pauses or OS scheduling.
Re: On latency, measurement, and optimization in algorithmic trading systems
#16In HFT context (as in the article) measurement is quite easy: you tap incoming and outgoing network fibers and measure this time. Also you can do this in production, as this kind of measurement does not impact latency at all
The article also touches on some reasons this isn't enough. You might want to test outside of production, you might want to measure the latency when you decide to send no order, and you might want to profile your code at a more granular level than the full lifecycle of market data to order.
Re: On latency, measurement, and optimization in algorithmic trading systems
#17You generally want the following in trading:
* mean/med/p99/p999/p9999/max over day, minute, second, 10ms
* software timestamps of rdtsc counter for interval measurements - am17 says why below
* all of that not just on a timer - but also for each event - order triggered for send, cancel sent, etc - for ease of correlation to markouts.
* hw timestamps off some sort of port replicator that has under 3ns jitter - and a way to correlate to above.
* network card timestamps for similar - solar flare card (amd now) support start of frame to start of Ethernet frame measurements.
Re: On latency, measurement, and optimization in algorithmic trading systems
#18I remember in 2017, I was trying to benchmark some highly concurrent code in F# using the async monad. I was using timers, and I was getting insanely different times for the same code, going anywhere from 0ms to 20ms without any obvious changes to the environment or anything. I was banging my head against it for hours, until I realized that async code is weird. Async code isn’t directly “run”, it’s “scheduled” and th…
Isn't that part of the point? If the code runs in the scheduler then its performance is relevant. Same with garbage collection, if the garbage collector slows your algorithm down then you usually want to know, you can try to avoid allocations and such to improve performance, and measure it using your benchmarks.
Maybe you don't always want to include this, I can see how it might be challenging to isolate just the code itself. It might be possible to swap out the scheduler, synchronization context etc for implementations more suited to that kind of benchmarks?
Re: On latency, measurement, and optimization in algorithmic trading systems
#19Is your code really fast if you haven't measured it properly? I'd say measuring is hard but a prerequisite for writing fast code, so truly fast code is harder. The number one mistake I see people make is measuring one time and taking the results at face value. If you do nothing else, measure three times and you will at least have a feeling for the variability of your data. If you want to compare two versions of your…
If you're serious about performance you generally want to use a benchmark library like JMH for Java or BenchmarkDotNet for .Net. At least for those kinds of languages where there's garbage collection and just in time compilation, runtime optimization all this stuff, there's a lot of things to consider and these libraries help you get accurate results.
Re: On latency, measurement, and optimization in algorithmic trading systems
#20You generally want the following in trading: * mean/med/p99/p999/p9999/max over day, minute, second, 10ms * software timestamps of rdtsc counter for interval measurements - am17 says why below * all of that not just on a timer - but also for each event - order triggered for send, cancel sent, etc - for ease of correlation to markouts. * hw timestamps off some sort of port replicator that has under 3ns jitter - and a…
How does rdtsc behave in the presence of multiple cores? As in: first time sample is taken on core P, process is pre-empted, then picked up again by core Q, second time sample is taken on core Q. Assume x64 Intel/AMD etc.