Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Low Latency Optimization: Understanding Pages (Part 1)

hudsonrivertrading.com

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Low Latency Optimization: Understanding Pages (Part 1)

#51

Post by deadcanard »

pca006132 wrote: post #32
vgatherps wrote: post #21

Earlier quoted context omitted.

Nothing needs to be changed about your kernel to bypass it. You can outright install the openonload drivers, preload to intercept epoll, and it literally just works. Going to efvi can cut out ~1us but that requires specifically targeting efvi, and has more operational / code setup pain. Works on stock Linux all the same though

Wondering if there is any guide to programming and accurately measuring low latency stuff. I am working on some low level memory management code and would like to see the latency behavior, but I always get several microseconds of standard deviation (~10%) when I try to benchmark it. I am pinning the cores, disabled SMT and turbo boost, but haven't tried isolcpu because this has to reboot the computer.

Have you tried one the link of the article: https://docs.kernel.org/admin-guide/kernel-per-CPU-kthreads....? Also try running "perf stat -d" on your run and see anything pops out

Re: Low Latency Optimization: Understanding Pages (Part 1)

#52

Post by bitcharmer »

deadcanard wrote: post #35
bitcharmer wrote: post #14

Earlier quoted context omitted.

Intel's RDT stack has that in the form of CAT (Cache Allocation Technology) and MBA (Memory Bandwidth Allocation). Some more advanced HFT shops use that extensively https://github.com/intel/intel-cmt-cat

CAT is indeed a good thing to look at. But there are some important caveats 1) unless you have a very small number of cores, it's not possible to reserve a cache slice for all programs running (some slices are shared for things like DDIO), 2) it's still not possible to lock some specific data in the cache because any collision will replace the data 3) the slices are kinda big, so it's hard to be properly fine grained…

1) fully agreed but most HFT apps with the exception of really simple ones like market data feed handlers which can easily fit their working set into L2 anyway will be the only thing running on a host

2) mutual cache eviction by hash collisions is solvable with a number of tricks (although those methods are not easy and often wasteful). The "DDIO slice" issue used to be a problem back when Intel used ring topology for LLC. These days they are built as a mesh thus minimizing this effect.

3) CAT doesn't recognize threads or processes. COS (class of service) uses CPU cores for way-of-cache assignments

Recent micro-architectures like SKX or CLX have 11 ways of L3 and what often happens is 1-2 ways get assigned to cpu0 for non latency-critical workloads while the rest are assigned to latency-sensitive, isolated cores usually running a single user space thread each.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#53

Post by deadcanard »

bitcharmer wrote: post #52
deadcanard wrote: post #35

Earlier quoted context omitted.

CAT is indeed a good thing to look at. But there are some important caveats 1) unless you have a very small number of cores, it's not possible to reserve a cache slice for all programs running (some slices are shared for things like DDIO), 2) it's still not possible to lock some specific data in the cache because any collision will replace the data 3) the slices are kinda big, so it's hard to be properly fine grained…

1) fully agreed but most HFT apps with the exception of really simple ones like market data feed handlers which can easily fit their working set into L2 anyway will be the only thing running on a host 2) mutual cache eviction by hash collisions is solvable with a number of tricks (although those methods are not easy and often wasteful). The "DDIO slice" issue used to be a problem back when Intel used ring topology fo…

2) Agreed about the solvability and difficulty of avoiding cache collisions. DDIO must write its data somewhere in the L3 cache. It ends up in the shareable slice. So either you're okay with sharing your cache or cannot use these slices if you want exclusive access for your processes. That was my point.

3) CAT does not recognize processes but resctl does. Feels we're kinda nitpicking here...

Last of your point: Agree, that gives you 9ish usable slices which is not very much depending on the number of cores. That was my point I was trying to make.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#54

Post by boshalfoshal »

mhh__ wrote: post #31
SkipperCat wrote: post #19

I've been working for HFT firms since I moved to NYC over a decade ago. The article looks like a good summation HugePage benefits (I'm a sysadmin, not a programmer so I understand it on a topical level only). What I do find fascinating is that HRT is actively blogging about this stuff. Ten years ago, everyone in the biz was super secretive and never made any public announcement about what we did - even stuff that I w…

This is ezpz optimization. Everyone and their dog knows about huge pages (or at least anyone I deem worthy). It's for recruiting, clout, and also generally expressing the culture of the firm.

This but unironically - I was at a top HFT firm and admittedly lots of SWE talent we try to get end up going to HRT (comp being equal) because they present themselves as more tech forward, due in part to articles like this. I'm guessing the author of the article wrote this with good intentions of providing some insight into their process, but tech blogs at the end of the day are recruiting tools.

Also agreed that this is a pretty surface level optimization, theres a reason why they are talking about it. If you are doing true HFT with purely software traders, you will probably lose to more serious players using FPGAs, which as OP mentioned isn't exactly new.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#55

Post by bitcharmer »

deadcanard wrote: post #53
bitcharmer wrote: post #52

Earlier quoted context omitted.

1) fully agreed but most HFT apps with the exception of really simple ones like market data feed handlers which can easily fit their working set into L2 anyway will be the only thing running on a host 2) mutual cache eviction by hash collisions is solvable with a number of tricks (although those methods are not easy and often wasteful). The "DDIO slice" issue used to be a problem back when Intel used ring topology fo…

2) Agreed about the solvability and difficulty of avoiding cache collisions. DDIO must write its data somewhere in the L3 cache. It ends up in the shareable slice. So either you're okay with sharing your cache or cannot use these slices if you want exclusive access for your processes. That was my point. 3) CAT does not recognize processes but resctl does. Feels we're kinda nitpicking here... Last of your point: Agree…

3) resctl just uses COS under the hood. The same limitation applies

> Yeah, that gives you 9ish usable slices which is not very much. Again that was my point

This is 9 ways that you can use for your latency-sensitive workloads exclusively. This is MUCH better than letting all that LLC get trashed by non-critical processes/threads. Typically after applying such partitioning we've observed a 15-20% speed up in our apps.

In my area shaving off a few micros that way is a huge deal and definitely worth spending a couple of minutes implementing.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#56

Post by smabie »

bob1029 wrote: post #42

If any of this is interesting to you, but you would like some deeper content to bite into, perhaps start here: https://lmax-exchange.github.io/disruptor/disruptor.html This technical paper sent me on a multi-year journey regarding one simple question: "If this stuff is fast enough for fintech, why can't we make everything work this way?" Handling millions of requests per second on 1 thread is well beyond the required…

HFT architecture tends to be grossly inefficient from a throughput perspective: i.e you'll usually be busy waiting for new data.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#57

Post by intelVISA »

bob1029 wrote: post #42

If any of this is interesting to you, but you would like some deeper content to bite into, perhaps start here: https://lmax-exchange.github.io/disruptor/disruptor.html This technical paper sent me on a multi-year journey regarding one simple question: "If this stuff is fast enough for fintech, why can't we make everything work this way?" Handling millions of requests per second on 1 thread is well beyond the required…

That's like asking why a motorbike isn't a car - they have the same goal: get from A to B but quite different considerations.

Most webscale products aren't written in performant languages and are, instead, optimized around fast feature generation and being easy (cheap) to hire for.

There's a reason laggy Electron apps are the norm $$.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#58

Post by mhh__ »

boshalfoshal wrote: post #54
mhh__ wrote: post #31

Earlier quoted context omitted.

This is ezpz optimization. Everyone and their dog knows about huge pages (or at least anyone I deem worthy). It's for recruiting, clout, and also generally expressing the culture of the firm.

This but unironically - I was at a top HFT firm and admittedly lots of SWE talent we try to get end up going to HRT (comp being equal) because they present themselves as more tech forward, due in part to articles like this. I'm guessing the author of the article wrote this with good intentions of providing some insight into their process, but tech blogs at the end of the day are recruiting tools. Also agreed that thi…

I wasn't being ironic, I genuinely think this should be required knowledge for programmers who want to use write native code.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#59

Post by boshalfoshal »

bob1029 wrote: post #42

If any of this is interesting to you, but you would like some deeper content to bite into, perhaps start here: https://lmax-exchange.github.io/disruptor/disruptor.html This technical paper sent me on a multi-year journey regarding one simple question: "If this stuff is fast enough for fintech, why can't we make everything work this way?" Handling millions of requests per second on 1 thread is well beyond the required…

The thing is, some tech companies do optimize to this degree (fb, google) - except it only really makes a lot of sense for companies that have

A. Capital (human and money)

B. Bespoke internal tools from the server up

C. Insane scale

Google and meta put in lots of effort to have fast C++ code for their core infra, and they have several teams contributing to the LLVM project. Even places like Figma optimize to some extent because part of their business alpha is being performant and smooth. When you are at the scale of FB or G, optimizing the small things can lead to massive aggregate gains, and they have the eng talent/time to justify it.

At smaller companies though, as others have mentioned, iteration time and efficient dev spend are paramount. Optimizing for microsecond latency with on your B2B SaaS product written in Python/React is most likely not part of your business case, and it is a waste of engineering time and effort to do this when you could be putting that time and money into new and better features. Most of the time, these very niche performance considerations are taken care of to a decent degree with off the shelf tools, maybe with a bit of tuning.

Post reply on HN