Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Low Latency Optimization: Understanding Pages (Part 1)

hudsonrivertrading.com

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Low Latency Optimization: Understanding Pages (Part 1)

#31

Post by mhh__ »

SkipperCat wrote: post #19

I've been working for HFT firms since I moved to NYC over a decade ago. The article looks like a good summation HugePage benefits (I'm a sysadmin, not a programmer so I understand it on a topical level only). What I do find fascinating is that HRT is actively blogging about this stuff. Ten years ago, everyone in the biz was super secretive and never made any public announcement about what we did - even stuff that I w…

This is ezpz optimization. Everyone and their dog knows about huge pages (or at least anyone I deem worthy).

It's for recruiting, clout, and also generally expressing the culture of the firm.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#32

Post by pca006132 »

vgatherps wrote: post #21
aldanor wrote: post #18

Earlier quoted context omitted.

Not really. Most HFTs would choose some sort of kernel bypass for critical path networking needs. (unless that's what you mean by near stock)

Nothing needs to be changed about your kernel to bypass it. You can outright install the openonload drivers, preload to intercept epoll, and it literally just works. Going to efvi can cut out ~1us but that requires specifically targeting efvi, and has more operational / code setup pain. Works on stock Linux all the same though

Wondering if there is any guide to programming and accurately measuring low latency stuff. I am working on some low level memory management code and would like to see the latency behavior, but I always get several microseconds of standard deviation (~10%) when I try to benchmark it.

I am pinning the cores, disabled SMT and turbo boost, but haven't tried isolcpu because this has to reboot the computer.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#33

Post by menaerus »

JedMartin wrote: post #23
menaerus wrote: post #20

Earlier quoted context omitted.

It can be done by manually remapping the relevant sections upon application startup. Perhaps [1] is a good resource to start with (page nr. 7). Example code is here [2]. And [3] makes some experiments with it. [1] https://www.kernel.org/doc/ols/2006/ols2006v2-pages-83-90.pd... [2] https://github.com/intel/iodlr/blob/master/large_page-c/exam... [3] https://easyperf.net/blog/2022/09/01/Utilizing-Huge-Pages-Fo...

Thanks, cool stuff. Especially liblppreload.so described in [2] and [3]. I'll give it a try. Do you have any tips how to achieve the same for the stack?

I haven't done this myself but given that ELF does not have a dedicated .stack region, I guess you first have to find out what memory address range will ELF use to store variables on the stack.

Finding out the beginning of the stack should be possible to deduce from memory addresses upon entering the main().

Finding out the end of the stack depends on how big the stack is and whether it grows upwards or downwards. Former is of dynamic nature but usually configured at system level on Linux and for the latter I am not sure but I think it grows downwards.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#34

Post by JedMartin »

pca006132 wrote: post #29
anonymoushn wrote: post #10

Earlier quoted context omitted.

Your comment is correct but might cause readers to underestimate how annoying this tuning work is and how difficult it is to get everything into hugepages (executable memory and stack memory and shared libraries if applicable, not just specific heap allocations). We are trading a joke asset class on joke venues that have millisecond-scale jitter, so we can get away with using io_uring instead of kernel bypass network…

Just wondering, how useful is it to get code and stack memory into hugepages? I thought you usually access them sequentially so it doesn't matter that much to put them in hugepages.

The most benefit comes from the fact that you end up with a lot less TLB misses, since single mapping covers a large chunk of memory. Predictable memory access pattern helps with caches misses thanks to hardware prefetch, but as far as I know hardware prefetch won't work if it would cause TLB miss on most CPUs.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#35

Post by deadcanard »

bitcharmer wrote: post #14
trh0awayman wrote: post #11

Earlier quoted context omitted.

are there any hardware/software systems/setups that can put guarantees around L1/L2/main memory usage? it seems like that would be a major boon rather than just put everything in tiny arrays and hope for the best

Intel's RDT stack has that in the form of CAT (Cache Allocation Technology) and MBA (Memory Bandwidth Allocation). Some more advanced HFT shops use that extensively https://github.com/intel/intel-cmt-cat

CAT is indeed a good thing to look at. But there are some important caveats 1) unless you have a very small number of cores, it's not possible to reserve a cache slice for all programs running (some slices are shared for things like DDIO), 2) it's still not possible to lock some specific data in the cache because any collision will replace the data 3) the slices are kinda big, so it's hard to be properly fine grained. Basically, CAT just prevents other processes from stealing all the cache. It does that by reserving ways (as in the cache associativity meaning)

Re: Low Latency Optimization: Understanding Pages (Part 1)

#36

Post by bitcharmer »

zozbot234 wrote: post #17
bitcharmer wrote: post #15

Earlier quoted context omitted.

It looks like you don't have a good understanding of how virtual memory works and how in that space the hardware (TLB), the OS (page tables) and higher level software are intertwined. Also, PREEMPT_RT is the worst option for low latency because it's about execution time guarantees and not speed specifically. If you're on PREEMPT_RT and give your critical thread highest prio, be prepared for some serious OS-level lock…

> If you're on PREEMPT_RT and give your critical thread highest prio, be prepared for some serious OS-level lock-ups. PREEMPT_RT includes priority inheritance, specifically to avoid this scenario. So your app should indeed be favored if you tune accordingly. What you also seem to be saying is that using PREEMPT_RT may lead to lower throughput, but that's not the same thing as latency.

Priority inheritance is a fundamental feature of all scheduling domains in the kernel and has nothing to do with the problem I described.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#37

Post by menaerus »

pca006132 wrote: post #29
anonymoushn wrote: post #10

Earlier quoted context omitted.

Your comment is correct but might cause readers to underestimate how annoying this tuning work is and how difficult it is to get everything into hugepages (executable memory and stack memory and shared libraries if applicable, not just specific heap allocations). We are trading a joke asset class on joke venues that have millisecond-scale jitter, so we can get away with using io_uring instead of kernel bypass network…

Just wondering, how useful is it to get code and stack memory into hugepages? I thought you usually access them sequentially so it doesn't matter that much to put them in hugepages.

Code is not really accessed sequentially. Just imagine a function calling another function which sits in another translation unit. Depending what the linker is going to do but there's a good chance that these won't sit near to each other unless you explicitly optimized for that case. This is why source-code level locality is also important - to minimize the instruction cache misses. And also why you don't want to go mad about making everything dynamically dispatched in the code unless you really need to (e.g. virtual functions).

EDIT: Putting the code segment into hugepages will relief some of the pressure of VADDR translation which is otherwise larger with 4K segments. Whether this will impact the runtime execution time positively or stay neutral I think it greatly depends on the workload and cannot be said upfront.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#38

Post by vgatherps »

pca006132 wrote: post #32
vgatherps wrote: post #21

Earlier quoted context omitted.

Nothing needs to be changed about your kernel to bypass it. You can outright install the openonload drivers, preload to intercept epoll, and it literally just works. Going to efvi can cut out ~1us but that requires specifically targeting efvi, and has more operational / code setup pain. Works on stock Linux all the same though

Wondering if there is any guide to programming and accurately measuring low latency stuff. I am working on some low level memory management code and would like to see the latency behavior, but I always get several microseconds of standard deviation (~10%) when I try to benchmark it. I am pinning the cores, disabled SMT and turbo boost, but haven't tried isolcpu because this has to reboot the computer.

Really? With Google Bench or Criterion I've gotten pretty good resolution on microbenchmarks

The gold standard (in my experience) for latency measurement is setting up a packet splitting, marking your outbound packets with some hash/id of the inbound packet, taking hardware timestamps of all these on some dedicated host, and putting it all together after the fact. Ultimately packet-in to packet out is all the matters anyways

You can actually get pretty solid internal timestamps (how long did I take to fully process event X) with TSC counters, but you then have a coordinated omissions problem: https://www.programmingtalks.org/talk/how-not-to-measure-lat...

Re: Low Latency Optimization: Understanding Pages (Part 1)

#39

Post by deadcanard »

pixelpoet wrote: post #22

So many puzzling things here, from the brand new user account created to post this (a portmanteau of Jump Trading and Citadel), to the very minimal information presented (even my own article on it for my software covers about as much), to the people in the comments here conflating virtual memory with hard disk paging in spite of TFA, to red herring comments about RT scheduling, ...

URL for your article?

Re: Low Latency Optimization: Understanding Pages (Part 1)

#40

Post by intelVISA »

trh0awayman wrote: post #11
WiSaGaN wrote: post #4

Low latency trading (sometimes referred as HFT as well) focuses a lot in data locality. That is to make sure the critical path, that is from market data coming in to new order or cancel order sending out (tick-to-trade), operates in cache as much as possible and avoid memory access as much as possible. Put all those needed data together within few cachelines as possible. To make sure those data are in the cache so th…

are there any hardware/software systems/setups that can put guarantees around L1/L2/main memory usage? it seems like that would be a major boon rather than just put everything in tiny arrays and hope for the best

Custom OS that runs in L3 only (cache as RAM).
Post reply on HN