Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Low Latency Optimization: Understanding Pages (Part 1)

hudsonrivertrading.com

Post reply on HN
Print view View original

11–20 of 59 posts

Re: Low Latency Optimization: Understanding Pages (Part 1)

#11

Post by trh0awayman »

WiSaGaN wrote: post #4

Low latency trading (sometimes referred as HFT as well) focuses a lot in data locality. That is to make sure the critical path, that is from market data coming in to new order or cancel order sending out (tick-to-trade), operates in cache as much as possible and avoid memory access as much as possible. Put all those needed data together within few cachelines as possible. To make sure those data are in the cache so th…

are there any hardware/software systems/setups that can put guarantees around L1/L2/main memory usage? it seems like that would be a major boon rather than just put everything in tiny arrays and hope for the best

Re: Low Latency Optimization: Understanding Pages (Part 1)

#12

Post by angry_octet »

This article is pretty thin but it's not wrong.

If you're interested in consistent low latency you do need to avoid TLB misses, and also page faults, cache contention, cache coherency delay (making sure no other cores are accessing your memory) from the CC protocol (MOESI/MESI(F)) and mis-prediction, and that's after you have put all your core's threads into SCHED_FIFO. Using https://lttng.org/ can be really helpful in checking what's happening.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#13

Post by menaerus »

gavinray wrote: post #5

An OS page size is such a prevalent notion in software it's shocking I was oblivious to this a year ago before I got interested in database internals Something that I found interesting, there's a recent presentation by Neumann about the Umbra DBMS where he fields a question about hugepages at the end. I recall him saying they don't use it, which I found interesting. I know Oracle and MySQL recommended Transparent Hug…

Transparent huge-pages are actually rarely recommended, and I think it's a quite prevalent idiom in database systems world, and that is because they can cause unexpected stalls and latency spikes during the application runtime. This is possible because THB is an in-kernel implementation of system-wide huge-page support and it's basically hard to get a lot of control over the process.

OTOH to make use of "normal" huge-pages, you have to allocate them up front so it's not possible to run into THB type of issues.

That said, I doubt that enabling huge-pages for complex database workloads, that cannot run solely in-memory, will show any noticeable performance improvement. There's a lot of IO and memory R/W involved and I think this is what shadows the TLB miss cost. What would be interesting, and what I haven't done so far, is to estimate the number of CPU cycles needed for a TLB miss.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#14

Post by bitcharmer »

trh0awayman wrote: post #11
WiSaGaN wrote: post #4

Low latency trading (sometimes referred as HFT as well) focuses a lot in data locality. That is to make sure the critical path, that is from market data coming in to new order or cancel order sending out (tick-to-trade), operates in cache as much as possible and avoid memory access as much as possible. Put all those needed data together within few cachelines as possible. To make sure those data are in the cache so th…

are there any hardware/software systems/setups that can put guarantees around L1/L2/main memory usage? it seems like that would be a major boon rather than just put everything in tiny arrays and hope for the best

Intel's RDT stack has that in the form of CAT (Cache Allocation Technology) and MBA (Memory Bandwidth Allocation). Some more advanced HFT shops use that extensively

https://github.com/intel/intel-cmt-cat

Re: Low Latency Optimization: Understanding Pages (Part 1)

#15

Post by bitcharmer »

zozbot234 wrote: post #7

If you're doing truly low latency stuff you shouldn't be swapping at all, everything should be 100% resident in memory at all times. So "pages" are totally irrelevant to you. (You should also probably be using something like the PREEMPT_RT patchset, adjust scheduling priorities and try your best to ensure that the CPU core(s) your app is running on aren't burdened by serving interrupts. Plus likely a lot of other stu…

It looks like you don't have a good understanding of how virtual memory works and how in that space the hardware (TLB), the OS (page tables) and higher level software are intertwined.

Also, PREEMPT_RT is the worst option for low latency because it's about execution time guarantees and not speed specifically. If you're on PREEMPT_RT and give your critical thread highest prio, be prepared for some serious OS-level lock-ups.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#16

Post by JedMartin »

anonymoushn wrote: post #10
vgatherps wrote: post #9

Earlier quoted context omitted.

Stock / near stock Linux is pretty close to fine for HFT. You basically only interact with the kernel on init/shutdown or outside of the fast path, and do something like isolcpus to delegate the kernel and interrupt handling to some garbage cores and give you the rest to do what you want with.

Your comment is correct but might cause readers to underestimate how annoying this tuning work is and how difficult it is to get everything into hugepages (executable memory and stack memory and shared libraries if applicable, not just specific heap allocations). We are trading a joke asset class on joke venues that have millisecond-scale jitter, so we can get away with using io_uring instead of kernel bypass network…

The part about getting everything into hugepages sounds interesting. Any idea where can I find some resources on that? Most of what I was able to find only tell you how to do that for heap allocations.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#17

Post by zozbot234 »

bitcharmer wrote: post #15
zozbot234 wrote: post #7

If you're doing truly low latency stuff you shouldn't be swapping at all, everything should be 100% resident in memory at all times. So "pages" are totally irrelevant to you. (You should also probably be using something like the PREEMPT_RT patchset, adjust scheduling priorities and try your best to ensure that the CPU core(s) your app is running on aren't burdened by serving interrupts. Plus likely a lot of other stu…

It looks like you don't have a good understanding of how virtual memory works and how in that space the hardware (TLB), the OS (page tables) and higher level software are intertwined. Also, PREEMPT_RT is the worst option for low latency because it's about execution time guarantees and not speed specifically. If you're on PREEMPT_RT and give your critical thread highest prio, be prepared for some serious OS-level lock…

> If you're on PREEMPT_RT and give your critical thread highest prio, be prepared for some serious OS-level lock-ups.

PREEMPT_RT includes priority inheritance, specifically to avoid this scenario. So your app should indeed be favored if you tune accordingly. What you also seem to be saying is that using PREEMPT_RT may lead to lower throughput, but that's not the same thing as latency.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#18

Post by aldanor »

vgatherps wrote: post #9
zozbot234 wrote: post #7

If you're doing truly low latency stuff you shouldn't be swapping at all, everything should be 100% resident in memory at all times. So "pages" are totally irrelevant to you. (You should also probably be using something like the PREEMPT_RT patchset, adjust scheduling priorities and try your best to ensure that the CPU core(s) your app is running on aren't burdened by serving interrupts. Plus likely a lot of other stu…

Stock / near stock Linux is pretty close to fine for HFT. You basically only interact with the kernel on init/shutdown or outside of the fast path, and do something like isolcpus to delegate the kernel and interrupt handling to some garbage cores and give you the rest to do what you want with.

Not really. Most HFTs would choose some sort of kernel bypass for critical path networking needs. (unless that's what you mean by near stock)

Re: Low Latency Optimization: Understanding Pages (Part 1)

#19

Post by SkipperCat »

I've been working for HFT firms since I moved to NYC over a decade ago. The article looks like a good summation HugePage benefits (I'm a sysadmin, not a programmer so I understand it on a topical level only).

What I do find fascinating is that HRT is actively blogging about this stuff. Ten years ago, everyone in the biz was super secretive and never made any public announcement about what we did - even stuff that I would take from HPE and RHEL low latency manuals (which were public knowledge). You never said anything publicly because protecting the "secret ingredients" of the trading system was paramount and any disclosure was one step towards breaking that barrier.

Now, I'm seeing HFT companies post articles like this and I'm thinking it has to be for recruiting. Why else would they do it?.

Anyway, as a side note, if you liked this article, you'd also probably like this:

http://hackingnasdaq.blogspot.com/

It was one of my favorite reads because it was written by someone going thru the journey of low latency exploration - before everything was taken over by FPGAs.

Re: Low Latency Optimization: Understanding Pages (Part 1)

#20

Post by menaerus »

JedMartin wrote: post #16
anonymoushn wrote: post #10

Earlier quoted context omitted.

Your comment is correct but might cause readers to underestimate how annoying this tuning work is and how difficult it is to get everything into hugepages (executable memory and stack memory and shared libraries if applicable, not just specific heap allocations). We are trading a joke asset class on joke venues that have millisecond-scale jitter, so we can get away with using io_uring instead of kernel bypass network…

The part about getting everything into hugepages sounds interesting. Any idea where can I find some resources on that? Most of what I was able to find only tell you how to do that for heap allocations.

It can be done by manually remapping the relevant sections upon application startup.

Perhaps [1] is a good resource to start with (page nr. 7). Example code is here [2]. And [3] makes some experiments with it.

[1] https://www.kernel.org/doc/ols/2006/ols2006v2-pages-83-90.pd...

[2] https://github.com/intel/iodlr/blob/master/large_page-c/exam...

[3] https://easyperf.net/blog/2022/09/01/Utilizing-Huge-Pages-Fo...

Post reply on HN