Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

System CPU Usage and Glibc

carun.github.io

Post reply on HN
Print view View original

11–20 of 24 posts

Re: System CPU Usage and Glibc

#11

Post by tyingq »

AshamedCaptain wrote: post #9
tyingq wrote: post #7

Earlier quoted context omitted.

He's saying that MAP_PRIVATE|MAP_ANONYMOUS causes copy-on-write not just for forked processes, but for threads using std::vector. So he redesigned with a RAII wrapper to explicitly use mmap() with MAP_SHARED to avoid the copy-on-write. It's confusing because his re-write isn't multi-process, but multi-thread. But then he keeps calling threads processes.

> He's saying that MAP_PRIVATE|MAP_ANONYMOUS causes copy-on-write not just for forked processes But that is just plain wrong; it's practically in the definition of threads that they share the VM between themselves. If MAP_PRIVATE meant "copy-on-write even for threads using std::vector" most multi-threaded programs would stop working, save for perhaps a couple of purely functional examples. On the positive side, there…

Maybe he means "copy on write" for copies of a std::vector, instead of real copy.

Re: System CPU Usage and Glibc

#12

Post by scottlamb »

AshamedCaptain wrote: post #2

Nothing in this article makes any sense. You simply cannot use MAP_SHARED by default because then it will wreak havoc the moment anyone uses fork() (for anything other than to immediately exec). And I fail to see absolutely any reason why MAP_SHARED vs MAP_PRIVATE would alter the performance characteristics of multi-threading within the same process, since altought the multiple threads are technically forks/clones, t…

> So can someone please explain? Maybe what he actually wants is madvise or hugetlbfs ?

I think you're getting warm. It's hard to know anything from this confused article but I'll assume they're correct that there was only one process, the pages were filled prior to the load test, and there were many minor page faults later. My best guess is that the page faults are due to the kernel's moving stuff to transparent huge pages in the background (and struggling to find/make unfragmented physical address space). The behavior change was due to the mmap size change they observed: before with the extra bytes the kernel didn't initially use huge pages. After, it did, so no background operation was necessary. (Or maybe MAP_SHARED helped because they had them backed by a non tmpfs filesystem, where huge issues aren't supported. Or maybe some behavior changes were just because they tested multiple times without rebooting and the physical page space got fragmented. Again, hard to know much from this article.)

If I were them, I'd confirm this by "perf record" or eBPF and/or by disabling transparent huge page compaction. Then I'd switch to using dedicated huge pages, because huge pages really do speed things up by like 15% for a memory heavy workload; they're worth it if you can avoid THP's sometimes pathological behavior.

Edit to add: Here is a much clearer article explaining the same problem. https://pingcap.com/blog/why-we-disable-linux-thp-feature-fo... I don't agree with the pingcap folks that you should permanently disable THP altogether but they did good debugging.

Re: System CPU Usage and Glibc

#13

Post by formerly_proven »

st_goliath wrote: post #8
AshamedCaptain wrote: post #2

Nothing in this article makes any sense. You simply cannot use MAP_SHARED by default because then it will wreak havoc the moment anyone uses fork() (for anything other than to immediately exec). And I fail to see absolutely any reason why MAP_SHARED vs MAP_PRIVATE would alter the performance characteristics of multi-threading within the same process, since altought the multiple threads are technically forks/clones, t…

> Nothing in this article makes any sense. Well, that's basically it. The author appears to have heard some explanations on Unix & OS internals, but not quite understood them and appears to confuse a lot of things. From the looks of things, I figure the author did some voodoo problem solving and then, convinced of having understood the problem, decided to write an article about it. There's a lot of second guessing, e…

This reminds me a little bit of a lengthy blog article by some FAANG engineer about picking optimal chunk sizes for I/O where he did a lot of benchmarking with /dev/null and /dev/zero.

Re: System CPU Usage and Glibc

#14

Post by gopalv »

scottlamb wrote: post #12
AshamedCaptain wrote: post #2

Nothing in this article makes any sense. You simply cannot use MAP_SHARED by default because then it will wreak havoc the moment anyone uses fork() (for anything other than to immediately exec). And I fail to see absolutely any reason why MAP_SHARED vs MAP_PRIVATE would alter the performance characteristics of multi-threading within the same process, since altought the multiple threads are technically forks/clones, t…

> So can someone please explain? Maybe what he actually wants is madvise or hugetlbfs ? I think you're getting warm. It's hard to know anything from this confused article but I'll assume they're correct that there was only one process, the pages were filled prior to the load test, and there were many minor page faults later. My best guess is that the page faults are due to the kernel's moving stuff to transparent hug…

> If I were them, I'd confirm this by "perf record" or eBPF and/or by disabling transparent huge page compaction

The report does look a lot like transparent hugepage defrag moving around memory randomly & system CPU spikes for no good reason.

And the workaround is valid, because THP can currently only map anonymous memory regions such as heap and stack space.

Literally the first thing I have to do to a Hadoop cluster is go around and turn off the defrag for the multi-threaded bits we have or system CPU goes over 20% as the system goes into workloads.

The other way you end up with a thundering herd causing permanent issues is with NUMA balancing (+ false sharing). So if you scale up a system slowly, you end up allocating in the same NUMA zone for a whole section (like a 1Gb array), which makes traversing it faster, but if this happens at the same time as another thread touching the same data, it will get scheduled over in a different zone & the kernel can do a lot of busy work with zone rebalancing as you go above 50% memory usage. This used to be a big deal when the first ccNUMA Intel boxes were coming out & mysql at Yahoo used to have so much trouble with NUMA messing up memory access speed assumptions[1].

The 96 core + 235GiB of data suggests there is some NUMA messiness going on for sure, particularly because there's a pinned worker to a CPU in the design.

The NUMA issues were front-and-center when the Power8 porting of Hadoop was going on , particularly because Java just gives you two things you can tweak directly in the GC (UseNUMA and UseTLAB). Because it was a lot of Cores on the bus (128 cores x 1TB - NUMA is like 1x-4x slower, which is hard to optimize for in a general sense).

[1] - https://blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-...

Re: System CPU Usage and Glibc

#15

Post by scottlamb »

gopalv wrote: post #14
scottlamb wrote: post #12

Earlier quoted context omitted.

> So can someone please explain? Maybe what he actually wants is madvise or hugetlbfs ? I think you're getting warm. It's hard to know anything from this confused article but I'll assume they're correct that there was only one process, the pages were filled prior to the load test, and there were many minor page faults later. My best guess is that the page faults are due to the kernel's moving stuff to transparent hug…

> If I were them, I'd confirm this by "perf record" or eBPF and/or by disabling transparent huge page compaction The report does look a lot like transparent hugepage defrag moving around memory randomly & system CPU spikes for no good reason. And the workaround is valid, because THP can currently only map anonymous memory regions such as heap and stack space. Literally the first thing I have to do to a Hadoop cluster…

Great point about NUMA. It sounds like one request is basically "scan the entire gallery for matches to this image". This is similar to websearch. The ideal thing would be for each of the pinned, per-core threads to allocate and initialize its own memory region and be responsible for scanning just that memory region, so the 96 threads all work on the same request then move on to the next. This would be more efficient as well as have lower latency at > And the workaround is valid, because THP can currently only map anonymous memory regions such as heap and stack space.

Also tmpfs, fwiw. https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... says "Currently THP only works for anonymous memory mappings and tmpfs/shmem. But in the future it can expand to other filesystems." I'm eagerly waiting for that future...

I don't think they're using tmpfs because they said "The default tmpfs on the host was untouched and was left at 50% (128 GiB)", and 235 GiB doesn't fit in 128 GiB.

Re: System CPU Usage and Glibc

#16

Post by nyrikki »

AshamedCaptain wrote: post #9
tyingq wrote: post #7

Earlier quoted context omitted.

He's saying that MAP_PRIVATE|MAP_ANONYMOUS causes copy-on-write not just for forked processes, but for threads using std::vector. So he redesigned with a RAII wrapper to explicitly use mmap() with MAP_SHARED to avoid the copy-on-write. It's confusing because his re-write isn't multi-process, but multi-thread. But then he keeps calling threads processes.

> He's saying that MAP_PRIVATE|MAP_ANONYMOUS causes copy-on-write not just for forked processes But that is just plain wrong; it's practically in the definition of threads that they share the VM between themselves. If MAP_PRIVATE meant "copy-on-write even for threads using std::vector" most multi-threaded programs would stop working, save for perhaps a couple of purely functional examples. On the positive side, there…

If you are using glibc you really need to see what clone() flags are being passed as fork() is just mapping to clone() with defaults now.

Now some of the other libc variants used by space optimized containers like alpine will still use the legacy fork()

But I constantly see developers confused because they were taught that the v7 style fork() is still used in modern POSIX.

It is didactic and not a rule, focusing on clone() helps get past the change in behavior for both threads and processes.

This author seems to have missed this change in behavior and went on a few crazy paths.

Re: System CPU Usage and Glibc

#17

Post by nyrikki »

nyrikki wrote: post #16
AshamedCaptain wrote: post #9

Earlier quoted context omitted.

> He's saying that MAP_PRIVATE|MAP_ANONYMOUS causes copy-on-write not just for forked processes But that is just plain wrong; it's practically in the definition of threads that they share the VM between themselves. If MAP_PRIVATE meant "copy-on-write even for threads using std::vector" most multi-threaded programs would stop working, save for perhaps a couple of purely functional examples. On the positive side, there…

If you are using glibc you really need to see what clone() flags are being passed as fork() is just mapping to clone() with defaults now. Now some of the other libc variants used by space optimized containers like alpine will still use the legacy fork() But I constantly see developers confused because they were taught that the v7 style fork() is still used in modern POSIX. It is didactic and not a rule, focusing on c…

To expand on this Linux threads, NPTL and fork() all call clone() under modern Linux and glibc.

The flags on that call and mostly what is inherited or dropped is the main difference.

Add in pid remapping with namespaces in containers and it is common for people to go on wild goose chases if they assume that the legacy fork() model is in play.

Re: System CPU Usage and Glibc

#18

Post by jcranberry »

AshamedCaptain wrote: post #2

Nothing in this article makes any sense. You simply cannot use MAP_SHARED by default because then it will wreak havoc the moment anyone uses fork() (for anything other than to immediately exec). And I fail to see absolutely any reason why MAP_SHARED vs MAP_PRIVATE would alter the performance characteristics of multi-threading within the same process, since altought the multiple threads are technically forks/clones, t…

Seems to me he intends to talk about the page table, not the TLB.

Re: System CPU Usage and Glibc

#19

Post by AshamedCaptain »

nyrikki wrote: post #16
AshamedCaptain wrote: post #9

Earlier quoted context omitted.

> He's saying that MAP_PRIVATE|MAP_ANONYMOUS causes copy-on-write not just for forked processes But that is just plain wrong; it's practically in the definition of threads that they share the VM between themselves. If MAP_PRIVATE meant "copy-on-write even for threads using std::vector" most multi-threaded programs would stop working, save for perhaps a couple of purely functional examples. On the positive side, there…

If you are using glibc you really need to see what clone() flags are being passed as fork() is just mapping to clone() with defaults now. Now some of the other libc variants used by space optimized containers like alpine will still use the legacy fork() But I constantly see developers confused because they were taught that the v7 style fork() is still used in modern POSIX. It is didactic and not a rule, focusing on c…

What change in behavior? "v7-style" fork is still used in modern POSIX, and most definitely in HPC/industrial/academic software. If you have no use for threads sharing memory, that is.

If any libc is translating fork() calls into pseudo-equivalents of "vfork" then that libc is just not-POSIX.

Re: System CPU Usage and Glibc

#20

Post by nyrikki »

AshamedCaptain wrote: post #19
nyrikki wrote: post #16

Earlier quoted context omitted.

If you are using glibc you really need to see what clone() flags are being passed as fork() is just mapping to clone() with defaults now. Now some of the other libc variants used by space optimized containers like alpine will still use the legacy fork() But I constantly see developers confused because they were taught that the v7 style fork() is still used in modern POSIX. It is didactic and not a rule, focusing on c…

What change in behavior? "v7-style" fork is still used in modern POSIX, and most definitely in HPC/industrial/academic software. If you have no use for threads sharing memory, that is. If any libc is translating fork() calls into pseudo-equivalents of "vfork" then that libc is just not-POSIX.

It is not vfork, it is similar to rfork in FreeBSD but note that fork(), vfork(), and __clone() all invoke clone() in Linux post 2.4. The costs of copying the parent tables and creating the task is the only additional cost for fork(), which is very different than the didactic explanation of fork()+exec().

https://man7.org/linux/man-pages/man2/fork.2.html

"C library/kernel differences Since version 2.3.3, rather than invoking the kernel's fork() system call, the glibc fork() wrapper that is provided as part of the NPTL threading implementation invokes clone(2) with flags that provide the same effect as the traditional system call. (A call to fork() is equivalent to a call to clone(2) specifying flags as just SIGCHLD.) The glibc wrapper invokes any fork handlers that have been established using pthread_atfork(3)."

Post reply on HN