System CPU Usage and Glibc

System CPU Usage and Glibc
carun.github.io

>Our biometric datasets are sized at 1 GiB for easier file management as we could turn on transparent huge pages, if required. So it is guaranteed that mmap is being used under the hood by std::vector, so why was there TLB misses in the first place? That’s because the MAP_PRIVATE flag instructs the kernel to turn off VM_SHARED flag when creating the mapping in the kernel. This causes TLB to be cold and populate it only on demand, via page faults. As long as this translation is small, there’s no overhead. However as the memory pressure increases with concurrency, the kernel has to fight hard to populate TLB.

There is MAP_POPULATE flag to populate the mapping eagerly without waiting for page faults. I'm not sure what MAP_SHARED has to do with that.

> That aside, the reasoning behind glibc allocator’s use of mmap with MAP_PRIVATE|MAP_ANONYMOUS as defaults, is privacy and anonymity. Which makes sense, because we do not want other processes to peak into the memory region of our process. That would be a security nightmare. But I’m not sure I agree on this for inter-thread design. Threads inherently share the same process space and thus the heap. MAP_PRIVATE gets a copy-on-write mapping for performance reasons (may be). MAP_ANONYMOUS will ensure the mapping is initialized to zero. So the performance optimization is wasted away. However this doesn’t make sense for user-space data structures std::vector or malloc for that matter (FWIW, malloc, although a function, has it’s own internal data structure under the hood). No one is going to throw away a memory region after allocation without writing something into it, after all, why else will they allocate in the first place? As the size of memory being requested is large, it makes sense to use MAP_SHARED|MAP_ANONYMOUS as the default in glibc.

CoW is for separate processes, not just for any tasks. A different thread in the same process writing to a page won't trigger a copy. You don't need to pass MAP_SHARED to mmap to share memory between threads, the arguments to the clone(2) syscall which created the thread already made sure that you share the memory.

What I really suspect happened is that they didn't call std::vector::reserve at the start, so there was a lot of overhead when resizing (allocating new memory, copying the content, freeing the old memory) of std::vector, and when they moved to calling mmap directly, they effectively reserved memory at the start, as would happen if they called std::vector::reserve. Unfortunately the post doesn't contain enough details to really be sure about that.

It's confusing because his re-write isn't multi-process, but multi-thread. But then he keeps calling threads processes.

But that is just plain wrong; it's practically in the definition of threads that they share the VM between themselves. If MAP_PRIVATE meant "copy-on-write even for threads using std::vector" most multi-threaded programs would stop working, save for perhaps a couple of purely functional examples.

On the positive side, there wouldn't be any data-races. :)