61–70 of 79 posts
Why, why do they make it header only? Is it so difficult to integrate a couple source file along with the existing headers? We should not forget compilation times. A project I use depends on spdlog, a header-only C++ logging library. The thing adds almost two seconds per compilation unit to single threaded build times. And since logging is kinda used everywhere, the whole project takes forever to build (trice the bui…
Earlier quoted context omitted.
“Compiled every time” is not really true if the header correctly contains “#pragma once” or #ifdef guards. It may be that you have used a lot of template-based headers, which may compile nearly every time because they are literally creating new code every time a new combination of template parameters is given.
It'll be compiled once for every Cpp file that includes it. Pragma once (or ifdefs) means that it only gets included once for that compilation, and has no effect on any other Cpp files. You are correct that I'm mostly talking about template heavy header files. There is a strong correlation between template based header files and header only libraries. The matter) latter generally means the former.
but are you going to mmap stuff in all your .cpps ? Most of the time when I use an external library, it does not get out of a single implementation file, so it being header only does not really make it worse
Earlier quoted context omitted.
It'll be compiled once for every Cpp file that includes it. Pragma once (or ifdefs) means that it only gets included once for that compilation, and has no effect on any other Cpp files. You are correct that I'm mostly talking about template heavy header files. There is a strong correlation between template based header files and header only libraries. The matter) latter generally means the former.
> It'll be compiled once for every Cpp file that includes it. but are you going to mmap stuff in all your .cpps ? Most of the time when I use an external library, it does not get out of a single implementation file, so it being header only does not really make it worse
I made a similar library for writing data into memory mapped files, also a self-contained header-only lib:
https://github.com/Morgan-Stanley/hobbes/blob/master/include...
This one also serializes a representation of the type structure of recorded data so that it can be safely concurrently mmapped and read either with the same code or with the generic PL/compiler that I've developed in this hobbes project.
Where unpredictable disk latency is a problem, we've got a similar header-only lib for logging into shared memory (then have another process to consume this shared memory ring buffer and dump it to disk for concurrent querying):
https://github.com/Morgan-Stanley/hobbes/blob/master/include...
This pipeline works well for having lightweight C++ processes feeding large volumes of data to generic query processes that we can run out of band to look at this data in various ways (with a Haskell-like query language).
We did hit a slight problem doing things this way that the straightforward representation of data (as in memory) for some cases just used too much space and too much time wasted in I/O. Basically for complex market data, where data structures aren't trivial and recording ~100GB/day makes it very awkward to keep around a few weeks of data for random querying.
So I also made this header-only lib to write data into these mmapped files with a simple compression method (I like to describe it as generalizing Curry-Howard to probabilities) that gives us much better throughput, much smaller files, faster query times, and still support concurrent constant time random access queries:
https://github.com/Morgan-Stanley/hobbes/blob/master/include...
It gives us compression ratios about the same as EOD gzip, but much faster and importantly works online and with these query use-cases we have with hobbes.
Anyway, maybe I should write up those details somewhere else, I just mean to say that this is a useful technique and you can push it very far and do many things with it in a very straightforward way.
Earlier quoted context omitted.
A library that helped manage MMAP errors would be extremely helpful.
I wanted to help address this problem with my signal sharing API proposal. Unfortunately, the glibc people have the attitude that nobody should be using signals, and so they refuse to improve the signals API at all. I strongly disagree. https://www.facebook.com/notes/daniel-colascione/toward-shar...
I read the page you linked to, and just off the top of my head, trying to manage paging by catching SIGSEGV is how do you determine that it's in response to a real bug (say, dereferencing an undefined pointer)? In my opinion, by the time you get a SIGSEGV, you can't trust the program at all. While it might be nice to have a process handle page faults itself, I think a better API than signal() is required.
Earlier quoted context omitted.
I wanted to help address this problem with my signal sharing API proposal. Unfortunately, the glibc people have the attitude that nobody should be using signals, and so they refuse to improve the signals API at all. I strongly disagree. https://www.facebook.com/notes/daniel-colascione/toward-shar...
There's a reason for that---signals are horribly abused as a general purpose signaling mechanism. Originally they were for signaling of actual problems in the code (SIGSEGV, SIGILL) but later signals were not (SIGWINCH I'm looking at you!). I read the page you linked to, and just off the top of my head, trying to manage paging by catching SIGSEGV is how do you determine that it's in response to a real bug (say, deref…
As I detailed in the doc and on libc-alpha, you really do need some kind of synchronous exception mechanism to match how real hardware behaves, and it would behoove libc authors to make this mechanism not suck instead of pretending that synchronous faults would just go away.
I wish people used mmap less. Creating a new memory mapping can be pretty expensive! On both Windows and Linux, it involves taking a process-wide reader-writer lock in exclusive mode (meaning you get to sit and wait behind page faults), doing a bunch of VMA tree manipulation work, doing various kinds of bookkeeping (hello, rmap!) and then, after you return to userspace, entering the kernel again in response to VM fau…
The cost of the second kernel trip (on the first page fault) is often mitigated by speculative read-ahead, or the fact that a given page is often in the UBC already. And file-backed memory doesn't contribute to dirty memory. And mmap() makes it easy to use read-only memory, which catches memory corruption bugs. Plus it's easier to use huge pages, which reduces TLB pressure. I <3 mmap
Do you know of a way around that, aside from building your own patched kernel?
Earlier quoted context omitted.
I wanted to help address this problem with my signal sharing API proposal. Unfortunately, the glibc people have the attitude that nobody should be using signals, and so they refuse to improve the signals API at all. I strongly disagree. https://www.facebook.com/notes/daniel-colascione/toward-shar...
There's a reason for that---signals are horribly abused as a general purpose signaling mechanism. Originally they were for signaling of actual problems in the code (SIGSEGV, SIGILL) but later signals were not (SIGWINCH I'm looking at you!). I read the page you linked to, and just off the top of my head, trying to manage paging by catching SIGSEGV is how do you determine that it's in response to a real bug (say, deref…
If you caught SIGSEGV on AIX 3.2.5 to manage a mapped NFS file, you deadlocked that filesystem. Great fun!
Thanks for the flashback!
I wish people used mmap less. Creating a new memory mapping can be pretty expensive! On both Windows and Linux, it involves taking a process-wide reader-writer lock in exclusive mode (meaning you get to sit and wait behind page faults), doing a bunch of VMA tree manipulation work, doing various kinds of bookkeeping (hello, rmap!) and then, after you return to userspace, entering the kernel again in response to VM fau…
The cost of the second kernel trip (on the first page fault) is often mitigated by speculative read-ahead, or the fact that a given page is often in the UBC already. And file-backed memory doesn't contribute to dirty memory. And mmap() makes it easy to use read-only memory, which catches memory corruption bugs. Plus it's easier to use huge pages, which reduces TLB pressure. I <3 mmap
You might appreciate this toy Go package I hacked together: https://github.com/lukechampine/freeze
It uses mmap and mprotect to "freeze" Go objects; if you try to modify a frozen object, the program crashes.