Note that some BSDs have a MAP_ZERO flag which makes invalid accesses read zeros instead of triggering SIGBUS.
Which ones? I don't see it in manual pages for any of the ones I know about (Free, Net, Open, Dragonfly).
Use mmap with care
151–160 of 218 posts
Re: Use mmap with care
#152Honestly, this reads like a thorough indictment of signals in user space. * Signal handlers are process global * Signal handlers need to be re-entrant safe Re-entrancy is painful but can be done, but process-global signal handlers means that pulling in a totally unrelated library can break your code. Moreover, it makes the combined use of certain libraries straight-up impossible. Similarly, it means that the use of l…
The real issue is that people are using threads for things that processes were originally intended for. Having one thread for the UI, one thread for network code, one thread for program logic, etc. is actually a mis-use of threads in POSIX-land. POSIX is designed for you to use multiple processes where in Windows you would use multiple threads. This gets you a lot of robust interprocess communication mechanisms and a…
It's much more feasible to fix the signals API to work well in a multithreaded world than to roll back the clock to the 1980s and get people to stop using threads.
Re: Use mmap with care
#153Earlier quoted context omitted.
As if "avoid virtual memory" is substantially more of a "sweeping generalization" than "avoid mmap." If you apply the same reasoning that you've used to conclude that everyone should avoid mmap, than you are led directly to the conclusion that everyone should avoid virtual memory. The "same problems" that you are pointing out are possible with anonymous memory aren't unique to memory you get directly from mmap, they…
Let's try and simplify things here: the post we're both currently commenting on relates to doing file IO via nmap . Of course 'avoiding all use of virtual memory' is ridiculous, but nobody except you is suggesting that, and you continue to suggest it even after a long reply. The "powerful (and consequently hazardous) OS feature" here is using mmap for general file IO , it: - introduces resources leaks many developers…
> nobody except you is suggesting [avoiding all use of virtual memory]
https://news.ycombinator.com/item?id=19807322
> Anonymous mappings
> typical apps should avoid mmap whenever possible
All virtual memory is either a user-mode wrapper around mmap, or sbrk (which is functionally a kernel-mode wrapper around mmap).
Re: Use mmap with care
#154Oh I see you didn't get to caveat 5: you can't read anything more complicated than raw bytes, i.e. chars, because of unaligned memory access errors. Let's say you mmap a file and do something like this: char *fileContents=...mmap etc...; int headerOffset=*(int*)fileContents; int *someListOfNumbers=(int*)(fileContents+headerOffset); int importantSum= someListOfNumbers[0] + someListOfNumbers[1] + someListOfNumbers[2] +…
It's invalid for the compiler to vectorize scalar operations with unknown alignment if those vector operations require alignment. (That said, on x86 many AVX operations work fine unaligned, and are just as fast as the aligned versions on recent microarchitectures.)
Re: Use mmap with care
#155The mistake here is using longjmp / siglongjmp. This is a possible way to handle SIGBUS, but in practice it will be intractable in larger programs written in C or C++. The compiler is generally free to move loads and stores around, and you might be completely blindsided by how the compiler has reordered your memory operations once you add side effects to one of the operations. Theoretically, if accessing a memory loc…
I solved the problem this way recently as well. This solution would also work with multiple threads (you can probably set the flag in thread local storage, though I haven't done that yet - currently using a global map with thread id as key). Though still not with multiple different signal handlers. That said, if I could do it over I wouldn't use mmap again. Especially since io_uring is around the corner (on Linux) th…
Thread-local storage is not async-signal safe. Some systems (including glibc!) will lazily initialize TLS slots on the first access, which would result in a call to malloc and/or locking of a mutex, neither of which are async-signal safe. Some slots may be initialized at link time but others lazily, depending on how and when the code was compiled and loaded, so just because it seems to work for you doesn't mean the behavior won't change.
The safest (and also portable) way I've found to setup per-thread async-signal-safe local storage is to install an alternate signal stack for each thread using sigaltstack. Allocate a larger buffer than needed (and reported) for the stack and use the remaining memory for storage and guard pages. For example, allocate ((PAGE_SIZE * 3) + roundup(MINSIGSTKSZ, PAGE_SIZE)--two guard pages, one page for your local storage, and the remainder for the stack.
Re: Use mmap with care
#156Earlier quoted context omitted.
Let's try and simplify things here: the post we're both currently commenting on relates to doing file IO via nmap . Of course 'avoiding all use of virtual memory' is ridiculous, but nobody except you is suggesting that, and you continue to suggest it even after a long reply. The "powerful (and consequently hazardous) OS feature" here is using mmap for general file IO , it: - introduces resources leaks many developers…
> nobody except you is suggesting [avoiding all use of virtual memory] https://news.ycombinator.com/item?id=19807322 > Anonymous mappings > typical apps should avoid mmap whenever possible All virtual memory is either a user-mode wrapper around mmap, or sbrk (which is functionally a kernel-mode wrapper around mmap).
This simply won't die, will it? I mean, while we're at it, let's advocate abandonment of all higher level languages because essentially they all boil down to machine code, and nobody could recommend working directly with machine code any more, could they.
(But that would be a sweeping generalization)
Re: Use mmap with care
#157Earlier quoted context omitted.
It would be better to mmap the file into N single-threaded processes instead of mmapping the file into 1 N-threaded process. This is exactly why signals are process-global instead of thread-local. The intent is that they were used for inter-process communication. The whole notion of processes and signals predates the notion of a thread, and threads are essentially a performance optimization for 30 or 40 year old hard…
>This is exactly why signals are process-global instead of thread-local. Signals can be sent either to a process or to a specific thread under linux. Signals sent to the process are handled by an arbitrary thread. There's a bunch of gotchas and whatnot past that. Mixing signals and threads is pretty much trash.
> Mixing signals and threads is pretty much trash.
Agreed, except for SIGSEGV, SIGBUS, and SIGPIPE, which are guaranteed to be delivered, if at all, to the thread which triggered the condition. (Unless, of course, they were raised explicitly.) Still trash in the sense that to do it right you're depending on installing global signal handlers and (very likely) arranging for thread-local storage (actual TLS isn't even async-signal safe; see my post elsethread regarding the sigaltstack trick), so these tricks are really only practical for the core application and not something you can stash in a library to perform transparently.
FWIW, none of this behavior is specific to Linux. Linux follows POSIX rather well in this regard.
Re: Use mmap with care
#158Earlier quoted context omitted.
As if "avoid virtual memory" is substantially more of a "sweeping generalization" than "avoid mmap." If you apply the same reasoning that you've used to conclude that everyone should avoid mmap, than you are led directly to the conclusion that everyone should avoid virtual memory. The "same problems" that you are pointing out are possible with anonymous memory aren't unique to memory you get directly from mmap, they…
Let's try and simplify things here: the post we're both currently commenting on relates to doing file IO via nmap . Of course 'avoiding all use of virtual memory' is ridiculous, but nobody except you is suggesting that, and you continue to suggest it even after a long reply. The "powerful (and consequently hazardous) OS feature" here is using mmap for general file IO , it: - introduces resources leaks many developers…
> here is using mmap for general file IO
> nobody has ever suggested avoiding virtual memory except you
Do you know what the "anonymous" in "anonymous mapping" means? You are the one that started asserting that anonymous mapping from mmap have the same difficulties as mapping of normal files and therefore too dangerous to use.
Re: Use mmap with care
#159Earlier quoted context omitted.
"strictly worse" despite no-copy memory access? Again, this needs proof.
The original post quantifies it. Around 50% better performance for the mmap version. They are saying that if they somehow knew up front what the performance gains would be, and what the cost in bugs and complexity would be, they wouldn’t have used mmap at all.
"some quick benchmarks for the way Sublime Merge reads git object files" is in not strong evidence.
Re: Use mmap with care
#160Earlier quoted context omitted.
> nobody except you is suggesting [avoiding all use of virtual memory] https://news.ycombinator.com/item?id=19807322 > Anonymous mappings > typical apps should avoid mmap whenever possible All virtual memory is either a user-mode wrapper around mmap, or sbrk (which is functionally a kernel-mode wrapper around mmap).
This simply won't die, will it? I mean, while we're at it, let's advocate abandonment of all higher level languages because essentially they all boil down to machine code, and nobody could recommend working directly with machine code any more, could they. (But that would be a sweeping generalization)
> This simply won't die, will it?
Please clarify how you think that statement is incorrect. As far as I'm concerned, it won't die because that's how virtual memory works, and you have something going on in your head that is either wrong or massive hairsplitting.
I would have expected a better understanding from someone bloviating about how the try of the commenters are too thick to understand mmap and virtual memory.