Use mmap with care
sublimetext.com
Use mmap with care
1–10 of 218 posts
Re: Use mmap with care
#2Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Re: Use mmap with care
#3Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Did you consider emulating mmap yourselves?
"Memory mapped files work by mapping the full file into a virtual address space and then using page faults to determine which chunks to load into physical memory. In essence it allows you to access the file as if you had read the whole thing into memory, without actually doing so."
Re: Use mmap with care
#4Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Just use signalfd and refactor into an event loop without jumping around.
Re: Use mmap with care
#5Even without NFS, using mmap requires being real careful about signals - SIGBUS can be raised any time the underlying file operation fails, including because someone else truncated the file, or because the underlying storage had an error (disk error, removed media, network storage). And, as this post so eloquently illustrates (and through my personal experience), handling SIGBUS/SIGSEGV cleanly in a multithreaded program on POSIX is incredibly painful.
Honestly, pread is just a much better solution for 90% of use cases, and it works for large files on 32-bit systems (mmap does not!). If you're doing largely sequential things, fread/fseek often work remarkably well as they handle all the caching for you.
mmap tends to shine performance-wise if you need random access to a file but access certain parts of the file frequently (for example, accessing the index in a header + contents of the file), because the page cache is literally designed for this type of usage. But the performance improvement is rarely worth the technical complexity.
Re: Use mmap with care
#6> Using setjmp and longjmping from a signal handler is actually unsafe. It seems to cause undefined behaviour, especially on MacOS.
Have you considered making a dispatch_source_t of type DISPATCH_SOURCE_TYPE_SIGNAL and handling all signals in a dispatch queue, instead of trying do figure out what kind of behavior is legal in a signal handler?
> If a library such as Breakpad registers for Mach exception messages, and handles those, it will prevent signals from being fired. This is of course at odds with our signal handling. The only workaround we've found so far involves patching Breakpad to not handle SIGBUS.
Would it be possible to install your own handler before Breakpad does?
Re: Use mmap with care
#7Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
"In hindsight it's difficult to justify using mmap over pread"
This needs a stronger justification. mmap allows reading and writing large data structures without copying, which can be a huge benefit depending on the use case.
Re: Use mmap with care
#8Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Thank you for the write-up. As an occasional scripter (a coder I am not), I found it very useful - a very nice and structured presentation of both coding practices, and of the challenges of supporting multiple environments with their own way of doing things.
Showing the scope of change within the editor is a rather nice touch. Visualization of complexity, if you will.
Re: Use mmap with care
#9Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Did you consider emulating mmap yourselves? "Memory mapped files work by mapping the full file into a virtual address space and then using page faults to determine which chunks to load into physical memory. In essence it allows you to access the file as if you had read the whole thing into memory, without actually doing so." I feel like this could be done in c++ directly, by maintaining an internal cache for each fil…
Well, then you have to implement some kind of plan for efficient caching - some kind of LRU scheme, for example, to prevent the cache from ballooning to unusable sizes - at which point you're reinventing the kernel page cache (poorly). mmap does have a big advantage here if you really need a lot of random accesses.
Re: Use mmap with care
#10That one is amazing...!