I feel like you should use some kind of library for this instead of handling it all yourself.
Use mmap with care
11–20 of 218 posts
Re: Use mmap with care
#12How widely-used code laying on mmap (LMDB maybe being one of the prominent cases) rolls?
Re: Use mmap with care
#13Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Just use signalfd and refactor into an event loop without jumping around.
This does not work - signalfd cannot catch synchronous signals, per the man page:
> Limitations
> The signalfd mechanism can't be used to receive signals that are synchronously generated, such as the SIGSEGV signal
This is because synchronous signals are fired at the thread that caused them, and signalfd read() calls can't be used to read signals fired at other threads.
Re: Use mmap with care
#14Oh I see you didn't get to caveat 5: you can't read anything more complicated than raw bytes, i.e. chars, because of unaligned memory access errors. Let's say you mmap a file and do something like this:
char *fileContents=...mmap etc...;
int headerOffset=*(int*)fileContents;
int *someListOfNumbers=(int*)(fileContents+headerOffset);
int importantSum=
someListOfNumbers[0] +
someListOfNumbers[1] +
someListOfNumbers[2] +
someListOfNumbers[3];
[1] http://pzemtsov.github.io/2016/11/06/bug-story-alignment-on-...
Re: Use mmap with care
#15Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
"In hindsight it's difficult to justify using mmap over pread" This needs a stronger justification. mmap allows reading and writing large data structures without copying, which can be a huge benefit depending on the use case.
I think it’s reasonable to say “it’s difficult to justify using mmap when other programs are expected to manipulate, and potentially even delete, the file while you are working with it.” But, honestly, that case will always be hard to handle, and mmap doesn’t make it any worse.
Re: Use mmap with care
#16Oh I see you didn't get to caveat 5: you can't read anything more complicated than raw bytes, i.e. chars, because of unaligned memory access errors. Let's say you mmap a file and do something like this: char *fileContents=...mmap etc...; int headerOffset=*(int*)fileContents; int *someListOfNumbers=(int*)(fileContents+headerOffset); int importantSum= someListOfNumbers[0] + someListOfNumbers[1] + someListOfNumbers[2] +…
Why not use a memcpy here and not rely on undefined behavior? I’m sure the compiler will optimize out the call anyways, especially if you enforce alignment.
Re: Use mmap with care
#17Earlier quoted context omitted.
"In hindsight it's difficult to justify using mmap over pread" This needs a stronger justification. mmap allows reading and writing large data structures without copying, which can be a huge benefit depending on the use case.
I think it’s reasonable to say “it’s difficult to justify using mmap when other programs are expected to manipulate, and potentially even delete, the file while you are working with it.” But, honestly, that case will always be hard to handle, and mmap doesn’t make it any worse.
I think the problem here is that mmap does make it worse - with (p)read, if you get an errno, you flag an error. with mmap, you have to handle a signal and jump back to an appropriate point in the code (and you can't do this in a cross-platform way), and then flag an error. Obviously, what the upper levels do with the error still has to be worked out, but you can hardly argue that sigsetjmp/siglongjmp + signal handlers + Windows SEH handlers is no worse than "ret = read(...); if (ret < 0) ERROR;"
Re: Use mmap with care
#18Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Did you consider emulating mmap yourselves? "Memory mapped files work by mapping the full file into a virtual address space and then using page faults to determine which chunks to load into physical memory. In essence it allows you to access the file as if you had read the whole thing into memory, without actually doing so." I feel like this could be done in c++ directly, by maintaining an internal cache for each fil…
It’s easy enough to read a file in chunks, parsing out the information as you go. This limits memory use as long as you release the chunks when you no longer need them. The operating system can swap out memory as-needed, even if you didn’t get the memory from mmap, so it’s irrelevant where you store the parsed data.
Unless you actually need to read the file multiple times (compared to looking at the parsed in-memory data multiple times), this should be fast enough.
Re: Use mmap with care
#19I feel like you should use some kind of library for this instead of handling it all yourself.
Which library would you propose?
Re: Use mmap with care
#20Even without NFS, using mmap requires being real careful about signals - SIGBUS can be raised any time the underlying file operation fails, including because someone else truncated the file, or because the underlying storage had an error (disk error, removed media, network storage). And, as this post so eloquently illustrates (and through my personal experience), handling SIGBUS/SIGSEGV cleanly in a multithreaded pro…
Alternatively one can run a separated process that does mmap and runs the calculations or whatever that needs to access the file as quickly as possible and do the the straightforward recovery in the parent process when the child process dies. The drawback is the need to some form of RPC, but there a lot of libraries to do that without much hustle.