Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use mmap with care

sublimetext.com

Post reply on HN
Print view View original

51–60 of 218 posts

Re: Use mmap with care

#51

Post by planteen »

Asooka wrote: post #14

Oh I see you didn't get to caveat 5: you can't read anything more complicated than raw bytes, i.e. chars, because of unaligned memory access errors. Let's say you mmap a file and do something like this: char *fileContents=...mmap etc...; int headerOffset=*(int*)fileContents; int *someListOfNumbers=(int*)(fileContents+headerOffset); int importantSum= someListOfNumbers[0] + someListOfNumbers[1] + someListOfNumbers[2] +…

You should not directly use structs for data serialization. If you end up having to support a big endian platform, your structure orders will change, not to mention potential packing issues as you said.

https://commandcenter.blogspot.com/2012/04/byte-order-fallac...

Re: Use mmap with care

#52

Post by eeZah7Ux »

quotemstr wrote: post #23
eeZah7Ux wrote: post #7

Earlier quoted context omitted.

"In hindsight it's difficult to justify using mmap over pread" This needs a stronger justification. mmap allows reading and writing large data structures without copying, which can be a huge benefit depending on the use case.

> without copying While it's true that memory bandwidth is sometimes a limiting factor in performance, I've found that much more frequently people overestimate the cost of memory operations and don't check their estimates against benchmarks.

Indeed. It depends on the use case.

Re: Use mmap with care

#53

Post by karussell »

ysleepy wrote: post #27

Really cool talk by Bryan Cantrill about the joys of mmap in a "simple" use-case: https://m.youtube.com/watch?v=vm1GJMp0QN4#

Such a great talk - thanks a lot! Here is the direct link as it is the last of the lightning talks: https://www.youtube.com/watch?time_continue=2462&v=vm1GJMp0Q...

Re: Use mmap with care

#54

Post by eeZah7Ux »

ben-schaaf wrote: post #22
eeZah7Ux wrote: post #7

Earlier quoted context omitted.

"In hindsight it's difficult to justify using mmap over pread" This needs a stronger justification. mmap allows reading and writing large data structures without copying, which can be a huge benefit depending on the use case.

Using pread would have been more work from the start but provides a more robust solution and doesn't have problems on Windows. I would argue that the incrementaly built mmap based solution is strictly worse, thus difficult to justify doing again.

"strictly worse" despite no-copy memory access? Again, this needs proof.

Re: Use mmap with care

#55

Post by eeZah7Ux »

eeZah7Ux wrote: post #7
ben-schaaf wrote: post #2

Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.

"In hindsight it's difficult to justify using mmap over pread" This needs a stronger justification. mmap allows reading and writing large data structures without copying, which can be a huge benefit depending on the use case.

...and why the downvotes?

Re: Use mmap with care

#56

Post by ben-schaaf »

eeZah7Ux wrote: post #54
ben-schaaf wrote: post #22

Earlier quoted context omitted.

Using pread would have been more work from the start but provides a more robust solution and doesn't have problems on Windows. I would argue that the incrementaly built mmap based solution is strictly worse, thus difficult to justify doing again.

"strictly worse" despite no-copy memory access? Again, this needs proof.

Strictly worse because it requires more work to maintain and write new code, there's no guarantee we haven't missed any access points in our codebase so it is less robust, still locks files while in use on Windows and requires maintaining patches to Breakpad. Performance is not an issue here, the program working correctly is, and doing so in the long run is strictly worse than mmap.

Re: Use mmap with care

#57

Post by iainmerrick »

eeZah7Ux wrote: post #54
ben-schaaf wrote: post #22

Earlier quoted context omitted.

Using pread would have been more work from the start but provides a more robust solution and doesn't have problems on Windows. I would argue that the incrementaly built mmap based solution is strictly worse, thus difficult to justify doing again.

"strictly worse" despite no-copy memory access? Again, this needs proof.

The original post quantifies it. Around 50% better performance for the mmap version.

They are saying that if they somehow knew up front what the performance gains would be, and what the cost in bugs and complexity would be, they wouldn’t have used mmap at all.

Re: Use mmap with care

#58

Post by _wmd »

quotemstr wrote: post #25
_0w8t wrote: post #20

Earlier quoted context omitted.

Alternatively one can run a separated process that does mmap and runs the calculations or whatever that needs to access the file as quickly as possible and do the the straightforward recovery in the parent process when the child process dies. The drawback is the need to some form of RPC, but there a lot of libraries to do that without much hustle.

You can do that with a MAP_ANONYMOUS | MAP_SHARED mapping too: that kind of mapping is writable by both parent and child, but isn't backed by a disk file and so can't be truncated or surprise-removed. The article's points about mmap infelicity applies mostly to mappings of disk files. Anonymous mappings don't have the same problems.

Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances

There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid mmap whenever possible -- it's clear almost nobody fully understands it

Re: Use mmap with care

#59

Post by hyc_symas »

planteen wrote: post #51
Asooka wrote: post #14

Oh I see you didn't get to caveat 5: you can't read anything more complicated than raw bytes, i.e. chars, because of unaligned memory access errors. Let's say you mmap a file and do something like this: char *fileContents=...mmap etc...; int headerOffset=*(int*)fileContents; int *someListOfNumbers=(int*)(fileContents+headerOffset); int importantSum= someListOfNumbers[0] + someListOfNumbers[1] + someListOfNumbers[2] +…

You should not directly use structs for data serialization. If you end up having to support a big endian platform, your structure orders will change, not to mention potential packing issues as you said. https://commandcenter.blogspot.com/2012/04/byte-order-fallac...

Wasting a good performance optimization on the rare chance that you might one day have to support a different endian architecture is IMO a poor tradeoff. The number of big-endian machines in use today is continually shrinking, and the number that are active on a heterogeneous network is even smaller.

In LMDB we simply document "don't use this with remote filesystems" and avoid the issue - if you're never sharing files with other machines, there's never a question of mixed endian accessors.

Post reply on HN