Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mio – Cross-platform header-only C++11 library for memory-mapped file IO

github.com

Post reply on HN
Print view View original

21–30 of 79 posts

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#21

Post by gmueckl »

quotemstr wrote: post #20
gmueckl wrote: post #18

Earlier quoted context omitted.

I think that this is the wrong way to use mmap. Just map the whole file at once. The operating system will automatically read the pages you access from disk. And if memory gets tight, these pages will be flushed to disk if they are dirty and then discarded before the system starts paging. These mmapped pages essentially live in the disk cache.

> The operating system will automatically read the pages you access from disk. And if memory gets tight, these pages will be flushed to disk if they are dirty and then discarded before the system starts paging. You can tell that you understand how modern OS memory management works when you realize that the OS "automatically read[ing] the pages...from disk" and "flush[ing them] to disk" on memory pressure is paging wh…

Therr is a subtle difference between anonymous and mapped readonly pages: the later can be discarded right away because their contents were read from permanemt storage to begin with. Anonymous pages need to be written to disk first and that is significantly slower.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#22

Post by mandreyel »

gmueckl wrote: post #18
mandreyel wrote: post #16

Earlier quoted context omitted.

This is a valid point. My use case was very frequent reads of large files at pretty much unpredictable positions, so in theory mmap seemed justified. However, I never got around thoroughly testing this assumption, and may indeed just have been better off using read(2) and its variants. You seem very experienced, so I hope you don't mind a question. In my use case the files were as large as tens of gigabytes and I was…

I think that this is the wrong way to use mmap. Just map the whole file at once. The operating system will automatically read the pages you access from disk. And if memory gets tight, these pages will be flushed to disk if they are dirty and then discarded before the system starts paging. These mmapped pages essentially live in the disk cache.

Hmm, I see, I haven't come across this suggestion but I may not have researched this topic well enough. I'll try your suggestion next time.

I appreciate the responses--learned something new today!

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#23

Post by CyberDildonics »

mandreyel wrote: post #8

Author here. Long time lurker, but made an an account now. Wow, I did not expect this. I'm really touched. I wrote this as a small utility for my own consumption because I was unsatisfied with the existing selection at the time, so I'm both surprised and delighted to learn that people are finding it useful. Although to be completely frank, I think this library is way too small and insignificant to deserve a spot on H…

I like and took a look at the github. I think there is actually a missed opportunity here to make it a single header library since there seem to only be 4 or 5 files that go into it.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#24

Post by jnordwick »

gmueckl wrote: post #18
mandreyel wrote: post #16

Earlier quoted context omitted.

This is a valid point. My use case was very frequent reads of large files at pretty much unpredictable positions, so in theory mmap seemed justified. However, I never got around thoroughly testing this assumption, and may indeed just have been better off using read(2) and its variants. You seem very experienced, so I hope you don't mind a question. In my use case the files were as large as tens of gigabytes and I was…

I think that this is the wrong way to use mmap. Just map the whole file at once. The operating system will automatically read the pages you access from disk. And if memory gets tight, these pages will be flushed to disk if they are dirty and then discarded before the system starts paging. These mmapped pages essentially live in the disk cache.

Yeah +1 for someone who understands mmap.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#25

Post by osivertsson »

quotemstr wrote: post #10

I wish people used mmap less. Creating a new memory mapping can be pretty expensive! On both Windows and Linux, it involves taking a process-wide reader-writer lock in exclusive mode (meaning you get to sit and wait behind page faults), doing a bunch of VMA tree manipulation work, doing various kinds of bookkeeping (hello, rmap!) and then, after you return to userspace, entering the kernel again in response to VM fau…

Another problem with mmap is that there is no good way to handle I/O errors.

The application will get a signal (SIGSEGV/SIGBUS, can't remember), and no information about what the problem could possibly be. Most applications do not catch these signals and will instead just terminate.

Even if you do catch the signal there is a real challenge to know what caused the signal and to keep consistent book-keeping to be able to perform any sane action in response.

At a previous employer we started seeing this problem when scaling things in production which was no fun.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#26

Post by mandreyel »

starbugs wrote: post #13
mandreyel wrote: post #8

Author here. Long time lurker, but made an an account now. Wow, I did not expect this. I'm really touched. I wrote this as a small utility for my own consumption because I was unsatisfied with the existing selection at the time, so I'm both surprised and delighted to learn that people are finding it useful. Although to be completely frank, I think this library is way too small and insignificant to deserve a spot on H…

Op here. Thank you for creating mio! Your project clearly deserves the attention. I just found it and thought it would belong here. A lot of people seem to share that opinion :) I am sure this won't be the last top HN post about one of your projects. Perfect is the enemy of good.

Thank you for the kind words. I'll buy you a beer if we ever meet!

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#27

Post by bsenftner »

ajross wrote: post #3

The code seems clean. I'm not sure this is a great idea in practice, though. Generally the only good reason for mapping stuff out of the filesystem is performance, and VM behavior with mmap() varies wildly across systems (and filesystem backends, and drivers if it's a hardware device, and hardware if it's a framebuffer, and...). Frankly on windows this is AFAIK a mostly-unheard-of technique. No one does mapping. This…

MMapping is used in most fault tolerant software as a simplified method of data persistence. Granted, not the only method in play, but it is a common data safety net.
Post reply on HN