Live data from Hacker News

Mio – Cross-platform header-only C++11 library for memory-mapped file IO

github.com

11–20 of 79 posts

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#11
post #3

The code seems clean. I'm not sure this is a great idea in practice, though. Generally the only good reason for mapping stuff out of the filesystem is performance, and VM behavior with mmap() varies wildly across systems (and filesystem backends, and drivers if it's a hardware device, and hardware if it's a framebuffer, and...). Frankly on windows this is AFAIK a mostly-unheard-of technique. No one does mapping. This…

> Frankly on windows this is AFAIK a mostly-unheard-of technique. No one does mapping.

It's not that uncommon. Even notepad uses it:

https://blogs.msdn.microsoft.com/oldnewthing/20180521-00/?p=...

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#12

The name definitely made me think of Rust's mio. On the other hand, `namespace cplusplus` and `mod rust` maybe are disjoint.

This is definitely unfortunate, but in my defense I was not aware of Rust's mio (or anything related to Rust beyond its existence) at the time of writing and naming my library. I have no emotional investment in the name, so I'm open to suggestions should anyone take issue with it.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#13

Author here. Long time lurker, but made an an account now. Wow, I did not expect this. I'm really touched. I wrote this as a small utility for my own consumption because I was unsatisfied with the existing selection at the time, so I'm both surprised and delighted to learn that people are finding it useful. Although to be completely frank, I think this library is way too small and insignificant to deserve a spot on H…

Op here. Thank you for creating mio! Your project clearly deserves the attention. I just found it and thought it would belong here. A lot of people seem to share that opinion :)

I am sure this won't be the last top HN post about one of your projects.

Perfect is the enemy of good.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#14
post #3

The code seems clean. I'm not sure this is a great idea in practice, though. Generally the only good reason for mapping stuff out of the filesystem is performance, and VM behavior with mmap() varies wildly across systems (and filesystem backends, and drivers if it's a hardware device, and hardware if it's a framebuffer, and...). Frankly on windows this is AFAIK a mostly-unheard-of technique. No one does mapping. This…

> Frankly on windows this is AFAIK a mostly-unheard-of technique. No one does mapping.

Really? Most larger (Windows) codebases I know of use memory mapping.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#15
post #3

The code seems clean. I'm not sure this is a great idea in practice, though. Generally the only good reason for mapping stuff out of the filesystem is performance, and VM behavior with mmap() varies wildly across systems (and filesystem backends, and drivers if it's a hardware device, and hardware if it's a framebuffer, and...). Frankly on windows this is AFAIK a mostly-unheard-of technique. No one does mapping. This…

> "Frankly on windows this is AFAIK a mostly-unheard-of technique. No one does mapping."

Unheared by you.

In a world consumed by Electron apps and Javascript, lots of people couldn't care any less about performant IPC, sharing data across processes, and multiprocessing in general.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#16

I wish people used mmap less. Creating a new memory mapping can be pretty expensive! On both Windows and Linux, it involves taking a process-wide reader-writer lock in exclusive mode (meaning you get to sit and wait behind page faults), doing a bunch of VMA tree manipulation work, doing various kinds of bookkeeping (hello, rmap!) and then, after you return to userspace, entering the kernel again in response to VM fau…

This is a valid point. My use case was very frequent reads of large files at pretty much unpredictable positions, so in theory mmap seemed justified. However, I never got around thoroughly testing this assumption, and may indeed just have been better off using read(2) and its variants.

You seem very experienced, so I hope you don't mind a question. In my use case the files were as large as tens of gigabytes and I was creating read-only mappings of 256KB-1MB chunks in them, keeping the mmap handles around according to a cache policy and RAM usage limit. Do you think in this case using mmap could in theory introduce performance gains?

[edit: typo]

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#17

I wish people used mmap less. Creating a new memory mapping can be pretty expensive! On both Windows and Linux, it involves taking a process-wide reader-writer lock in exclusive mode (meaning you get to sit and wait behind page faults), doing a bunch of VMA tree manipulation work, doing various kinds of bookkeeping (hello, rmap!) and then, after you return to userspace, entering the kernel again in response to VM fau…

If you are reading large chunks of data at once - and you should - then the same process happens, but it is hidden inside the memory manager.

I, for one, used memory mapping in the past to significantly speed up code that operates on gigantic data sets. I guess that your mileage does vary.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#18

I wish people used mmap less. Creating a new memory mapping can be pretty expensive! On both Windows and Linux, it involves taking a process-wide reader-writer lock in exclusive mode (meaning you get to sit and wait behind page faults), doing a bunch of VMA tree manipulation work, doing various kinds of bookkeeping (hello, rmap!) and then, after you return to userspace, entering the kernel again in response to VM fau…

This is a valid point. My use case was very frequent reads of large files at pretty much unpredictable positions, so in theory mmap seemed justified. However, I never got around thoroughly testing this assumption, and may indeed just have been better off using read(2) and its variants. You seem very experienced, so I hope you don't mind a question. In my use case the files were as large as tens of gigabytes and I was…

I think that this is the wrong way to use mmap. Just map the whole file at once. The operating system will automatically read the pages you access from disk. And if memory gets tight, these pages will be flushed to disk if they are dirty and then discarded before the system starts paging. These mmapped pages essentially live in the disk cache.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#19

I wish people used mmap less. Creating a new memory mapping can be pretty expensive! On both Windows and Linux, it involves taking a process-wide reader-writer lock in exclusive mode (meaning you get to sit and wait behind page faults), doing a bunch of VMA tree manipulation work, doing various kinds of bookkeeping (hello, rmap!) and then, after you return to userspace, entering the kernel again in response to VM fau…

This is a valid point. My use case was very frequent reads of large files at pretty much unpredictable positions, so in theory mmap seemed justified. However, I never got around thoroughly testing this assumption, and may indeed just have been better off using read(2) and its variants. You seem very experienced, so I hope you don't mind a question. In my use case the files were as large as tens of gigabytes and I was…

Random access to large files is a legitimate use case! LMDB [1] uses a similar technique, and it works well for them. But depending on the specific application, explicitly application-managed caching via O_DIRECT IO with something like threaded pread or AIO might be even better, because with this explicit model, you control the cache sizing and eviction policy, and it's certainly possible with application-level knowledge to do better than the generic kernel-level LRU/active/inactive/kinda-sorta-works-heuristic stuff can do without application-specific knowledge.

Another advantage of using application-managed caching is the ability to take advantage of things like huge pages (which can drastically reduce TLB miss rates), whereas with conventional mmap of conventional files, you're limited to regular 4kB (or whatever) small pages and associated management overhead. (There's no reason in principle filesystems can't use huge pages for page cache, but AFAIK, nobody does it yet.)

OTOH, kernel management of page cache allows for better integration of cache eviction with system memory pressure signals and allows for multiple users of a single file to share the memory mirroring the contents of that file.

> Do you think in this case using mmap could in theory introduce performance gains?

It depends. The right approach depends on a lot of factors, including workload and developer complexity budget. It's funny, really: the more experience you get, the less likely you are to say "$SOLUTION is the bestest evar!" and the more often you say "well, it really depends, so I can't give you an answer".

What really strikes me as needless is someone using mmap to read a 10kB ~/.myapplication.lol.ini file or something.

[1] http://www.lmdb.tech/doc/

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#20
post #18

Earlier quoted context omitted.

This is a valid point. My use case was very frequent reads of large files at pretty much unpredictable positions, so in theory mmap seemed justified. However, I never got around thoroughly testing this assumption, and may indeed just have been better off using read(2) and its variants. You seem very experienced, so I hope you don't mind a question. In my use case the files were as large as tens of gigabytes and I was…

I think that this is the wrong way to use mmap. Just map the whole file at once. The operating system will automatically read the pages you access from disk. And if memory gets tight, these pages will be flushed to disk if they are dirty and then discarded before the system starts paging. These mmapped pages essentially live in the disk cache.

> The operating system will automatically read the pages you access from disk. And if memory gets tight, these pages will be flushed to disk if they are dirty and then discarded before the system starts paging.

You can tell that you understand how modern OS memory management works when you realize that the OS "automatically read[ing] the pages...from disk" and "flush[ing them] to disk" on memory pressure is paging whether those pages are anonymous pages or mmaped file pages. :-)

[Edit: flushing dirty file-paged pages is analogous to swapping anonymous memory to the swapfile. Discarding clean file-backed pages is a bit like discarding anonymous pages that have been made unused through munmap, process death, etc.]

But to the GP's point: you don't need (except to conserve address space) to limit file mapping size. I think he really wants something like MADV_FREE. But it's complicated.

Post reply on HN