Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mio – Cross-platform header-only C++11 library for memory-mapped file IO

github.com

Post reply on HN
Print view View original

71–79 of 79 posts

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#71

Post by tjalfi »

loup-vaillant wrote: post #55

Why, why do they make it header only? Is it so difficult to integrate a couple source file along with the existing headers? We should not forget compilation times. A project I use depends on spdlog, a header-only C++ logging library. The thing adds almost two seconds per compilation unit to single threaded build times. And since logging is kinda used everywhere, the whole project takes forever to build (trice the bui…

Here is why I used a header only library for my last C++ project.

Earlier this year I wrote a Win32 program to solve an obscure issue for my employer.

My main concern was making it possible for programmers with limited knowledge of C++ to maintain the application. We are a C# shop and I want my coworkers to be able to maintain the application after I leave.

Here are some of the other things I did to make life easier for the maintenance programmer.

I documented how the program works, what APIs it calls, and added links to Pluralsight courses on ATL and COM. The program uses printf format strings so there is a link to a tutorial on them.

I added extensive logging with line numbers for every action that the program takes. You can reconstruct the control flow by looking at the log file. Yes, I tried doing this.

I picked a header only logging library so that it would be easier for a maintenance programmer to update it. All they have to do is update the submodule that contains the library.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#72

Post by gok »

nemo1618 wrote: post #70
gok wrote: post #37

Earlier quoted context omitted.

The cost of the second kernel trip (on the first page fault) is often mitigated by speculative read-ahead, or the fact that a given page is often in the UBC already. And file-backed memory doesn't contribute to dirty memory. And mmap() makes it easy to use read-only memory, which catches memory corruption bugs. Plus it's easier to use huge pages, which reduces TLB pressure. I <3 mmap

>mmap() makes it easy to use read-only memory, which catches memory corruption bugs You might appreciate this toy Go package I hacked together: https://github.com/lukechampine/freeze It uses mmap and mprotect to "freeze" Go objects; if you try to modify a frozen object, the program crashes.

Nice!

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#73

Post by gok »

jzwinck wrote: post #67
gok wrote: post #37

Earlier quoted context omitted.

The cost of the second kernel trip (on the first page fault) is often mitigated by speculative read-ahead, or the fact that a given page is often in the UBC already. And file-backed memory doesn't contribute to dirty memory. And mmap() makes it easy to use read-only memory, which catches memory corruption bugs. Plus it's easier to use huge pages, which reduces TLB pressure. I <3 mmap

Memory mapped I/O actually prevents the use of huge pages if the backing storage is an actual disk. See https://lwn.net/Articles/718102/ Do you know of a way around that, aside from building your own patched kernel?

Yeah you need to move your files to tmpfs, but I still consider that easier than dealing with making it happen with malloc+read

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#74

Post by __sr__ »

osivertsson wrote: post #25
quotemstr wrote: post #10

I wish people used mmap less. Creating a new memory mapping can be pretty expensive! On both Windows and Linux, it involves taking a process-wide reader-writer lock in exclusive mode (meaning you get to sit and wait behind page faults), doing a bunch of VMA tree manipulation work, doing various kinds of bookkeeping (hello, rmap!) and then, after you return to userspace, entering the kernel again in response to VM fau…

Another problem with mmap is that there is no good way to handle I/O errors. The application will get a signal (SIGSEGV/SIGBUS, can't remember), and no information about what the problem could possibly be. Most applications do not catch these signals and will instead just terminate. Even if you do catch the signal there is a real challenge to know what caused the signal and to keep consistent book-keeping to be able…

I was burnt by a third party application using mmap over NFS recently. Definitely not fun by any definition of it.

And it gets a SIGSEGV and segfaults.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#75

Post by Fronzie »

yread wrote: post #28

not a C++ person here but is header-only library an advantage?

Setting up the dependency infrastructure is a bit of work in C++. Recently things have been moving forward, with tools like Cmake+Conan or vcpkg becoming more popular.

Once the infrastructure is sorted out, header-only has mainly disadvantages, like making it harder to separate interface from implementation.

There's a good talk on the topic: https://youtu.be/sBP17HQAQjk

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#76

Post by asveikau »

orbifold wrote: post #31
yread wrote: post #28

not a C++ person here but is header-only library an advantage?

Yes because there is no standardised build system for C++, which makes integrating non-header only libraries a pain (or at least somewhat more painful), especially if your aim is cross platform code. In that case you can not rely on a reasonable package manager being present and will have to essentially include all your dependencies in the build, this is trivial for header only libraries.

Overstated problem. It's not that hard under most circumstances to build a static lib.

I think there are two issues. In C++, a header only library makes sense due to templates, which don't produce object code before specialization. In C, I have seen the same with entirely macro-based "libraries" that are really simulating templates in order to make abstract data structures.

The second case though, my theory is that there is a new generation of programmers who don't understand the traditional compile and link phases because they are used to other languages. It doesn't fit their expectation of how it should work so they bend the tool to their expectation instead of figuring out the old way. The very fact that people on here are saying a programming language should have a "standard package manager" demonstrates the cultural divide: this sounds a little nutty to an old time C person.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#77

Post by loup-vaillant »

ur-whale wrote: post #61
loup-vaillant wrote: post #55

Why, why do they make it header only? Is it so difficult to integrate a couple source file along with the existing headers? We should not forget compilation times. A project I use depends on spdlog, a header-only C++ logging library. The thing adds almost two seconds per compilation unit to single threaded build times. And since logging is kinda used everywhere, the whole project takes forever to build (trice the bui…

Doesn't your compiler support pre-compiled headers?

The "rebuild from scratch" integration server does not.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#78

Post by anonknowsaguy »

asveikau wrote: post #76
orbifold wrote: post #31

Earlier quoted context omitted.

Yes because there is no standardised build system for C++, which makes integrating non-header only libraries a pain (or at least somewhat more painful), especially if your aim is cross platform code. In that case you can not rely on a reasonable package manager being present and will have to essentially include all your dependencies in the build, this is trivial for header only libraries.

Overstated problem. It's not that hard under most circumstances to build a static lib. I think there are two issues. In C++, a header only library makes sense due to templates, which don't produce object code before specialization. In C, I have seen the same with entirely macro-based "libraries" that are really simulating templates in order to make abstract data structures. The second case though, my theory is that t…

As newer generation C++ programmer, can confirm I'm guilty of writing things header only when they "probably shouldn't be". But it's not so much I don't understand the old way. Having everything in one file is just so much easier to work with. The price you pay for it is relatively low(for small projects like this library).

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#79

Post by chris_wat »

chris_wot wrote: post #41

I wonder if this could replace the cross platform memory mapping in LibreOffice. This is part of the Operating System Layer (OSL) which is at least several decades old and uses a C interface. https://opengrok.libreoffice.org/xref/core/include/osl/file....

That would not be a good idea at all. The memory mapping works cross platform to prevent an unsubstantiated configuration of the elon-burrow mechanisms.
Post reply on HN