Mio – Cross-platform header-only C++11 library for memory-mapped file IO • Hacker News Board

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#51

Post by quotemstr » Sat, Oct 20, 2018, 7:07 PM UTC

pradn wrote: post #35

osivertsson wrote: post #25

Earlier quoted context omitted.

Another problem with mmap is that there is no good way to handle I/O errors. The application will get a signal (SIGSEGV/SIGBUS, can't remember), and no information about what the problem could possibly be. Most applications do not catch these signals and will instead just terminate. Even if you do catch the signal there is a real challenge to know what caused the signal and to keep consistent book-keeping to be able…

A library that helped manage MMAP errors would be extremely helpful.

I wanted to help address this problem with my signal sharing API proposal. Unfortunately, the glibc people have the attitude that nobody should be using signals, and so they refuse to improve the signals API at all. I strongly disagree.

https://www.facebook.com/notes/daniel-colascione/toward-shar...

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#52

Post by gpderetta » Sat, Oct 20, 2018, 7:17 PM UTC

rwbt wrote: post #40

For those of you already using Boost, Boost also has a MMAP in their IOStreams library and it works pretty well (like most things boost). https://www.boost.org/doc/libs/1_68_0/libs/iostreams/doc/cla...

There is also boost interprocess which, despite the name, provides a very general way to access shared memory; in particular it provides shared memory allocators and a full reimplantation of the stl that can take advantage of them.

I've always wanted to try irt, but it can't handle unexpected process failure (i.e. a crashed process will leave the memory in an unknown state) which is something I always end up needing.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#53

Post by twic » Sat, Oct 20, 2018, 7:19 PM UTC

StreamBright wrote: post #45

orbifold wrote: post #31

Earlier quoted context omitted.

Yes because there is no standardised build system for C++, which makes integrating non-header only libraries a pain (or at least somewhat more painful), especially if your aim is cross platform code. In that case you can not rely on a reasonable package manager being present and will have to essentially include all your dependencies in the build, this is trivial for header only libraries.

Is there a reason not to have a standard build system?

There's no reason not to have one, but there isn't one. There are a few build systems for C++, but none of them are perfect, and none has won out.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#54

Post by bsenftner » Sat, Oct 20, 2018, 7:21 PM UTC

ajross wrote: post #42

bsenftner wrote: post #27

Earlier quoted context omitted.

MMapping is used in most fault tolerant software as a simplified method of data persistence. Granted, not the only method in play, but it is a common data safety net.

Flushing behavior is precisely the kind of thing that varies the most between systems. I'd be very suspicious of a "fault tolerant" system that tried to use a library like this to be "cross platform". That's almost a contradiction in terms.

Was not talking about that library, just memory mapped files.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#55

Post by loup-vaillant » Sat, Oct 20, 2018, 7:21 PM UTC

Why, why do they make it header only? Is it so difficult to integrate a couple source file along with the existing headers?

We should not forget compilation times. A project I use depends on spdlog, a header-only C++ logging library. The thing adds almost two seconds per compilation unit to single threaded build times. And since logging is kinda used everywhere, the whole project takes forever to build (trice the build time it would have had without spdlog, I've measured).

What benefit is so great that it is worth killing compilation times?

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#56

Post by coss » Sat, Oct 20, 2018, 7:26 PM UTC

loup-vaillant wrote: post #55

Why, why do they make it header only? Is it so difficult to integrate a couple source file along with the existing headers? We should not forget compilation times. A project I use depends on spdlog, a header-only C++ logging library. The thing adds almost two seconds per compilation unit to single threaded build times. And since logging is kinda used everywhere, the whole project takes forever to build (trice the bui…

Easy integration into projects and you can always make a precompiled header.

Personally I love being able to test something out with ease and then make it work with the build system if I like the results.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#57

Post by cleeus » Sat, Oct 20, 2018, 7:31 PM UTC

mandreyel wrote: post #46

StreamBright wrote: post #44

Earlier quoted context omitted.

And? I am not sure what is your comment about.

It's a valid point, I think. I'd probably also trust something so established as Boost more than some random guy's lib on GitHub. However, I specifically wrote mio because I prefer not to use Boost, and from what I understand, many others don't either.

boosts quality varies

I've seen parts that weren't much better then someones lib on github, because essentially that's what boost is.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#58

Post by flohofwoe » Sat, Oct 20, 2018, 7:33 PM UTC

loup-vaillant wrote: post #55

Why, why do they make it header only? Is it so difficult to integrate a couple source file along with the existing headers? We should not forget compilation times. A project I use depends on spdlog, a header-only C++ logging library. The thing adds almost two seconds per compilation unit to single threaded build times. And since logging is kinda used everywhere, the whole project takes forever to build (trice the bui…

Proper "stb-style" header-only libs split the header into 2 parts: the declaration part that's always included and parsed but only contains the public API declarations, and the implementation part inside an #ifdef XXX_IMPLEMENTATION section which is only included and parsed in a single source file.

The same idea can also be used for C++ headers (unless it's all template interfaces).

With such stb-style headers, you can even get better compile times, because you can include all header-libs into a single implementation source file, giving the same advantages as unity-builds (merging all sources into one file).

PS: origin of the name "stb-style": https://github.com/nothings/stb (although I guess the general idea existed before)

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#59

Post by bno1 » Sat, Oct 20, 2018, 7:53 PM UTC

loup-vaillant wrote: post #55

Why, why do they make it header only? Is it so difficult to integrate a couple source file along with the existing headers? We should not forget compilation times. A project I use depends on spdlog, a header-only C++ logging library. The thing adds almost two seconds per compilation unit to single threaded build times. And since logging is kinda used everywhere, the whole project takes forever to build (trice the bui…

I once tried to compile nheko, a Matrix client written in C++ and Qt. It makes heavy use of boost and compiling it crashed my system when I tried to compile using 2 jobs or more because it would eat up all my ram (6GB available at that time). Factor in the 2 seconds per compilation unit and it takes ages to compile. I managed to compile the native webrtc library with 8 jobs faster than this bare bones chat client on the same computer.

Re: Mio – Cross-platform header-only C++11 library for memory-mapped file IO

#60

Post by hyperman1 » Sat, Oct 20, 2018, 8:04 PM UTC

ajross wrote: post #3

The code seems clean. I'm not sure this is a great idea in practice, though. Generally the only good reason for mapping stuff out of the filesystem is performance, and VM behavior with mmap() varies wildly across systems (and filesystem backends, and drivers if it's a hardware device, and hardware if it's a framebuffer, and...). Frankly on windows this is AFAIK a mostly-unheard-of technique. No one does mapping. This…

> Frankly on windows this is AFAIK a mostly-unheard-of technique. No one does mapping.

This must be incorrect.

Jeffrey Richter's advanced windows, a very widespread win32 book, introduced this technique to a LOT of developers, just when windows was the hot stuff for programmers. I don't think I've seen a win32 code base >100 Kloc without mapping. Understanding and using it is one of those rites of passage that marks the switch from junior to medior win32 developer.