Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use mmap with care

sublimetext.com

Post reply on HN
Print view View original

161–170 of 218 posts

Re: Use mmap with care

#161

Post by wahern »

loeg wrote: post #147
qiqitori wrote: post #91

Earlier quoted context omitted.

Normal libraries should never register signal handlers. Google Breakpad is a crash-reporting system and as such requires signal handling to function. Also since we're on the topic, here's a nice vulnerability caused by bad signal handling: https://news.ycombinator.com/item?id=16753013 Pretty sure there are no plans to replace signals, but maybe there are libraries that make signal handling easier?

> Pretty sure there are no plans to replace signals, but maybe there are libraries that make signal handling easier? Non-portable, but sigprocmask() SIG_BLOCK plus signalfd() (Linux) or kqueue() EVFILT_SIGNAL (BSDs). Neither is a good solution for handling mmap SIGBUSes, but they're generally good for handling most signals (USRn, TERM, HUP, CHLD, etc) more similarly to other kinds of events.

signalfd does NOT work the same as kqueue EVFILT_SIGNAL. On Linux signalfd basically behaves the same as installing a signal handler--only a single signalfd object will receive a signal, so whether installing a signal handler or a signalfd listener you're changing behavior globally. It's even messier than that, because signalfd only works if there are no other signal handlers installed.

By contrast, with kqueue all EVFILT_SIGNAL listeners will be notified of a signal, even if the signal was also delivered to a handler. Big difference.

This is one among many reasons why Linux's event syscalls are widely considered inferior to BSD kqueue. It's ridiculous considering that kqueue not only predated epoll by several years and signalfd by nearly a decade, with ample use cases, it was well documented in a paper that described the rationale for all the semantics. Somehow the authors of epoll, signalfd, etc couldn't even bothered do to basic research; they just spitballed semantics without having much real-world experience about what was needed and most useful.

Re: Use mmap with care

#162

Post by _wmd »

tntn wrote: post #158
_wmd wrote: post #134

Earlier quoted context omitted.

Let's try and simplify things here: the post we're both currently commenting on relates to doing file IO via nmap . Of course 'avoiding all use of virtual memory' is ridiculous, but nobody except you is suggesting that, and you continue to suggest it even after a long reply. The "powerful (and consequently hazardous) OS feature" here is using mmap for general file IO , it: - introduces resources leaks many developers…

> here is using mmap for general file IO > nobody has ever suggested avoiding virtual memory except you Do you know what the "anonymous" in "anonymous mapping" means? You are the one that started asserting that anonymous mapping from mmap have the same difficulties as mapping of normal files and therefore too dangerous to use. https://news.ycombinator.com/item?id=19807322

I hate to do this, but:

> paragraph, n.: a distinct section of a piece of writing, usually dealing with a single theme and indicated by a new line, indentation, or numbering

In the original comment you will find two of these, the former correcting an error in the parent comment, the latter making an observation based on the obvious brainwrong riddled throughout this thread

I'm done replying, you're of course free to continue checking in hazardous and suspect file IO code, as the rest of us are free to giggle at such things before ripping them out

Re: Use mmap with care

#163

Post by [deleted] »

dullgiulio wrote: post #96
rocqua wrote: post #86

Earlier quoted context omitted.

Vim needs to do a lot more special stuff, because it has to handle random insertions and deletions. Doing those things to a file (mmaped or otherwise) requires moving all data after the edit. If I recall correctly, vim/vi uses a linked list of 'chunks' that is dynamically merged. For long files I would expect some form of lazy loading of chunks.

That's right. The key data structure is the rope: https://en.wikipedia.org/wiki/Rope_(data_structure)

[deleted]

Re: Use mmap with care

#164

Post by rocqua »

jschwartzi wrote: post #105
rocqua wrote: post #84

Honestly, this reads like a thorough indictment of signals in user space. * Signal handlers are process global * Signal handlers need to be re-entrant safe Re-entrancy is painful but can be done, but process-global signal handlers means that pulling in a totally unrelated library can break your code. Moreover, it makes the combined use of certain libraries straight-up impossible. Similarly, it means that the use of l…

The real issue is that people are using threads for things that processes were originally intended for. Having one thread for the UI, one thread for network code, one thread for program logic, etc. is actually a mis-use of threads in POSIX-land. POSIX is designed for you to use multiple processes where in Windows you would use multiple threads. This gets you a lot of robust interprocess communication mechanisms and a…

This seems like an ivory tower argument, people shouldn't want shared mutable state, so anyone suffering is doing so at their own hand.

At the same time, shared mutable state, specifically a fully shared and transparent (ignoring caches) address space is the least effort way to take advantage of multi-core CPUs. Hence, people will be using it. There is essentially no getting around that, and there are even some reasons for wanting it.

Being multi-process would solve a lot of issues, but so would settling on a single convention for endianness, rewriting C++ to use unique_ and shared_ptr where applicable, and many other nice to haves.

At the end of the day, the easiest road is going to be taken more often. If that happens road is lined with bandits and barely visible traps, that is a problem. No matter how often you tell people not to take that road, and climb a slippery mountain path instead.

Re: Use mmap with care

#165

Post by rocqua »

beat wrote: post #114
rocqua wrote: post #84

Honestly, this reads like a thorough indictment of signals in user space. * Signal handlers are process global * Signal handlers need to be re-entrant safe Re-entrancy is painful but can be done, but process-global signal handlers means that pulling in a totally unrelated library can break your code. Moreover, it makes the combined use of certain libraries straight-up impossible. Similarly, it means that the use of l…

Signals are fundamental to the unix design. That'd be like "I want a car, but without any wheels, because wheels are often implicated in crashes". Depending on OS internals within threading is a problem. Libraries that implement their own signal handling are a problem. Trying to get thread-local signal handling behavior when signals are process-global is a problem. Threads are not a wholesale substitute for processes…

A feature that

1. cannot be used if we compose two systems that use it 2. precludes using other mainstream features 3. is an essential part of the interface between userspace and kernelspace

is a bad feature. I am not saying we should drop the idea of allowing userspace to respond to things the kernel raises. I am saying we need a way that will allow 1. program composition and 2. play nice with threads.

I am not a unix/linux expert, but those requirements seem like they could be met.

Re: Use mmap with care

#166

Post by beat »

rocqua wrote: post #165
beat wrote: post #114

Earlier quoted context omitted.

Signals are fundamental to the unix design. That'd be like "I want a car, but without any wheels, because wheels are often implicated in crashes". Depending on OS internals within threading is a problem. Libraries that implement their own signal handling are a problem. Trying to get thread-local signal handling behavior when signals are process-global is a problem. Threads are not a wholesale substitute for processes…

A feature that 1. cannot be used if we compose two systems that use it 2. precludes using other mainstream features 3. is an essential part of the interface between userspace and kernelspace is a bad feature. I am not saying we should drop the idea of allowing userspace to respond to things the kernel raises. I am saying we need a way that will allow 1. program composition and 2. play nice with threads. I am not a un…

Signals play nice with threads.

It's sloppy thread programming that doesn't play nice with signals.

Re: Use mmap with care

#167

Post by rocqua »

qiqitori wrote: post #91
rocqua wrote: post #84

Honestly, this reads like a thorough indictment of signals in user space. * Signal handlers are process global * Signal handlers need to be re-entrant safe Re-entrancy is painful but can be done, but process-global signal handlers means that pulling in a totally unrelated library can break your code. Moreover, it makes the combined use of certain libraries straight-up impossible. Similarly, it means that the use of l…

Normal libraries should never register signal handlers. Google Breakpad is a crash-reporting system and as such requires signal handling to function. Also since we're on the topic, here's a nice vulnerability caused by bad signal handling: https://news.ycombinator.com/item?id=16753013 Pretty sure there are no plans to replace signals, but maybe there are libraries that make signal handling easier?

> Normal libraries should never register signal handlers.

That statement alone should be enough to re-examine a feature. Code composition is a rather nice thing to have, making that very complicated is not a good thing.

Re: Use mmap with care

#168

Post by rocqua »

quotemstr wrote: post #146
rocqua wrote: post #84

Honestly, this reads like a thorough indictment of signals in user space. * Signal handlers are process global * Signal handlers need to be re-entrant safe Re-entrancy is painful but can be done, but process-global signal handlers means that pulling in a totally unrelated library can break your code. Moreover, it makes the combined use of certain libraries straight-up impossible. Similarly, it means that the use of l…

There's nothing wrong with signals per se. The problem is the signals API . Why do signals handlers need to be process-global? Why do we need to live with global signal handler registration clobbering any previous registration? We can change these things! These problems are all fixable without "replac[ing] signals" as the mechanism. Ultimately, as long as processors have traps (which they will, as long as we have vir…

For others [1] is the proposal meant, suggested in HN comment [2]

[1] https://www.facebook.com/notes/daniel-colascione/toward-shar... [2] https://news.ycombinator.com/item?id=19806448

Re: Use mmap with care

#169

Post by rocqua »

beat wrote: post #166
rocqua wrote: post #165

Earlier quoted context omitted.

A feature that 1. cannot be used if we compose two systems that use it 2. precludes using other mainstream features 3. is an essential part of the interface between userspace and kernelspace is a bad feature. I am not saying we should drop the idea of allowing userspace to respond to things the kernel raises. I am saying we need a way that will allow 1. program composition and 2. play nice with threads. I am not a un…

Signals play nice with threads. It's sloppy thread programming that doesn't play nice with signals.

Consider the following scenario:

Two threads both want to MMAP a file / catch SIGSEGV for different pieces of code. Moreover, these threads come from different modules of a system maintained by different people. All of a sudden, these modules become coupled because we need some system for delegation of signal handlers between them. Or, we need to introduce a custom signal-handler module to deal with our delegation.

I would not call that 'playing nice with threads' nor would I call that sloppy thread programming.

Re: Use mmap with care

#170

Post by knorker »

ajross wrote: post #123
beat wrote: post #108

The first serious bug I ever dealt with professionally was a result of the hazards of mmap(). This was 1995, and I was working on AIX with a system that used a series of shared memory buffers for IPC. It was originally written with shmat(), and on AIX (at least in those days), shmat was limited to three shared segments, so we had a lot of performance-wrecking blocking going on while waiting for the buffers to be clea…

Yeah. It's sometimes hard to remember just How Bad software was even in living memory. These days, we all just assume that the basics all work and are surprised to see bugs, which we vote up to the top of HN. But it wasn't always like that, things just failed in crazy ways, at all levels of the stack. At the start of the dotcom boom, a server uptime measured in months was considered notable, and the idea of a client…

To me in the 90s uptime was mostly limited by the fact that every month or so there was a new Linux kernel with a feature, driver, or fix that actually mattered to me. And I'd do `make oldconfig` to answer only new questions.

Now, it just works... mostly. systemd came in and made it like it's the 90s all over again with just plain basic things no longer working. It's pretty much stabilized in my experience in the last year, though, but it took years. The latest disaster in sound systems was well into the '10s, and actually not quite "just works" yet.

As for phone, I bounce it about once a month when the bluetooth stack just stops working.

And that's all for Linux. *BSD tends to go "Yeah, we don't do that here" about POSIX, so it can be unpredictable. And Solaris is just madness.

So no, I wouldn't say we're quite into "basic stuff works" just yet.

Post reply on HN