131–140 of 218 posts
Things are so much better if you are not writing apps for general public (mine are trading-related). You can tell your few clients — make sure that the access to mmapped file is exclusive — and get away with it. And yes, mmap is the awesomest thing out there.
Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Earlier quoted context omitted.
It is not constructive to form a sweeping generalization from a statement and then claim the sweeping generalization is ludicrous, implying the original statement is ludicrous. :) My first comment was in reply to one claiming anonymous memory did not have the same problems as file-backed memory, indicating the parent did not understand they are the same thing. The subsequent reply was to another comment continuing to…
As if "avoid virtual memory" is substantially more of a "sweeping generalization" than "avoid mmap." If you apply the same reasoning that you've used to conclude that everyone should avoid mmap, than you are led directly to the conclusion that everyone should avoid virtual memory. The "same problems" that you are pointing out are possible with anonymous memory aren't unique to memory you get directly from mmap, they…
The exotic error paths are always there, but you don't always need to handle them. You can push some error handling to other systems, such as clients and supervisors / orchestrators. The reason why mmap with files is tractable is because we have a good error handling strategy (remap with zeroes, mark error) and we have a few understandable reasons why we might expect the error (IO errors, lost media/network failures, or even truncate). In general the problem that IO is done outside of direct syscalls like write() can be difficult even when you're not using mmap, like when Postgres was losing errors when calling fsync.
But when you have an IO error in your swap file, go ahead, eat the SIGBUS and die. This is fine.
Earlier quoted context omitted.
It is not constructive to form a sweeping generalization from a statement and then claim the sweeping generalization is ludicrous, implying the original statement is ludicrous. :) My first comment was in reply to one claiming anonymous memory did not have the same problems as file-backed memory, indicating the parent did not understand they are the same thing. The subsequent reply was to another comment continuing to…
As if "avoid virtual memory" is substantially more of a "sweeping generalization" than "avoid mmap." If you apply the same reasoning that you've used to conclude that everyone should avoid mmap, than you are led directly to the conclusion that everyone should avoid virtual memory. The "same problems" that you are pointing out are possible with anonymous memory aren't unique to memory you get directly from mmap, they…
The "powerful (and consequently hazardous) OS feature" here is using mmap for general file IO, it:
- introduces resources leaks many developers can't profile
- introduces VM bottlenecks 99% of developers can't profile
- introduces random segfaults delivered to arbitrary threads in the running process, leading to crashes many developers can't diagnose
- the mmap() interface itself is fundamentally unsafe in that it allows partially overwriting random bits of VM (MAP_FIXED) with file views, and worse still, allows those mappings to be read-write
Once again, nobody has ever suggested avoiding virtual memory except you -- once again, that is impossible in a modern environment, but it is more than possible, and ultimately incredibly sensible, not to mention entirely on topic with regards to this thread and the article it is attached to, to suggest avoiding use of mmap for general file IO
Earlier quoted context omitted.
Couldn’t you mmap a shared, read-only page so that your OS doesn’t copy it?
It would be better to mmap the file into N single-threaded processes instead of mmapping the file into 1 N-threaded process. This is exactly why signals are process-global instead of thread-local. The intent is that they were used for inter-process communication. The whole notion of processes and signals predates the notion of a thread, and threads are essentially a performance optimization for 30 or 40 year old hard…
Signals can be sent either to a process or to a specific thread under linux. Signals sent to the process are handled by an arbitrary thread. There's a bunch of gotchas and whatnot past that.
Mixing signals and threads is pretty much trash.
Earlier quoted context omitted.
Yeah. It's sometimes hard to remember just How Bad software was even in living memory. These days, we all just assume that the basics all work and are surprised to see bugs, which we vote up to the top of HN. But it wasn't always like that, things just failed in crazy ways, at all levels of the stack. At the start of the dotcom boom, a server uptime measured in months was considered notable, and the idea of a client…
Yep, this is part of the reason why it drives me crazy when people say the '90s were the heyday of computing. Computers were awful back then.
It's no longer that easy now. Most critical things don't suck that hard anymore, so it's hard to find a way to feel special, or feel like it's a special time to be a part of.
Even in startup land most work done is "build this website" or "build this app" - what the software does could be cool and easy enough to get passionate about, but the hope that "we're going to use this code to revolutionize the entire world for the better" that was omnipresent in '90s hacker culture has pretty much been murdered by the sins of Facebook and others in Big Tech. Now it's "what can I do to make sure the stuff I work on hurts the least amount of people as possible?"
The first serious bug I ever dealt with professionally was a result of the hazards of mmap(). This was 1995, and I was working on AIX with a system that used a series of shared memory buffers for IPC. It was originally written with shmat(), and on AIX (at least in those days), shmat was limited to three shared segments, so we had a lot of performance-wrecking blocking going on while waiting for the buffers to be clea…
AIX 3.2.5, around the same time, had a different mmap issue. If you mmap a file and write a byte into the mapped region, but beyond the end of the file, you get a signal; in handling the signal, you can extend the file. You also have 1. Made an error, as documented in the mmap man page, 2. Done something that works on JFS, 3. In fact, done what JFS does itself under the covers, and 4. Made a huge, huge mistake. If yo…
Earlier quoted context omitted.
As if "avoid virtual memory" is substantially more of a "sweeping generalization" than "avoid mmap." If you apply the same reasoning that you've used to conclude that everyone should avoid mmap, than you are led directly to the conclusion that everyone should avoid virtual memory. The "same problems" that you are pointing out are possible with anonymous memory aren't unique to memory you get directly from mmap, they…
Let's try and simplify things here: the post we're both currently commenting on relates to doing file IO via nmap . Of course 'avoiding all use of virtual memory' is ridiculous, but nobody except you is suggesting that, and you continue to suggest it even after a long reply. The "powerful (and consequently hazardous) OS feature" here is using mmap for general file IO , it: - introduces resources leaks many developers…
- Nobody ever said mmap was always faster than the alternative. If you care about performance then you should do whole-application performance testing with and without features enabled (like mmap IO). This is not unusual, there are plenty of aspects of performance that are counter-intuitive, where speeding up one part of your program causes a seemingly unrelated part of your part of your program to slow down.
- The signals are SIGBUS, and they can be intercepted, mapped with zeroes, and the errors can be propagated back to your app code later. This is not trivial but neither is it outrageous.
- You can overwrite arbitrary memory with read(), too, you just have to pass it a pointer to something you want to overwrite. mmap() is not any less safe. Recall that typical use of MAP_FIXED is so you can overwrite an existing mmap() region with something else, not so you can nuke random parts of your address space.
Keep in mind that you are, if nothing else, an indirect user of mmap(). The question is whether using mmap() directly is advantageous for your applicaiton. "Yes" is not an unreasonable nor outrageous answer for some applications.
Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Small side note for completeness: The effects of oh-noes-my-file-is-gone can be somewhat mitigated by using the heuristics built into NSData (instead of using mmap directly). For example, you call NSData’s `dataWithContentsOfFile:options:error:` with the `NSDataReadingMappedIfSafe` option [1]. The framework will then transparently mmap the file unless it believes there’s an elevated risk of the file going away. Apple…
Earlier quoted context omitted.
Yeah. It's sometimes hard to remember just How Bad software was even in living memory. These days, we all just assume that the basics all work and are surprised to see bugs, which we vote up to the top of HN. But it wasn't always like that, things just failed in crazy ways, at all levels of the stack. At the start of the dotcom boom, a server uptime measured in months was considered notable, and the idea of a client…
Yep, this is part of the reason why it drives me crazy when people say the '90s were the heyday of computing. Computers were awful back then.
And the improvements are less than you might think. On that same job, we were managing a couple dozen servers in two different data centers, and 200+ client machines, all remotely, able to deploy different versions of the software to specific machines. It was like "cloud", except we did it all without so much as ssh. We had a configuration management database in Sybase, and rsh/perl scripts, along with SMIT commands (I miss SMIT).