Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
I'm just going to leave this right here: http://man7.org/linux/man-pages/man2/signalfd.2.html
Use mmap with care
181–190 of 218 posts
Re: Use mmap with care
#182Earlier quoted context omitted.
Well the problematic example given in the article was NTFS where the whole filesystem can disappear, but the problem applied to local files too, eg if their size is changed by another process.
Where did you see mention of NTFS? I was referring to "As it turns out, the ticket comes from someone using a networked drive."
Sorry, it was a typo, I meant NFS or more generally some type of networked drive.
Re: Use mmap with care
#183Earlier quoted context omitted.
I think the problem here is that mmap does make it worse - with (p)read, if you get an errno, you flag an error. with mmap, you have to handle a signal and jump back to an appropriate point in the code (and you can't do this in a cross-platform way), and then flag an error. Obviously, what the upper levels do with the error still has to be worked out, but you can hardly argue that sigsetjmp/siglongjmp + signal handle…
If the file changes out from under you, all you can do is reread the file. mmap gives you a pointer that you can read from, but you still have to loop through the file’s bytes. I’m pretty sure most programmers are capable of writing a signal handler that sets a flag (volatile sig_atomic_t) for “parsing failed,” and a loop that checks that flag in addition to checking whether the loop is finished for other reasons. Si…
It’s not quite as simple as setting a flag - if the signal is SIGBUS/SIGSEGV then returning from the handler normally will retry the faulting instruction. So, you either have to map dummy memory into place so the fault doesn’t reoccur, or use sigsetjmp/siglongjmp to jump out to a previously configured exception handler (as Sublime Text did here.
I think you’ve mixed up synchronous signals - SEGV, BUS, ABRT - with asynchronous signals like QUIT, INT, USR1. Asynchronous signals can be handled easily with a flag and a loop as you mentioned; synchronous signals are much trickier and much more complicated.
Re: Use mmap with care
#184The first serious bug I ever dealt with professionally was a result of the hazards of mmap(). This was 1995, and I was working on AIX with a system that used a series of shared memory buffers for IPC. It was originally written with shmat(), and on AIX (at least in those days), shmat was limited to three shared segments, so we had a lot of performance-wrecking blocking going on while waiting for the buffers to be clea…
>>> One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is.
How come you give this example just now? This can't be a coincidence. select() just caused us a major production outage.
FYI: select() is broken on pretty much all Linux kernels up to very recent ones. Doesn't work when there are more than 1024 opened file descriptors, not a lot for a server app. Side effects include crashing the application.
Re: Use mmap with care
#185Earlier quoted context omitted.
The original post quantifies it. Around 50% better performance for the mmap version. They are saying that if they somehow knew up front what the performance gains would be, and what the cost in bugs and complexity would be, they wouldn’t have used mmap at all.
"some quick benchmarks for the way Sublime Merge reads git object files" is in not strong evidence.
I can’t tell what answer you’re looking for here.
You said above that “mmap allows reading and writing large data structures without copying, which can be a huge benefit depending on the use case.”
Yes, of course that’s true, and the Sublime Text authors are clearly well aware it’s true. That’s why they decided to use mmap in the first place. They agreed with you.
This is them reporting, with hindsight, that for their use case mmap introduced a lot of tricky bugs that required complex platform-dependent fixes, and that the performance gains were real but modest. Therefore, in hindsight, it probably wasn’t a good choice.
Which part are you arguing with?
Re: Use mmap with care
#186Earlier quoted context omitted.
Yep, this is part of the reason why it drives me crazy when people say the '90s were the heyday of computing. Computers were awful back then.
On the other hand, there are some things that are decidedly worse today than on some old machines. For example keyboard input latency on modern computers can be horrendous, and the physical keyboard hardware is utter garbage compared to what was available in the 80s. For a single-purpose writing machine (and as long as you don’t mind sitting at a desk), my parents’ Mac SE from 1989 running a version of MS Word from t…
> my parents’ Mac SE from 1989 running a version of MS Word from the early 1990s is IMO significantly more usable than a top-of-the-line 2019 computer
This is a bit spun. I can't tell you what to like, but it's easy to romanticize older machines and forget what life was like with a 512x342 monochrome 60 Hz CRT, no backbuffer, and a CPU that took multiple frames to do something as simple as scroll.
I actually have a still-working Mac Plus I like to show off to friends. And yeah -- it was a great writing machine and in lots of ways current MS Office is a disaster of complexity. It was elegant in a historic way, but.. no, it's not "better" by the definition used by pretty much any working writer.
Re: Use mmap with care
#187Earlier quoted context omitted.
On the other hand, there are some things that are decidedly worse today than on some old machines. For example keyboard input latency on modern computers can be horrendous, and the physical keyboard hardware is utter garbage compared to what was available in the 80s. For a single-purpose writing machine (and as long as you don’t mind sitting at a desk), my parents’ Mac SE from 1989 running a version of MS Word from t…
> my parents’ Mac SE from 1989 running a version of MS Word from the early 1990s is IMO significantly more usable than a top-of-the-line 2019 computer This is a bit spun. I can't tell you what to like, but it's easy to romanticize older machines and forget what life was like with a 512x342 monochrome 60 Hz CRT, no backbuffer, and a CPU that took multiple frames to do something as simple as scroll. I actually have a s…
My father continued to use it for the majority of his work until at least 2005, and the computer still boots and works just fine today. He had far more frequent problems with a succession of Windows laptops in the subsequent 15 years than from the 15 years of using the Mac SE.
Having a small monochrome screen is not an insurmountable problem in practice for people used to doing most of their work by spreading physical paper around on the floor. Yes it could be better if it took advantage of 30 years of CPU, etc. improvements, but it’s nonetheless a fantastic tool.
Your “working writers” probably also need their computers for web browsing or whatever. And probably waste a ton of time on computer-related or computer-aided bullshit.
Re: Use mmap with care
#188The first serious bug I ever dealt with professionally was a result of the hazards of mmap(). This was 1995, and I was working on AIX with a system that used a series of shared memory buffers for IPC. It was originally written with shmat(), and on AIX (at least in those days), shmat was limited to three shared segments, so we had a lot of performance-wrecking blocking going on while waiting for the buffers to be clea…
Yeah. It's sometimes hard to remember just How Bad software was even in living memory. These days, we all just assume that the basics all work and are surprised to see bugs, which we vote up to the top of HN. But it wasn't always like that, things just failed in crazy ways, at all levels of the stack. At the start of the dotcom boom, a server uptime measured in months was considered notable, and the idea of a client…
My favorite “how did this go unnoticed” bug in that era was in Windows 95 (so … target rich) when working on an IDE where the File Open dialog showed the C: drive as empty, but only on one PC at a single customer site and one test box in our QA lab – every other available system worked, as did all of the Windows NT systems. Someone wrote a test program and confirmed that it was immediately exiting with the normal no-more-entries status, exactly as in the documentation except for the lack of entries.
I had a lucky guess remembering that the test box wasn’t on the LAN and discovered that the problem depended on whether you’d enabled network sharing. You didn’t need to have shared anything or be using a share – simply having it enabled meant that a file system filter was installed and the local hard drive magically started returning results.
It turns out that there were multiple APIs which did this and apparently the original developer had unluckily picked the less common one. Switching was easy and had no apparent drawbacks. The best theory we had was that Microsoft’s QA group disliked copying builds around on floppies as much as we did and didn’t have many completely offline systems around.
Re: Use mmap with care
#189Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Small side note for completeness: The effects of oh-noes-my-file-is-gone can be somewhat mitigated by using the heuristics built into NSData (instead of using mmap directly). For example, you call NSData’s `dataWithContentsOfFile:options:error:` with the `NSDataReadingMappedIfSafe` option [1]. The framework will then transparently mmap the file unless it believes there’s an elevated risk of the file going away. Apple…
Wouldn't that mean that networked files would be loaded entirely into RAM, negating the whole reason we started using mmap in the first place?
Re: Use mmap with care
#190Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
Is there a post where it's covered why Sublime Merge implements things like packfile reading on its own, rather than using git's own plumbing? E.g. in this case presumably keeping a "git cat-file --batch" would do the trick. I contribute to git.git, and it would be interesting to know if there's inherent issues stopping you from doing that, or if it's implementation problems in some cases (e.g. missing plumbing comma…
We do defer to Git for all write operations, but for reading, we do it ourselves partly for efficiency, and partly to get the right data.
In terms of getting the right data, one example is that we need to know the full set of non-ignored sub-directories in the working directory, so we can watch them for changes. It's easy enough to generate this ourselves as we calculate the status output, but I don't believe that git will emit it.
In terms of performance, we rely on being able to read objects efficiently. For example, to show a commit, we can't just use the output of "git diff", as we need the full file contents to be able to calculate syntax highlighting correctly. You could go a long way with "git cat-file --batch", but there are plenty of contexts where you can't practically batch requests, and process creation costs + the lack of caching across requests (which can be quite significantly due to the delta encoding of objects) would be quite significant.