Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use mmap with care

sublimetext.com

Post reply on HN
Print view View original

171–180 of 218 posts

Re: Use mmap with care

#171

Post by knorker »

nexuist wrote: post #136
pcwalton wrote: post #130

Earlier quoted context omitted.

Yep, this is part of the reason why it drives me crazy when people say the '90s were the heyday of computing. Computers were awful back then.

I think they still have a point. It was the heyday of computing precisely because they were awful; there was always something to work on or improve. Fixing bugs in operating systems offered clear improvements to the livelihoods of thousands or millions. It's no longer that easy now. Most critical things don't suck that hard anymore, so it's hard to find a way to feel special, or feel like it's a special time to be a…

On the flip side, while OpenSSH may be great and high quality now, and wasn't in the '90s, in the '90s people used telnet on the public Internet, and telnet worked.[1]

IOW: It wasn't basic tech and great opportunity at the same time.

Today you can work on immature things like OpenBSD's MPLS implementation and who knows, maybe in 20 years people will say that back in 2019 there were so many things that could be improved about that basic tech! (cue replies anti-labelswitching dweebs)

[1] Not me, I used ssh since before OpenSSH.

Re: Use mmap with care

#172

Post by ak217 »

beat wrote: post #108

The first serious bug I ever dealt with professionally was a result of the hazards of mmap(). This was 1995, and I was working on AIX with a system that used a series of shared memory buffers for IPC. It was originally written with shmat(), and on AIX (at least in those days), shmat was limited to three shared segments, so we had a lot of performance-wrecking blocking going on while waiting for the buffers to be clea…

I had a similarly maddening issue (circa 2014) once where small chunks of memory would randomly get overwritten on PV-virtualized AWS instances, but not on HVM-virtualized instances. We spent an enormous amount of time chasing this problem; the most senior engineer on the team wrote a test case that would have to write 100TB of data before reliably reproducing the issue.

We never did figure out what caused it. Eventually we migrated all our instances to be HVM-only, and the problem went away.

Re: Use mmap with care

#173

Post by avar »

loeg wrote: post #149
avar wrote: post #71

Earlier quoted context omitted.

Is there a post where it's covered why Sublime Merge implements things like packfile reading on its own, rather than using git's own plumbing? E.g. in this case presumably keeping a "git cat-file --batch" would do the trick. I contribute to git.git, and it would be interesting to know if there's inherent issues stopping you from doing that, or if it's implementation problems in some cases (e.g. missing plumbing comma…

The license of git (GPL2) might be an issue for a commercial product. libgit2 is also GPL. (Also, IPC and fork+exec has overhead that mmap or thread in the same program does not.)

There are no license issues with bundling the git command-line tools in a commercial product, there's multiple existing proprietary commercial applications built on top of git that do so.

The libgit2 code is GPL with a linking exception, so you can use it (unlike "git" itself) as a C library in a proprietary commercial product.

> IPC and fork+exec has overhead[...]

The "git cat-file --batch" command is something you'd invoke once, and then as your program runs you keep feeding it SHA-1s on stdin and it spews out their content on stdout. So even on Windows the overhead of that should be fine.

It's clear from ben-schaaf's other comments (which I read later) that one concern was the simplicity of downstream APIs being able to read the data using a normal C variable.

But that just leaves more questions. People in this thread are mentioning pack files, assuming that a multi-GB "git object" must be in a pack, but I notice the original post doesn't say anything about it.

If they're reading packs with this they'll need to parse it, resolve deltas etc. So likely the code that deals with the mmap()'d variable is small in any sane codebase (they're surely not doing delta resolution repeatedly all over the place...).

If they're very large loose objects those will most likely be zlib compressed, so wouldn't this need to go through some intermediary API layer anyway? I guess if SM itself is adding them it could add them uncompressed.

Since ben-schaaf mentioned this not being about performance, but about saving memory I thought this might be something like wanting to extract a small part of a 1GB object from git for display. That seems like a thing an editor might want to do.

In that case "git cat-file --batch" would suck, but not for some intrinsic reason. An API could be added that could take the start/end of an object to print out.

Re: Use mmap with care

#174

Post by lgg »

ben-schaaf wrote: post #35
saagarjha wrote: post #6

> Using setjmp and longjmping from a signal handler is actually unsafe. It seems to cause undefined behaviour, especially on MacOS. Have you considered making a dispatch_source_t of type DISPATCH_SOURCE_TYPE_SIGNAL and handling all signals in a dispatch queue, instead of trying do figure out what kind of behavior is legal in a signal handler? > If a library such as Breakpad registers for Mach exception messages, and…

> Have you considered making a dispatch_source_t I think that would have been considerably more work than finding the SO answer that says you need to use sigsetjmp, and would probably still conflict with Breakpad ;) > Would it be possible to install your own handler before Breakpad does? I may be wrong, but I think you can only register one exception handler per "task" (process), so Breakpad would override ours.

When you install a mach exception handler you can get the port of the previous exception handler, which you can use to forward the messages your newly installed exception handler receives. Of course (as with all raw mach APIs) it is poorly documented and error prone.

Re: Use mmap with care

#176

Post by jacobolus »

pcwalton wrote: post #130
ajross wrote: post #123

Earlier quoted context omitted.

Yeah. It's sometimes hard to remember just How Bad software was even in living memory. These days, we all just assume that the basics all work and are surprised to see bugs, which we vote up to the top of HN. But it wasn't always like that, things just failed in crazy ways, at all levels of the stack. At the start of the dotcom boom, a server uptime measured in months was considered notable, and the idea of a client…

Yep, this is part of the reason why it drives me crazy when people say the '90s were the heyday of computing. Computers were awful back then.

On the other hand, there are some things that are decidedly worse today than on some old machines. For example keyboard input latency on modern computers can be horrendous, and the physical keyboard hardware is utter garbage compared to what was available in the 80s.

For a single-purpose writing machine (and as long as you don’t mind sitting at a desk), my parents’ Mac SE from 1989 running a version of MS Word from the early 1990s is IMO significantly more usable than a top-of-the-line 2019 computer running the latest version of MS Word.

Given the amount of research and implementation effort/investment that has gone into improving computers over the past few decades, the state of user-facing software (including browsers and web apps) for the average person today is in my opinion shamefully bad.

Re: Use mmap with care

#177

Post by legulere »

Asooka wrote: post #14

Oh I see you didn't get to caveat 5: you can't read anything more complicated than raw bytes, i.e. chars, because of unaligned memory access errors. Let's say you mmap a file and do something like this: char *fileContents=...mmap etc...; int headerOffset=*(int*)fileContents; int *someListOfNumbers=(int*)(fileContents+headerOffset); int importantSum= someListOfNumbers[0] + someListOfNumbers[1] + someListOfNumbers[2] +…

If you dereference pointers with arbitrary offsets read from a file, alignment issues are the least of your issues.

Re: Use mmap with care

#178

Post by icedchai »

knorker wrote: post #171
nexuist wrote: post #136

Earlier quoted context omitted.

I think they still have a point. It was the heyday of computing precisely because they were awful; there was always something to work on or improve. Fixing bugs in operating systems offered clear improvements to the livelihoods of thousands or millions. It's no longer that easy now. Most critical things don't suck that hard anymore, so it's hard to find a way to feel special, or feel like it's a special time to be a…

On the flip side, while OpenSSH may be great and high quality now, and wasn't in the '90s, in the '90s people used telnet on the public Internet, and telnet worked.[1] IOW: It wasn't basic tech and great opportunity at the same time. Today you can work on immature things like OpenBSD's MPLS implementation and who knows, maybe in 20 years people will say that back in 2019 there were so many things that could be improv…

Telnet and rlogin were both very common. I remember installing ssh on my Slackware Linux box back in 1996. I had to build it from source... fun times!

Re: Use mmap with care

#179

Post by maxlybbert »

nneonneo wrote: post #17
maxlybbert wrote: post #15

Earlier quoted context omitted.

I think it’s reasonable to say “it’s difficult to justify using mmap when other programs are expected to manipulate, and potentially even delete, the file while you are working with it.” But, honestly, that case will always be hard to handle, and mmap doesn’t make it any worse.

I think the problem here is that mmap does make it worse - with (p)read, if you get an errno, you flag an error. with mmap, you have to handle a signal and jump back to an appropriate point in the code (and you can't do this in a cross-platform way), and then flag an error. Obviously, what the upper levels do with the error still has to be worked out, but you can hardly argue that sigsetjmp/siglongjmp + signal handle…

If the file changes out from under you, all you can do is reread the file. mmap gives you a pointer that you can read from, but you still have to loop through the file’s bytes.

I’m pretty sure most programmers are capable of writing a signal handler that sets a flag (volatile sig_atomic_t) for “parsing failed,” and a loop that checks that flag in addition to checking whether the loop is finished for other reasons. Signal handling doesn’t have to be complicated.

Re: Use mmap with care

#180

Post by ben-schaaf »

0xffff2 wrote: post #121
ben-schaaf wrote: post #48

Earlier quoted context omitted.

> Oh I see you didn't get to caveat 5 > On x86 you'll get away with it... Guess we'll have to wait for the proliferation of a different architecture to encounter this one :)

Different architectures like ARM? Plenty of code will only ever run on an x86 chip, but a non-trivial amount will run on x86 and ARM at some point. I've certainly been bitten by this before.

Sure, but my point is that both Sublime products only support x86 currently. There's likely a fair amount of other stuff that would break if we ported to ARM.
Post reply on HN