Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
You've used thread_local for the sigjmp_buf. Are you sure the mechanism used by thread_local is safe to use in a signal handler? Relevant bug from Rust: "TLS accesses aren't async-signal-safe", https://github.com/rust-lang/rust/issues/43146 I'm thinking even if it works on certain OSes, it's not guaranteed, because a signal handler's context is not a thread context - or is it? E.g. the thread_local mechanism might de…
Use mmap with care
191–200 of 218 posts
Re: Use mmap with care
#192Author here, if anyone has any questions in relation to me or Sublime HQ please feel free to ask.
I'm just going to leave this right here: http://man7.org/linux/man-pages/man2/signalfd.2.html
> The signalfd mechanism can't be used to receive signals that are synchronously generated, such as the SIGSEGV
Re: Use mmap with care
#193Re: Use mmap with care
#194Earlier quoted context omitted.
Different architectures like ARM? Plenty of code will only ever run on an x86 chip, but a non-trivial amount will run on x86 and ARM at some point. I've certainly been bitten by this before.
Sure, but my point is that both Sublime products only support x86 currently. There's likely a fair amount of other stuff that would break if we ported to ARM.
Re: Use mmap with care
#195Earlier quoted context omitted.
> It's possible to do much better than sigaction(2). I wrote up a detailed proposal for improvement in [1]. Thanks for interesting read. That said, I can see why glibc people didn't appreciate the proposal. The first part of article doesn't mention async-signal safety at all. Second part papers over async-signal safety, as if it were a non-issue. There are some dangerous-sounding paragraphs too: > It’s occasionally u…
> The first part of article doesn't mention async-signal safety at all. Second part papers over async-signal safety, as if it were a non-issue. If you're writing a signal handler, the signal handler needs to be async-signal-safe. You can't just wave your arms and make the problem of async signal safety disappear, because CPU traps themselves are async-signal-unsafe. Even userfaultfd has to deal with async signal safe…
I don't believe, that everyone holds that opinion. Even if they did, the world does not revolve around Red Hat's team, — there are still kernel mail lists and other venues for discussion. But if proposed improvements aren't well thought-out, would anyone there back them up?
In my opinion, async-signal safety in itself is much bigger problem than robust registration of signals. The later is mostly solved by chaining signal handlers, while former is mostly unsolved (and keeps getting worse). Proliferation of new libraries and async-signal unsafe conventions. People keep using printf() in signal handlers. Occurrences of fork() in multi-threaded apps. Still no async-signal safe malloc() (some Googlers tried, but the idea didn't get much traction). And then you come and propose new interface for registering signals handlers, and say that "It’s okay for two functions can be async-signal-unsafe". If your proposed API is async-signal unsafe, how would it deal with signals arriving during dispatch of signal handler list?
Re: Use mmap with care
#196Earlier quoted context omitted.
The license of git (GPL2) might be an issue for a commercial product. libgit2 is also GPL. (Also, IPC and fork+exec has overhead that mmap or thread in the same program does not.)
There are no license issues with bundling the git command-line tools in a commercial product, there's multiple existing proprietary commercial applications built on top of git that do so. The libgit2 code is GPL with a linking exception, so you can use it (unlike "git" itself) as a C library in a proprietary commercial product. > IPC and fork+exec has overhead[...] The "git cat-file --batch" command is something you'…
Re: Use mmap with care
#197The first serious bug I ever dealt with professionally was a result of the hazards of mmap(). This was 1995, and I was working on AIX with a system that used a series of shared memory buffers for IPC. It was originally written with shmat(), and on AIX (at least in those days), shmat was limited to three shared segments, so we had a lot of performance-wrecking blocking going on while waiting for the buffers to be clea…
>>> One of the first rules in the marvelous book The Pragmatic Programmer is "Select() isn't broken". Yeah, but sometimes it is. How come you give this example just now? This can't be a coincidence. select() just caused us a major production outage. FYI: select() is broken on pretty much all Linux kernels up to very recent ones. Doesn't work when there are more than 1024 opened file descriptors, not a lot for a serve…
Re: Use mmap with care
#198Earlier quoted context omitted.
> The first part of article doesn't mention async-signal safety at all. Second part papers over async-signal safety, as if it were a non-issue. If you're writing a signal handler, the signal handler needs to be async-signal-safe. You can't just wave your arms and make the problem of async signal safety disappear, because CPU traps themselves are async-signal-unsafe. Even userfaultfd has to deal with async signal safe…
> The glibc people literally think that nobody should be using signals. That's their objection, not anything you've talked about. I don't believe, that everyone holds that opinion. Even if they did, the world does not revolve around Red Hat's team, — there are still kernel mail lists and other venues for discussion. But if proposed improvements aren't well thought-out, would anyone there back them up? In my opinion,…
I think the proposal is thought-out. It's the objections I've seen that suggest a lack of thorough consideration.
> the world does not revolve around Red Hat's team
The facility I'm describing needs to be in libc to be useful, and for better or worse, if it's not in glibc and isn't something you can use as a raw system call, it might as well not exist.
> In my opinion, async-signal safety in itself is much bigger problem than robust registration of signals
Async signal safety concerns are inherent in any approach that exposes CPU traps to userspace, since traps occur at instruction granularity. As I've said elsewhere, writing async-signal-safe code isn't that hard if you follow a few basic rules. You can't solve the async signal safety problem, and we shouldn't need an all-singing, all-dancing async-signal-safety-requirement-avoidance system just to improve on the signal API.
The alternative to what I'm proposing isn't that everyone abandons signals. The alternative is that everyone keeps using sigaction, which sucks.
> The later is mostly solved by chaining signal handlers,
No, it isn't. I go into great detail in my note, in which I explain why current solutions to this problem are lacking and describe ways we can do better.
> Proliferation of new libraries and async-signal unsafe conventions. People keep using printf() in signal handlers. Occurrences of fork() in multi-threaded apps.
Okay, so don't do those things. The problems you're describing all come from people ignorant of safe signal-handler programming practices writing bad code. The problems disappear with education. The problems I'm describing don't disappear with education, since the current signal API imposes unavoidable limitations that even the best code can't work around, even in principle.
It's as if we have a car with hand-cranked windows and no brakes and you're annoyed that we would fix the brakes before adding power windows. The brakes are necessary to drive the car properly. Power windows are just an ergonomic feature.
> If your proposed API is async-signal unsafe, how would it deal with signals arriving during dispatch of signal handler list?
I don't understand this objection. Registration doesn't have to be async-signal-safe. Dispatch must be. There are multiple ways of implementing such a system --- e.g., CAS on a word pointing to a signal control structure.
Re: Use mmap with care
#199Earlier quoted context omitted.
> Have you considered making a dispatch_source_t I think that would have been considerably more work than finding the SO answer that says you need to use sigsetjmp, and would probably still conflict with Breakpad ;) > Would it be possible to install your own handler before Breakpad does? I may be wrong, but I think you can only register one exception handler per "task" (process), so Breakpad would override ours.
When you install a mach exception handler you can get the port of the previous exception handler, which you can use to forward the messages your newly installed exception handler receives. Of course (as with all raw mach APIs) it is poorly documented and error prone.
Re: Use mmap with care
#200Earlier quoted context omitted.
Is there a post where it's covered why Sublime Merge implements things like packfile reading on its own, rather than using git's own plumbing? E.g. in this case presumably keeping a "git cat-file --batch" would do the trick. I contribute to git.git, and it would be interesting to know if there's inherent issues stopping you from doing that, or if it's implementation problems in some cases (e.g. missing plumbing comma…
We do defer to Git for all write operations, but for reading, we do it ourselves partly for efficiency, and partly to get the right data. In terms of getting the right data, one example is that we need to know the full set of non-ignored sub-directories in the working directory, so we can watch them for changes. It's easy enough to generate this ourselves as we calculate the status output, but I don't believe that gi…
There's going to be cases where it sucks, e.g. what you point out with wanting both raw blobs and their diffs, you'd need to do that in two plumbing commands now.
But just on that example: Having poked at some of the diff code recently I can tell you there's no big technical hurdle to just exposing that sort of thing. I.e. spewing out machine-readable raw blobs and their diffs, it just happens not to be exposed now.
I think what a program like Sublime Merge would want/need short of C API access (which is unlikely to happen) is a git version of an open-ended "plumbing" IPC protocol of the sort that Common Lisp VMs tend to expose. I.e. being able to have one (or few) "git command-server" processes spawned, and ask them questions like "look up this blob" or "diff these two blobs" (where the previous blob lookup would be cached).
Obviously patching/coordinating/upstreaming those sorts of changes is going to take work, but so is duplicating and keeping up-to-date with the diff, pack, status etc. code.
I'm not trying to tell you what to do, just saying that the git project is definitely friendly to "we're a commercial product and need this missing plumbing for our editor" (unlike say, GCC).
The plumbing that's there now is mostly in the state it's in because it's what git itself needed in the past when it was more of a collection of shellscripts, as well as being biased towards what git server operators like GitHub needed (because they sent more patches), which is why plumbing for say batch blob operations tends to be better than the one for "status".
In any case it would be very interesting to have some post about the sort of read-only operations Sublime Merge is doing with its own custom git code.