131–140 of 184 posts
Fork has really weird semantics, and a lot of fun gotchas around managing resources. Good riddance?
Between the `fork()` and an `exec()`, I can: * redirect stdin, stdout, and stderr * open files that might be needed and close files that aren't * change process limits * drop privileges * change the root directory * change namespaces And there are a few other things I am probably forgetting.
Earlier quoted context omitted.
I'm not sure I agree it's the ideal way to do it. That's a heck of a lot of work for one function to do, and it necessarily duplicates the functionality of a ton of other functions. And that's ignoring the fact that forking without ever exec'ing can be really useful in many cases. I haven't yet read the paper, but considering the incredible simplicity from the programmer's PoV that fork provides, and the fact that at…
When you read the paper, you'll see this covered in section 6 ("REPLACING FORK") subsection "Low-level: Cross-process operations" > While a spawn-like API is preferred for most instances of starting a program, for full generality it requires a flag, parameter, or new helper function controlling every possible aspect of process state. It is infeasible for a single OS API to give complete control over the initial state…
fork() is also used to daemonize and for privilege separation, two tasks where posix_spawn() cannot be used. I suppose daemonization can be seen as something of the past, but privilege separation is not. On Linux, privileges are attached to a thread, so it should be possible to spawn a new thread instead of a new process. However, a privileged thread sharing the same address space as an unprivileged one doesn't seem…
Removing fork() will take a long, long time. Every popular use case needs an alternative that doesn't suck.
But then again, fork() is kinda awful[0].
[0] https://gist.github.com/nicowilliams/a8a07b0fc75df05f684c23c...
Earlier quoted context omitted.
Not even just the semantics, the performance is awful. Even when the fork is virtual (as any modern fork is) and there's no memory copying because it's COW, all the kernel page tables still need to be copied and for a multi-GB process that's nontrivial. That's why any sane large service that needs to fork anything will early on start up a slave subprocess whose only job is to fork quickly when the master process need…
>all the kernel page tables still need to be copied and for a multi-GB process that's nontrivial Only in the pathological case where the large process is backed solely by the 4kb pages. The hardware has long now supported large pages - on x86 since Pentium Pro, if memory serves - and huge pages. The popular OSes (Linux 2.6+ and Windows 2003+) also do support large and huge pages. A 2GB process can easily be three pag…
Is it really popular wisdom though, or is it the opinion of one person and it got hyped up, much like the same hype happened on a subpar programming language that same person worked on?
Earlier quoted context omitted.
You want to ditch shared libraries and mmap to map your big processes using GB pages to make fork fast again (despite it not being the main and only drawback)??? OS research might be relevant, and it's good that some people have wild idea, but honestly I doubt this one will go anywhere :P
Ah sorry, only want to ditch the shared libraries; added mmap is a later edition didn't realize it's unclear. Of course mmap is necessary.
With Spectre, proper process bounds (well, address spaces) are more important than ever -- and oh well even without that I'd still have cited them as incredibly important, in the sense that I'd rather have more than fewer. Given that, code reuse involves shared libraries, for several good reasons; the obvious one being not wasting RAM, but then there is the update problem (how to patch programs when security holes are discovered, especially if multiple parties are involved), and on top of that there is the cache pollution problem, which is related to the code duplication problem, and which is quite insidious because it is probably simultaneously hard to benchmark and very real (ambient loss of perf, just not in very hot paths, but this will still have an impact on the general perf of a system, quite like Spectre mitigations are having a big impact)
Now we could like address space boundaries so much that we would want to just use even MORE processes in place of shared libraries, but this obviously does not work for all services (and Spectre is biting us again because context switches are not cheap), plus if you take it to the extreme this makes systems extremely hard to design, and even bigger. This is part of the reasons we are using Linux instead of Hurd... (well Linux is too much in the opposite direction, but there are hopes that it will in the long term evolve toward a middle ground)
And anyway that does not fit the narrative at all of using more huge pages.
Now there are the usual radical ideas about how everything should be running on some kind of VM (sometimes even including the kernel), drastically reducing the amount of "native" code; but given the reality of our current systems that "everything" both relies on multiple VMs and I doubt it will tend to only one, nor should it (because of the monoculture this would induce). Plus the ambient perfs are still lower than native code, and TBH I don't expect that to change ever.
So, why and how would you like to get rid of shared libraries?
Earlier quoted context omitted.
The WSL processes are called pico processes. https://blogs.msdn.microsoft.com/wsl/2016/05/23/pico-process...
That was a super interesting read (and view), thank you. I've been in Linux land for almost two decades, but I've also spent a week (or so) porting our Linux-based development environment over to Windows with the help of WSL. This sheds some light on how it actually works. Maybe I'll have to look over it once more armed with this new information and see if I can squash some of those remaining problems with our soluti…
Earlier quoted context omitted.
Ah sorry, only want to ditch the shared libraries; added mmap is a later edition didn't realize it's unclear. Of course mmap is necessary.
About shared libraries, I know that there is this line of thought considering them "evil" (well at least sufficiently to want to get rid of them); but I'm quite unsure about what a modern system would look like without them (although this is less a problem at the application level on e.g. Android, the system level is still extremely important) With Spectre, proper process bounds (well, address spaces) are more import…
Most high integrity real time OSes are microkernels.
Interesting that you mention Android, one of the key points of Project Treble is using separate processes for drivers with Android IPC to talk to the kernel (including hardware buffer handles).
Okay, this one has me laughing out loud. Of COURSE Microsoft doesn't like fork()... Windows pretty much can't do it. I'll admit, there have been a lot of times I wish there was a more streamlined way to spawn processes on Linux (particularly daemons) but when I don't have fork() I always end up missing it. I'd take this paper a lot more seriously if it came from someone with a less obvious bias.
Earlier quoted context omitted.
Cleaning up your own process between fork and exec is hard. Several programs resort to terrible hacks like force-closing everything except file IDs 0,1,2 in a loop. Or they look into their /proc directory to discover whichnfile IDs exist, which is only marginally better. But when your process is a house of cards built on third party libraries with their own minds, there are not a lot of other options.
Use O_CLOEXEC everywhere (even third party libs). It's really annoying, but necessary. Means you need to use accept4(), dup3(), popen with an additional "e" (of course all of that needs to be feature tested, during compilation/runtime).
Earlier quoted context omitted.
They must have considered it many times (not least when making the partial posix support for NT) but felt that supporting it wouldn’t help deprecating it either. AFAIK it’s only unix/Linux (posix) OSes that implement fork. Perhaps that’s what you meant by “every other system”, ie unix + clones/derivatives?
VAX/VMS implement "vfork()" which is an implementation of the most common use-cases of fork. VAX and VMS are not POSIX or UNIX-like.