Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving beyond fork() + exec()

lwn.net

Post reply on HN
Print view View original

201–210 of 358 posts

Re: Moving beyond fork() + exec()

#201

Post by sanderjd »

1718627440 wrote: post #39
sanderjd wrote: post #17

I just ran into this recently, where I had an obscure bug caused by needing to close more file descriptors in the forked process. "I want a clone of the current process" is just way less common in my experience than "I want a completely new process". It feels crazy that we don't have a way to directly express the latter thing, and can only approximate it by cloning and then fixing things up in post.

But you generally want to communicate with that process, so you do need to setup e.g. file descriptors and stuff, which needs information from the parent process to be passed.

Nevertheless, inclusion would be a better default than exclusion in most use cases I've ever had for process spawning.

Re: Moving beyond fork() + exec()

#202

Post by sanderjd »

jonhohle wrote: post #44
1718627440 wrote: post #39

Earlier quoted context omitted.

But you generally want to communicate with that process, so you do need to setup e.g. file descriptors and stuff, which needs information from the parent process to be passed.

Most programming languages abstract this out to be able to connect or drop the 3 standard pipes. Typically this is the only thing that can be shared anyway unless the other program is specifically shared and expects other file handles to be available, in which case fork might be the right system call anyway.

Right. It's not that fork is useless, it's that it's weird that it's the only way to do this thing that it isn't particularly well suited for.

Re: Moving beyond fork() + exec()

#203

Post by nvme0n1p1 »

dcrazy wrote: post #125
nvme0n1p1 wrote: post #72

Earlier quoted context omitted.

That's not the reason for the performance difference. Windows does have a fork primitive (ZwCreateProcess) and it's still slower than Linux's equivalent.

Again, NtCreateProcess does not implement fork(). The fundamental characteristic of fork is that the child is an exact replica of the parent, down to the instruction pointer. Windows does not have a way to create a process object with such a configuration. Also, using the Zw prefix doesn’t make you look more knowledgeable, it makes you look like you’re trying way too hard to borrow credibility.

Okay but people don't claim that copying the instruction pointer (a single machine register) is the reason for any speed difference. They claim it's due to the memory sharing. And that's easily disproven since you can share pages, just like on Linux, simply by passing null for the section handle, yet there's still a performance difference.

Why does it matter which prefix I used? They both point to the same routine so my point applies either way.

Re: Moving beyond fork() + exec()

#204

Post by vbezhenar »

mpyne wrote: post #198
mort96 wrote: post #171

Earlier quoted context omitted.

CoW is probably a good idea regardless, yeah. Overcommit is more questionable. Regardless, both ought to be argued based on their own merits. It's unfortunate that both are necessary as a consequence of fork().

I don't think fork() mandates overcommit. OpenBSD doesn't seem to even allow overcommit or have an OOM killer, memory allocations that exceed available capacity fail immediately even if the memory is not touched.

Let's say you have 1GB RAM. You're running program that occupies 600 MB. Now this program wants to launch second small program that occupies 1 MB.

You're doing fork + exec.

If you're overcommiting, fork will not reserve another 600 MB, and exec immediately after fork will cause total system usage to be 601 MB.

If you're not overcommiting, that fork will fail, because total memory consumption will be 1200 MB which is more than 1GB. That somewhat restricts program design.

Re: Moving beyond fork() + exec()

#205

Post by nvme0n1p1 »

dcrazy wrote: post #114
nvme0n1p1 wrote: post #106

Earlier quoted context omitted.

We don't have any broadly used non-fork samples. Windows, macOS, and Linux all have fork. So the presence of fork can't be the reason for the performance difference. (Windows's fork is called ZwCreateProcess)

NtCreateProcess does not implement a forking model. It is analogous to posix_spawn.

If you pass null for the section handle, it shares pages with the calling process, thus implementing a forking model. Or at least the parts of a forking model that some people erroneously believe are responsible for performance differences.

Re: Moving beyond fork() + exec()

#206

Post by sanderjd »

jerf wrote: post #31
sanderjd wrote: post #23

Earlier quoted context omitted.

A process that shares nothing with the process that spawned it.

A thing that makes that complicated is that while you want that conceptually, you don't want that in reality. For instance, if the spawning process is in a container of some sort and it spawned a process that "shares nothing with the process that spawned it", the spawned process would no longer be in that container, because the state of "being in the container" is one of the things it shares with the parent process.…

Yes, stipulated. And it it's true that we should have a primitive for spawning a completely new process, because that's what we usually want. I agree that the details are both non trivial and soluble.

Re: Moving beyond fork() + exec()

#207

Post by sanderjd »

jerf wrote: post #176
dcrazy wrote: post #169

Earlier quoted context omitted.

It’s such a bad idea that every OS except Linux implements it? On macOS it’s posix_spawn, on Windows it’s NtCreateProcess.

Who said anything about it being a "bad idea"? I also explicitly said this wasn't unsolvable. My point isn't about technical implementations or code, my point is that the casual "I want to share nothing about the parent process" thought in sanderj's mind, and presumably a lot others, is much more ill-defined than they realize. There's a lot more state that a process has than what file descriptors are open in a modern…

It's not a casual thought. I recognize that there are lots of details, there always are, we're talking about computers :)

I don't think it is necessary (or the best implementation) to clone the parent process, in order to maintain important properties like the process tree / container state, etc. I recognize that it's a sorta neat hack, "well if we just start by cloning the parent, then we don't have to figure out what state to include!", but that just pushes the details to the child process needing to figure out what to exclude, which IMO is a worse default.

Re: Moving beyond fork() + exec()

#208

Post by mpyne »

toast0 wrote: post #89
pizlonator wrote: post #86

Earlier quoted context omitted.

You're referring to something else, and maybe I'm using the term "zygote" incorrectly. In all uses of zygotes that I have seen, here's what's really happening: - `fork` is being used to reduce the cost of starting a process that has a high start-up cost. So, you start one process, run it through the expensive initialization, and then fork it from there to start new processes. - To make this even faster, you have a po…

Oh I see. I guess your zygotes have developed more than mine. I think Google may have coined or at least popularized the term zygote for this in Chrome and Android, Chrome documentation [1] says: > A zygote process is one that listens for spawn requests from a main process and forks itself in response. Generally they are used because forking a process after some expensive setup has been performed can save time and sh…

> Oh I see. I guess your zygotes have developed more than mine. I think Google may have coined or at least popularized the term zygote for this in Chrome and Android, Chrome documentation [1] says:

Google may have popularized the term, but this approach was already in use by KDE developers in the KDE 2.x timeframe, where it was used as part of a system called kdeinit.

In this scheme, launching KDE apps from a KDE desktop could bypass much of the startup cost of dynamic linking by forking from a long-running kdeinit process (with kdeinit itself deliberately linked to all large dependency libs like Qt and kdelibs), dynamically loading the application logic (stored as a .so) and then launching the app.

This was more to save startup time due to how long it took to dynamically resolve a multitude of C++-based symbols back then, all the common logic came before the app's own main() would ever be called. But it did also save a bit of memory as well.

Re: Moving beyond fork() + exec()

#210

Post by loeg »

dcrazy wrote: post #174
loeg wrote: post #172

Earlier quoted context omitted.

> The things you can do between fork and exec are sometimes underestimated. Off the top of my head, you can call dup2(), you can set a process group id, probably a few other things. What do you mean underestimated? You can do anything between fork and exec; there are no limitations.

That’s not true. man 7 signal-safety

You're talking about libc design choices, not constraints imposed by the kernel. To the kernel, a post-fork pre-exec process is just any old process. GP was suggesting post-fork processes were constrained in the syscalls they could invoke; they are not.
Post reply on HN