Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving beyond fork() + exec()

lwn.net

Post reply on HN
Print view View original

281–290 of 358 posts

Re: Moving beyond fork() + exec()

#281

Post by cyberax »

pizlonator wrote: post #57
rom1v wrote: post #36

Related to the discussion: "A fork() in the road": https://www.microsoft.com/en-us/research/wp-content/uploads/... > ABSTRACT > The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog t…

Fork is marvelous for the zygote pattern Hard to come up with an optimization that is equally efficient and elegant

Unless you want to create a thread in your zygote. Then it breaks down.

Raw fork() is terrible. Instead we need a proper primitive to stop and make a snapshot of a process.

Re: Moving beyond fork() + exec()

#282

Post by emmelaich »

mort96 wrote: post #79
anarazel wrote: post #58

Earlier quoted context omitted.

It is somewhat interesting that the most widely used "big" OS that doesn't use fork, i.e. Windows, has dog slow process creation... I agree that there should be non-fork primitives, I'm just not that sure that performance is the best argument.

The problem with fork isn't really that it's slow. The problem is that if you want it to be not-slow, it locks you into a bunch of OS design decisions: you more or less need a memory subsystem where all writable pages are refcounted and copy-on-write when the refcount is bigger than 1, and you need overcommit. Now these decisions aren't objectively bad , but they have significant trade-offs and it's probably not a go…

The nice thing about fork+exec is that's its simple and flexible.

To avoid the problems, see roc's comment under the article. Esp use of a zygote process.

Re: Moving beyond fork() + exec()

#283

Post by cryptonector »

rom1v wrote: post #36

Related to the discussion: "A fork() in the road": https://www.microsoft.com/en-us/research/wp-content/uploads/... > ABSTRACT > The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog t…

Ah, my one time on the HN front-page: Fork() is evil; vfork() is goodness; afork() would be better; clone() is stupid (https://news.ycombinator.com/item?id=30502392).

Re: Moving beyond fork() + exec()

#284

Post by cryptonector »

cylemons wrote: post #268
vbezhenar wrote: post #204

Earlier quoted context omitted.

Let's say you have 1GB RAM. You're running program that occupies 600 MB. Now this program wants to launch second small program that occupies 1 MB. You're doing fork + exec. If you're overcommiting, fork will not reserve another 600 MB, and exec immediately after fork will cause total system usage to be 601 MB. If you're not overcommiting, that fork will fail, because total memory consumption will be 1200 MB which is…

> If you're not overcommiting, that fork will fail, because total memory consumption will be 1200 MB which is more than 1GB. That somewhat restricts program design. Does this accounting apply to vfork as well?

Correct: it does NOT apply to vfork().

Re: Moving beyond fork() + exec()

#285

Post by cryptonector »

thayne wrote: post #215
Someone wrote: post #152

Earlier quoted context omitted.

> The problem with fork isn't really that it's slow. The problem is that if you want it to be not-slow, it locks you into a bunch of OS design decisions: you more or less need a memory subsystem where all writable pages are refcounted and copy-on-write when the refcount is bigger than 1 It may not be slow, but for the common case where fork is almost immediately followed by exec in the process where fork returns zero…

Another possible design is instead of forking the current process, you create a new empty process, then the parent calls syscalls to set up the new process, and eventually call exec on the child process. That does mean you either need new syscalls for that, or adapt existing syscalls to take a pidfd as an argument. That also solves some other problems with fork/exec where the default is to inherit a lot of things you…

https://gist.github.com/nicowilliams/a8a07b0fc75df05f684c23c...

Though I want a posix_spawn-as-a-system-call approach as well / instead of that.

Re: Moving beyond fork() + exec()

#286

Post by cryptonector »

thayne wrote: post #212
mort96 wrote: post #79

Earlier quoted context omitted.

The problem with fork isn't really that it's slow. The problem is that if you want it to be not-slow, it locks you into a bunch of OS design decisions: you more or less need a memory subsystem where all writable pages are refcounted and copy-on-write when the refcount is bigger than 1, and you need overcommit. Now these decisions aren't objectively bad , but they have significant trade-offs and it's probably not a go…

With large enough processes, like say a server JVM process that uses 10s of GBs of RAM, even just copying the page tables for CoW can be slow. And unless you have aggressive overcommit settings you can get an OOM on fork, even if you're just going to exec something small. vfork helps a little, but it has a lot of restrictions on what you can do before the exec, and on unix that's basically the only place you can do t…

vfork() helps a LOT. The restrictions on what you can do on the child-side of vfork() are pretty much the same ones as for fork() + you must not do anything to damage the stack frame of the vfork() caller (i.e., you can't return).

Re: Moving beyond fork() + exec()

#287

Post by cryptonector »

j16sdiz wrote: post #269
josefx wrote: post #186

Earlier quoted context omitted.

Isn't that what vfork tried to address? No COW, the child starts in its parents address space and only gets its own after calling exec.

Yes, the next sentence in TFA is: > Attempts (such as vfork()) have been made over the years to optimize for this case, but the pattern still is more expensive than it could be. Basically vfork do a "stop the world".

> Basically vfork do a "stop the world".

vfork() does NOT stop the world in many / most implementations. The ones that do stop the world do it because someone misunderstood the whole "vfork() stops the parent process" -- yes, it stops the parent process in a pre-threads world, but it doesn't have to stop any other threads but the one that called vfork(). Indeed, many implementations don't do that.

(Someone once tried to make NetBSD's vfork() stop the world because that's what the pre-threading man page said it does. I did my utter best to keep that from happening at the time, and it didn't then. Hopefully no one tried again later.)

Re: Moving beyond fork() + exec()

#288

Post by cryptonector »

__david__ wrote: post #77
uecker wrote: post #26

The elegance of the fork() + exec() model is that every kind of configuration can be done after the fork using all the usual APIs. Every attempt to replace it with a combined call that I have seen so far seemed fundamentally poorer because it needs to add all configuration options as parameters to the call and then do this in away that you can extend it later and does not become a mess.

I agree. I think the current way is very nice to use (in c). I think the best way would be to have something similar to vfork() but not bound by posix rules. Then make the normal posix apis (close, setuid, etc.) act like the Rust “builder” pattern. Possibly giving them a prefix for explicitness. That way the “fill out a giant structure” people could have their wish and the people that just want a faster posix experie…

> something similar to vfork() but not bound by posix rules

POSIX says nothing much about vfork() anymore. It was a mistake removing it. Zealots failed to understand that vfork() >> fork(). https://news.ycombinator.com/item?id=30502392

Re: Moving beyond fork() + exec()

#289

Post by cryptonector »

Animats wrote: post #123
rom1v wrote: post #36

Related to the discussion: "A fork() in the road": https://www.microsoft.com/en-us/research/wp-content/uploads/... > ABSTRACT > The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog t…

> The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. No, it was done that way so that you could launch a program that was too big to fit in memory with the parent program. The original implementation worked by swapping out the forking program to disk on a fork() call. Then, at the moment the program was swapped out but control had not returne…

Cygwin's fork() is similar to what you describe for QNX.

Re: Moving beyond fork() + exec()

#290

Post by cryptonector »

Animats wrote: post #123
rom1v wrote: post #36

Related to the discussion: "A fork() in the road": https://www.microsoft.com/en-us/research/wp-content/uploads/... > ABSTRACT > The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog t…

> The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. No, it was done that way so that you could launch a program that was too big to fit in memory with the parent program. The original implementation worked by swapping out the forking program to disk on a fork() call. Then, at the moment the program was swapped out but control had not returne…

> > The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design.

> No, it was done that way so that you could launch a program that was too big to fit in memory with the parent program.

Ironically vfork() is even better in this regard. I wish Unix had only ever had vfork().

Post reply on HN