Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving beyond fork() + exec()

lwn.net

Post reply on HN
Print view View original

81–90 of 358 posts

Re: Moving beyond fork() + exec()

#81

Post by omoikane »

lokar wrote: post #15

This seems unnecessary to me. In the example, the core of git should be a library yo can link so you don't need to run the binary. That would be better in every way.

Launching git repeatedly was probably not the best example. But it's hard to think of good examples where launching processes repeatedly is the most performant thing to do, probably because launching processes had been expensive and everyone has learned to do something else (libraries, zygotes, etc). Maybe a different question is: if launching processes were cheap, is there something we would implement as processes instead of libraries?

I can recall just one program that's intentionally not implemented as a library, but I think people have since built a library on top of it:

https://dechifro.org/dcraw/#:~:text=Why%20don%27t%20you%20im...

Re: Moving beyond fork() + exec()

#82

Post by mort96 »

FooBarWidget wrote: post #45
mrkeen wrote: post #35

> fork() is a relatively expensive system call; it must copy the entire process state (including memory) for the child process. Many optimizations have been made over the years, but a fork is still a fundamentally costly operation. To make things worse, a fork() call is often immediately followed by an exec(), which will discard all of that memory that was so carefully copied for the child. It's weird to leave out a…

It says state. Copy on write still means it's O(number of page table entries) even if you don't copy the contents. It's a well known issue that forking a program with large virtual memory size is slow.

It says "(including memory)". It's pretty natural to read this as "(including the contents of allocated pages)".

Re: Moving beyond fork() + exec()

#83

Post by __david__ »

amluto wrote: post #29
uecker wrote: post #26

The elegance of the fork() + exec() model is that every kind of configuration can be done after the fork using all the usual APIs. Every attempt to replace it with a combined call that I have seen so far seemed fundamentally poorer because it needs to add all configuration options as parameters to the call and then do this in away that you can extend it later and does not become a mess.

I have the entirely opposite opinion. IMO a big mistake of the UNIXy model is that so much state is preserved across the creation of a process. For example, there are APIs to have a specific thing be fd number 4 so you can run a program and have it find that thing at fd 4. This is weird . Windows, for all its many, many faults, did not use fork+exec and instead mostly has options for how one creates a process. It was…

Having fd 4 mean something specific is no weirder than having fds 0,1, and 2 mean something specific, which is probably never going to change. At some point you just gotta embrace the Unix.

Re: Moving beyond fork() + exec()

#84

Post by vlovich123 »

pizlonator wrote: post #57
rom1v wrote: post #36

Related to the discussion: "A fork() in the road": https://www.microsoft.com/en-us/research/wp-content/uploads/... > ABSTRACT > The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog t…

Fork is marvelous for the zygote pattern Hard to come up with an optimization that is equally efficient and elegant

The paper explicitly covers it that various memory COW/snapshot mechanisms are probably faster and safer than the zygote pattern. As it stands getting the zygote pattern correct and safe is something you have to plan for upfront. You can’t retrofit it which is why the paper mentions it has poor composability. Also the advantages of the zygote pattern can be overstated since the memory sharing benefit is minimal since it has to happen so early and modern OSes already transparently CoW duplicate pages in the background.

Re: Moving beyond fork() + exec()

#85

Post by 201984 »

monocasa wrote: post #34
sjmulder wrote: post #28

Earlier quoted context omitted.

Doesn't the loaded code have to be patched for relocations?

Not on modern archs that provide decent support for PIE (position independent executables).

How do you think position independent code can call functions from other .so's without being patched with their addresses?

They can't, so even PIC code still has to have a relocation table that gets patched. It's in a different page than the code though, so code does still get reused.

Re: Moving beyond fork() + exec()

#86

Post by pizlonator »

toast0 wrote: post #74
pizlonator wrote: post #57

Earlier quoted context omitted.

Fork is marvelous for the zygote pattern Hard to come up with an optimization that is equally efficient and elegant

The zygote pattern[1] is a great optimization to deal with the cost of forking, but IMHO, being able to inexpensively spawn a carefully tailored process regardless of the size and scope of the current process would be better. I would guess it would be a small difference in measurable performance between zygote and a direct clean spawn, but it's one less trick an application needs to do, and it would be very helpful f…

You're referring to something else, and maybe I'm using the term "zygote" incorrectly.

In all uses of zygotes that I have seen, here's what's really happening:

- `fork` is being used to reduce the cost of starting a process that has a high start-up cost. So, you start one process, run it through the expensive initialization, and then fork it from there to start new processes.

- To make this even faster, you have a pool of pre-forked processes sit around.

- Having pre-forked processes sitting around ready to be used is not expensive because of the CoW property and the fact that a process that forks and then immediately pauses will not have triggered any significant CoW yet.

So, the zygote optimization you speak of is in practice only meaningful on top of systems that are using an optimization uniquely enabled by `fork` (avoiding process initialization costs by cloning a process), and that zygote optimization is further optimized by another property of `fork` (memory sharing of forked processes that haven't done anything else yet).

Re: Moving beyond fork() + exec()

#87

Post by Panzerschrek »

The whole approach of using fork seems to be unnatural for me. In many cases (even in the majority of them) it's not needed to inherit the whole structure of the parent process, but to start a given executable. Windows does this better with its CreateProcessW interface.

Re: Moving beyond fork() + exec()

#88

Post by ajkjk »

Fork always seemed conceptually terrible even when I first learned about it.. If you want to do one thing (start a process) you should not have to use a mysterious incantation that does a different unrelated thing (forks your process) in order to do it.

I am curious about what the best way to handle the example in the article of one process spawning many git subprocesses is. Surely it just doesn't make sense to repeatedly start git from scratch in the course of a long-running parent operation. What's the low cost abstraction for the same result, though?

Re: Moving beyond fork() + exec()

#89

Post by toast0 »

pizlonator wrote: post #86
toast0 wrote: post #74

Earlier quoted context omitted.

The zygote pattern[1] is a great optimization to deal with the cost of forking, but IMHO, being able to inexpensively spawn a carefully tailored process regardless of the size and scope of the current process would be better. I would guess it would be a small difference in measurable performance between zygote and a direct clean spawn, but it's one less trick an application needs to do, and it would be very helpful f…

You're referring to something else, and maybe I'm using the term "zygote" incorrectly. In all uses of zygotes that I have seen, here's what's really happening: - `fork` is being used to reduce the cost of starting a process that has a high start-up cost. So, you start one process, run it through the expensive initialization, and then fork it from there to start new processes. - To make this even faster, you have a po…

Oh I see. I guess your zygotes have developed more than mine. I think Google may have coined or at least popularized the term zygote for this in Chrome and Android, Chrome documentation [1] says:

> A zygote process is one that listens for spawn requests from a main process and forks itself in response. Generally they are used because forking a process after some expensive setup has been performed can save time and share extra memory pages.

I think reading the first sentance and stopping covers my zygote, but adding the second sentance covers yours. So I think we're both right!

I think both paths are useful. If your children need time to startup and become ready, spawn one that does start up work, and then it (pre)forks at the ready state to have processes ready to handle requests (your zygote). This does require a traditional fork() to avoid duplication of work.

But if forking is expensive at runtime because you have a million FDs open and a whole lot of memory allocations, spawn spawners before you start doing work (my zygote). This could be unnecessary with a inexpensive way to spawn a new process from an process that has lots of resources in use.

Of course, you can also use my zygotes to spawn your zygotes. Zygoteception.

[1] https://chromium.googlesource.com/chromium/src/+/HEAD/docs/l...

Re: Moving beyond fork() + exec()

#90

Post by 7jjjjjjj »

sanderjd wrote: post #17

I just ran into this recently, where I had an obscure bug caused by needing to close more file descriptors in the forked process. "I want a clone of the current process" is just way less common in my experience than "I want a completely new process". It feels crazy that we don't have a way to directly express the latter thing, and can only approximate it by cloning and then fixing things up in post.

>It feels crazy that we don't have a way to directly express the latter thing

Isn't that what posix_spawn is for?

Post reply on HN