Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving beyond fork() + exec()

lwn.net

Post reply on HN
Print view View original

351–358 of 358 posts

Re: Moving beyond fork() + exec()

#351

Post by vbezhenar »

cylemons wrote: post #268
vbezhenar wrote: post #204

Earlier quoted context omitted.

Let's say you have 1GB RAM. You're running program that occupies 600 MB. Now this program wants to launch second small program that occupies 1 MB. You're doing fork + exec. If you're overcommiting, fork will not reserve another 600 MB, and exec immediately after fork will cause total system usage to be 601 MB. If you're not overcommiting, that fork will fail, because total memory consumption will be 1200 MB which is…

> If you're not overcommiting, that fork will fail, because total memory consumption will be 1200 MB which is more than 1GB. That somewhat restricts program design. Does this accounting apply to vfork as well?

As I understand it, the whole purpose of vfork is to avoid that problem. So vfork does not copy memory in any way and you're not allowed to do anything but exec after vfork. But at this point question arises: why not call it `vfork_and_exec` and get rid of undefined behaviour. Or choose better name like `CreateProcessW` hehe

Re: Moving beyond fork() + exec()

#352

Post by trumpdong »

eggnet wrote: post #323
trumpdong wrote: post #154

Earlier quoted context omitted.

Which argument to clone starts the process with an empty address space?

That happens with execve(). clone() allows you to not copy the page table prior to the execve() call.

Which argument to clone does that?

Re: Moving beyond fork() + exec()

#353

Post by dooglius »

Animats wrote: post #123
rom1v wrote: post #36

Related to the discussion: "A fork() in the road": https://www.microsoft.com/en-us/research/wp-content/uploads/... > ABSTRACT > The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog t…

> The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. No, it was done that way so that you could launch a program that was too big to fit in memory with the parent program. The original implementation worked by swapping out the forking program to disk on a fork() call. Then, at the moment the program was swapped out but control had not returne…

Fork isn't necessary for this, you could just exec directly?

Re: Moving beyond fork() + exec()

#354

Post by MayCXC »

nyrikki wrote: post #146
codedokode wrote: post #141

The problem with replacing exec/fork is that you usually want to configure new process: for example, set up signal handlers, close or open FDs, switch namespaces, setup seccomp, adjust permissions. And all the system calls to do it apply only to the current process and you need something to replace them. The proposal in the article was to create a new API for this. My idea is that we could make a new syscall, for exa…

Luckily someone with a time machine saw your post and added it to POSIX.1-2001 :) (Sorry if you weren't joking) but yes, posix_spawn() has been a thing and in glibc fork is just a alias to clone() Not exactly that OP idea, but fork/exec is legacy really.

other people in the thread say that posix_spawn is more or less implemented as a fork+exec wrapper though? it sounds like the idea is more like if there were a separate deferred_fork that made an intermediate "process factory" that let you set up a process without actually creating a new one until the exec. obviously the if() construct would have to be replaced with an in-process handle that mimics calls to the posix api.

Re: Moving beyond fork() + exec()

#355

Post by thayne »

cryptonector wrote: post #286
thayne wrote: post #212

Earlier quoted context omitted.

With large enough processes, like say a server JVM process that uses 10s of GBs of RAM, even just copying the page tables for CoW can be slow. And unless you have aggressive overcommit settings you can get an OOM on fork, even if you're just going to exec something small. vfork helps a little, but it has a lot of restrictions on what you can do before the exec, and on unix that's basically the only place you can do t…

vfork() helps a LOT. The restrictions on what you can do on the child-side of vfork() are pretty much the same ones as for fork() + you must not do anything to damage the stack frame of the vfork() caller (i.e., you can't return).

> the behavior is undefined if the process created by vfork() either modifies any data other than a variable of type pid_t used to store the return value from vfork(), or returns from the function in which vfork() was called, or calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions

That's a lot more restrictive. You can't use local variables, or call any functions other than _exit or execve. On linux specifically, I _think_ those restrictions are more relaxed and you can call async-signal-safe functions, however I'm not entirely clear on how relaxed that is, and as far as I understand that isn't portable.

Re: Moving beyond fork() + exec()

#356

Post by cryptonector »

thayne wrote: post #355
cryptonector wrote: post #286

Earlier quoted context omitted.

vfork() helps a LOT. The restrictions on what you can do on the child-side of vfork() are pretty much the same ones as for fork() + you must not do anything to damage the stack frame of the vfork() caller (i.e., you can't return).

> the behavior is undefined if the process created by vfork() either modifies any data other than a variable of type pid_t used to store the return value from vfork(), or returns from the function in which vfork() was called, or calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions That's a lot more restrictive. You can't use local variables, or call any functions oth…

But some of that is nonsense and incorrect. You can very much use local variables, and you'll find tons of vfork()-using code that does that and calls plenty of async-signal-safe functions.

The real restrictions are:

  - you can't damage the function call frame
    of the caller of vfork(), thus you can't
    return from it

  - you may only call async-signal-safe
    functions on the child side of vfork()
That's basically it. Yes, you'll want to call execve(2) or _exit(2) before long, but there is no time limit as to that, it's just that the whole point of calling vfork() is to make it real cheap to spawn a process, which means ultimately calling execve(2), with _exit(2) being what you do if it execve(2) fails (e.g., because ENOENT).

There is a ton of vfork()-using code that adheres to these real restrictions and has been working fine for decades. That includes several posix_spawn() implementations, the C shell, etc.

I demand evidence that this part: "the behavior is undefined if the process created by vfork() either modifies any data other than a variable of type pid_t used to store the return value from vfork()" is remotely true. That evidence must be of the form of bug reports that were accepted and which stand to scrutiny.

I've never found any such evidence. Have you?

Meanwhile I have a proof by existence that vfork() is safe used much more liberally than you say it may be used.

> You can't use local variables, or call any functions other than _exit or execve.

There are other async-signal-safe functions, and they get used routinely by posix_spawn() and other code to do child-side setup before execve(2), including: I/O redirection, process group setup, signal handling changes, etc.

Re: Moving beyond fork() + exec()

#357

Post by epcoa »

JdeBP wrote: post #109
epcoa wrote: post #93

Earlier quoted context omitted.

This is not true. NT never had fork, was always based on the assumption of an MMU and Dave Cutler was a well known fork hater in the 80s long before this paper came out and made it cool to be so. By the time Windows 95 was out, the baseline was 386 with an MMU. CreateThread was initially designed for NT in 1993 though (which didn’t support pre-386 CPUs).

As mentioned elsewhere on this page, Windows NT had fork from the start. Vide NtCreateProcess and what happens if an image file is not explicitly supplied. * https://computernewb.com/~lily/files/Documents/NTDesignWorkb...

NtCreateProcess was not a public Windows API. NT was flexible, that’s not what was being discussed, which should have been clear from the context.

Re: Moving beyond fork() + exec()

#358

Post by cylemons »

vbezhenar wrote: post #351
cylemons wrote: post #268

Earlier quoted context omitted.

> If you're not overcommiting, that fork will fail, because total memory consumption will be 1200 MB which is more than 1GB. That somewhat restricts program design. Does this accounting apply to vfork as well?

As I understand it, the whole purpose of vfork is to avoid that problem. So vfork does not copy memory in any way and you're not allowed to do anything but exec after vfork. But at this point question arises: why not call it `vfork_and_exec` and get rid of undefined behaviour. Or choose better name like `CreateProcessW` hehe

In the manpage for vfork, Linux in particular

> As with fork(2), the child process created by vfork() inherits copies of various of the caller's process attributes (e.g., file descriptors, signal dispositions, and current working directory); the vfork() call differs only in the treatment of the virtual address space, as described above.

so it seems Linux does define the behavior of vfork, but if you rely on it, your code won't be portable to other POSIX systems

https://man7.org/linux/man-pages/man2/vfork.2.html

Post reply on HN