Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving beyond fork() + exec()

lwn.net

Post reply on HN
Print view View original

271–280 of 358 posts

Re: Moving beyond fork() + exec()

#271

Post by gmueckl »

1718627440 wrote: post #39
sanderjd wrote: post #17

I just ran into this recently, where I had an obscure bug caused by needing to close more file descriptors in the forked process. "I want a clone of the current process" is just way less common in my experience than "I want a completely new process". It feels crazy that we don't have a way to directly express the latter thing, and can only approximate it by cloning and then fixing things up in post.

But you generally want to communicate with that process, so you do need to setup e.g. file descriptors and stuff, which needs information from the parent process to be passed.

A variant of exec could take an initial table of file descriptors in the current process that get cloned into the new child. Pipe creation could also get rolled into this mechanism. That should take care of the most obvious leaky bit of fork()/exec(), at least.

Re: Moving beyond fork() + exec()

#272

Post by BobbyTables2 »

Animats wrote: post #123
rom1v wrote: post #36

Related to the discussion: "A fork() in the road": https://www.microsoft.com/en-us/research/wp-content/uploads/... > ABSTRACT > The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog t…

> The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. No, it was done that way so that you could launch a program that was too big to fit in memory with the parent program. The original implementation worked by swapping out the forking program to disk on a fork() call. Then, at the moment the program was swapped out but control had not returne…

The QNX approach is also pretty much how the dynamic linker loads shared libraries today in Linux .

“An era of really expensive memory”. That sounds familiar…

Re: Moving beyond fork() + exec()

#273

Post by dcrazy »

peterfirefly wrote: post #256
dcrazy wrote: post #161

Earlier quoted context omitted.

NT was designed to be platform-agnostic, and its original target was the DEC Alpha. Its process model owes nothing to pre-386 CPUs. The WinAPI CreateProcess function is a layer atop NtCreateProcess, so that is where the pre-386 heritage lives. But even the WinAPI process model changed significantly with 32-bit Windows.

No. https://en.wikipedia.org/wiki/Windows_NT#Development Windows NT was developed on various different CPUs before the Alpha was a thing. When it was released in 1993, it was released for three CPUs: IA-32, MIPS, and Alpha.

Sorry, I had conflated Windows NT development with development of 64-bit Windows as told by Raymond Chen: https://learn.microsoft.com/en-us/previous-versions/technet-...

Raymond also says elsewhere that most WinNT engineers did development on i386, but doesn’t explicitly say what time period he is describing: https://devblogs.microsoft.com/oldnewthing/20250513-00/?p=11...

Re: Moving beyond fork() + exec()

#274

Post by infogulch »

dcrazy wrote: post #221
thayne wrote: post #215

Earlier quoted context omitted.

Another possible design is instead of forking the current process, you create a new empty process, then the parent calls syscalls to set up the new process, and eventually call exec on the child process. That does mean you either need new syscalls for that, or adapt existing syscalls to take a pidfd as an argument. That also solves some other problems with fork/exec where the default is to inherit a lot of things you…

Syscalls aren’t all that cheap either.

io_uring taught us that if syscalls are expensive, queue them up in a buffer with one syscall to transfer the thread to the os to process it. So, queue up the new process mutations in a buffer with a single syscall to process all of them in a batch. This model should have replaced repetitive syscalls across the kernel years ago.

Re: Moving beyond fork() + exec()

#275

Post by fc417fc802 »

dcrazy wrote: post #148
Animats wrote: post #123

Earlier quoted context omitted.

> The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. No, it was done that way so that you could launch a program that was too big to fit in memory with the parent program. The original implementation worked by swapping out the forking program to disk on a fork() call. Then, at the moment the program was swapped out but control had not returne…

Don’t pretty much all OSes implement process startup in userspace? On macOS, the kernel creates a process with an image of dyld and points it at dyld_start, which actually takes care of parsing the Mach-O header. I assumed ld.so does the same job on Linux.

Yes, it can all be done in userspace. When the "fork in the road" paper came up a while back someone linked to an example. https://grugq.github.io/docs/ul_exec.txt

Re: Moving beyond fork() + exec()

#276

Post by thayne »

dcrazy wrote: post #221
thayne wrote: post #215

Earlier quoted context omitted.

Another possible design is instead of forking the current process, you create a new empty process, then the parent calls syscalls to set up the new process, and eventually call exec on the child process. That does mean you either need new syscalls for that, or adapt existing syscalls to take a pidfd as an argument. That also solves some other problems with fork/exec where the default is to inherit a lot of things you…

Syscalls aren’t all that cheap either.

This true, but these methods don't increase the number of syscalls you need to make.

Re: Moving beyond fork() + exec()

#277

Post by fc417fc802 »

thayne wrote: post #215
Someone wrote: post #152

Earlier quoted context omitted.

> The problem with fork isn't really that it's slow. The problem is that if you want it to be not-slow, it locks you into a bunch of OS design decisions: you more or less need a memory subsystem where all writable pages are refcounted and copy-on-write when the refcount is bigger than 1 It may not be slow, but for the common case where fork is almost immediately followed by exec in the process where fork returns zero…

Another possible design is instead of forking the current process, you create a new empty process, then the parent calls syscalls to set up the new process, and eventually call exec on the child process. That does mean you either need new syscalls for that, or adapt existing syscalls to take a pidfd as an argument. That also solves some other problems with fork/exec where the default is to inherit a lot of things you…

> you create a new empty process, then the parent calls syscalls to set up the new process ...

That does seem like a much better design to me. But I wonder if that was considered way back at the dawn of computing and rejected for good reason?

> I think this is technically possible on linux, but there isn't a readily available interface for it.

Yes there is, see `man clone`. POSIX and glibc are quite different from the kernel in this regard. AFAIK under linux there are just threads of execution that might or might not share various namespaces and memory mappings. That said, the kernel does place a few artificial restrictions on what combinations are allowed in order to (as I understand it) guard against the unintended exercise of entirely untested combinations that serve no known practical purpose.

The practical problem is that if you start doing as you please with the various namespaces and mappings you quickly become incompatible with glibc and by extension most likely the majority of the dynamic libraries available on your system.

Re: Moving beyond fork() + exec()

#279

Post by Too »

kps wrote: post #224
ajkjk wrote: post #88

Fork always seemed conceptually terrible even when I first learned about it.. If you want to do one thing (start a process) you should not have to use a mysterious incantation that does a different unrelated thing (forks your process) in order to do it. I am curious about what the best way to handle the example in the article of one process spawning many git subprocesses is. Surely it just doesn't make sense to repea…

Fork is conceptually simple. Without bringing in any other layers, you start a process with the one thing known to exist: yourself. Otherwise you need multiple steps to create a process, fill it with something to run, and arrange for it to execute. Or like Win32 you permanently smush them together with other layers, like filesystems and object loaders and linkers.

Fill with what stuff exactly?

The only thing I want to inherit from the parent process is its cwd and environment variables, even those are often overridden. The rest can easily be passed explicitly through other channels like pipes or command line arguments.

Back to the example from the article. It makes no sense that a git-subprocess forked from a web server need to have any process state inherited from the web server.

Re: Moving beyond fork() + exec()

#280

Post by ajkjk »

kps wrote: post #224
ajkjk wrote: post #88

Fork always seemed conceptually terrible even when I first learned about it.. If you want to do one thing (start a process) you should not have to use a mysterious incantation that does a different unrelated thing (forks your process) in order to do it. I am curious about what the best way to handle the example in the article of one process spawning many git subprocesses is. Surely it just doesn't make sense to repea…

Fork is conceptually simple. Without bringing in any other layers, you start a process with the one thing known to exist: yourself. Otherwise you need multiple steps to create a process, fill it with something to run, and arrange for it to execute. Or like Win32 you permanently smush them together with other layers, like filesystems and object loaders and linkers.

I gues that way of thinking makes sense if you have a certain model of what a process is, in terms of the data structures and runtime state etc. But, tbh, I think of processes as glorified function calls, which happen to have that stuff involved as an implementation detail. And if spawning a process call is supposed to act like a function call, then of course it should not inherit state. You should call the function you want to call, not call yourself with an instruction to switch over to it instead.
Post reply on HN