Moving beyond fork() + exec() • Hacker News Board

Re: Moving beyond fork() + exec()

#171

Post by mort96 » Sat, Jun 06, 2026, 6:47 PM UTC

marcosdumay wrote: post #108

mort96 wrote: post #79

Earlier quoted context omitted.

The problem with fork isn't really that it's slow. The problem is that if you want it to be not-slow, it locks you into a bunch of OS design decisions: you more or less need a memory subsystem where all writable pages are refcounted and copy-on-write when the refcount is bigger than 1, and you need overcommit. Now these decisions aren't objectively bad , but they have significant trade-offs and it's probably not a go…

CoW is probably a good idea whether you use fork or not. Or rather, fork is probably a better option than just exec exactly because it can benefit from CoW. At least on systems with virtual addressing. If you want to go into physical addressing, then yes, maybe it's a problem. But Linux will never touch anything with physical addressing, so I don't see what people are complaining about.

CoW is probably a good idea regardless, yeah. Overcommit is more questionable. Regardless, both ought to be argued based on their own merits. It's unfortunate that both are necessary as a consequence of fork().

Re: Moving beyond fork() + exec()

#172

Post by loeg » Sat, Jun 06, 2026, 6:49 PM UTC

asveikau wrote: post #135

The things you can do between fork and exec are sometimes underestimated. Off the top of my head, you can call dup2(), you can set a process group id, probably a few other things. If you contrast that with win32, where you optionally pack a bunch of initial values into a struct, win32 is a much more narrow, less pleasant, less freeform interface, where it is harder to introduce more features. But I think there is alr…

> The things you can do between fork and exec are sometimes underestimated. Off the top of my head, you can call dup2(), you can set a process group id, probably a few other things.

What do you mean underestimated? You can do anything between fork and exec; there are no limitations.

Re: Moving beyond fork() + exec()

#173

Post by dcrazy » Sat, Jun 06, 2026, 6:55 PM UTC

asveikau wrote: post #135

The things you can do between fork and exec are sometimes underestimated. Off the top of my head, you can call dup2(), you can set a process group id, probably a few other things. If you contrast that with win32, where you optionally pack a bunch of initial values into a struct, win32 is a much more narrow, less pleasant, less freeform interface, where it is harder to introduce more features. But I think there is alr…

posix_spawn is emulated on Linux, but it is a native syscall on macOS (and possibly other OSes?). As discussed in the linked article, there is interest in changing Linux to adopt this model, where posix_spawn is its own fundamental primitive.

Re: Moving beyond fork() + exec()

#174

Post by dcrazy » Sat, Jun 06, 2026, 6:56 PM UTC

loeg wrote: post #172

asveikau wrote: post #135

The things you can do between fork and exec are sometimes underestimated. Off the top of my head, you can call dup2(), you can set a process group id, probably a few other things. If you contrast that with win32, where you optionally pack a bunch of initial values into a struct, win32 is a much more narrow, less pleasant, less freeform interface, where it is harder to introduce more features. But I think there is alr…

> The things you can do between fork and exec are sometimes underestimated. Off the top of my head, you can call dup2(), you can set a process group id, probably a few other things. What do you mean underestimated? You can do anything between fork and exec; there are no limitations.

That’s not true. man 7 signal-safety

Re: Moving beyond fork() + exec()

#175

Post by JdeBP » Sat, Jun 06, 2026, 6:59 PM UTC

dcrazy wrote: post #120

JdeBP wrote: post #109

Earlier quoted context omitted.

As mentioned elsewhere on this page, Windows NT had fork from the start. Vide NtCreateProcess and what happens if an image file is not explicitly supplied. * https://computernewb.com/~lily/files/Documents/NTDesignWorkb...

NtCreateProcess doesn’t accept an image file parameter.

You haven't read the doco. I did point to some. The image file is supplied (or not) via the section object.

Think it through. Windows NT supported fork from the start in its POSIX subsystem, that subsystem was layered on top of the Native API, and this is the Native API mechanism that the POSIX subsystem employed. Although it took until Gary Nebbett for someone to publicly show how, even though people knew informally back in 1993.

Re: Moving beyond fork() + exec()

#176

Post by jerf » Sat, Jun 06, 2026, 7:01 PM UTC

dcrazy wrote: post #169

jerf wrote: post #31

Earlier quoted context omitted.

A thing that makes that complicated is that while you want that conceptually, you don't want that in reality. For instance, if the spawning process is in a container of some sort and it spawned a process that "shares nothing with the process that spawned it", the spawned process would no longer be in that container, because the state of "being in the container" is one of the things it shares with the parent process.…

It’s such a bad idea that every OS except Linux implements it? On macOS it’s posix_spawn, on Windows it’s NtCreateProcess.

Who said anything about it being a "bad idea"?

I also explicitly said this wasn't unsolvable. My point isn't about technical implementations or code, my point is that the casual "I want to share nothing about the parent process" thought in sanderj's mind, and presumably a lot others, is much more ill-defined than they realize. There's a lot more state that a process has than what file descriptors are open in a modern system.

Moreover, as things like "in which container is this running" demonstrate, those are also not "create a process that has nothing to do with this process", because, again, there's a lot more to "having to do with this process" than "what file descriptors are open".

Also, as the name might have been a clue, Linux has posix_spawn: https://linux.die.net/man/3/posix_spawn. It also has a thing called "clone": https://www.man7.org/linux/man-pages/man2/clone.2.html Nor do I claim this paragraph is an entire overview of all the ways of starting a process in Linux. If you want to understand what I mean by "lots of details in a modern OS", your assignment is to carefully read the entire "clone" man page, and you'll start to see what I mean, though I'm not sure even that is all the state associated with a process nowadays.

Re: Moving beyond fork() + exec()

#177

Post by tadfisher » Sat, Jun 06, 2026, 7:03 PM UTC

skydhash wrote: post #143

redleader55 wrote: post #127

Earlier quoted context omitted.

When cores start needing more than 9 bits to be represented and RAM is in terabytes, many of the old assumptions need to change. Schedulers need to be implemented in userspace, RAM needs to be allocated in GB, not in 4k, io needs to require less round-trips between kernel and user space and NICs need to do a lot more work before the data reaches the CPU.

Does it need to be the same OS? Most consumer device are in the low 16GB range for memory with some outliers in the 64 and 128 GB. 32 cores are still in the realm of specialized devices. Yes, we’re not the one paying for Linux development, but its subsystems are so complicated for general purpose computing. Like fitting formula 1 car parts onto a camry.

Our software is littered with the consequences of these kinds of assumptions, and they have an impact on consumer use cases.

x86 still runs in real mode on boot despite dropping the PC BIOS.

Lots of software still assumes a 4kb page size, to the point where migrating Android to 16kb is an ongoing multi-year effort involving far too many people. And this is an OS for phones, which you might assume would lack the memory to benefit from a larger page size.

And one of the most popular consumer CPUs for enthusiasts, the Ryzen X3D chips, broke assumptions in both Linux and Windows schedulers that all cores have access to the same amount of L3 cache.

I would probably not assume the kinds of hardware limitations that we have now will persist into the useful lifetime of current software. Splitting the OS into "consumer" and "enterprise" variants is one of those moves that would bake in a ton of assumptions and make things messier in the future.

Re: Moving beyond fork() + exec()

#178

Post by uecker » Sat, Jun 06, 2026, 7:03 PM UTC

trumpdong wrote: post #165

jcranmer wrote: post #144

Earlier quoted context omitted.

Calling that elegant is a path dependence of the history of fork+exec. In an alternative world where fork+exec never existed, a lot of those "usual APIs" would probably have had an explicit pid argument to them that let you modify process configuration from a different process. (This is how Fuschia works, e.g.). There's a lot of benefit to this world: the most obvious is that you don't have to magic up some IPC syste…

Or you could call ptrace_syscall (that doesn't currently exist) on your child processes that you are tracing because you'd always be tracing them by default, or get an io_uring for the child process, or...

A ptrace_syscall would be interesting and would seem to be a full replacement for having the pid argument everywhere.

But frankly, I am not really seeing the value.

Re: Moving beyond fork() + exec()

#179

Post by dcrazy » Sat, Jun 06, 2026, 7:03 PM UTC

jerf wrote: post #176

dcrazy wrote: post #169

Earlier quoted context omitted.

It’s such a bad idea that every OS except Linux implements it? On macOS it’s posix_spawn, on Windows it’s NtCreateProcess.

Who said anything about it being a "bad idea"? I also explicitly said this wasn't unsolvable. My point isn't about technical implementations or code, my point is that the casual "I want to share nothing about the parent process" thought in sanderj's mind, and presumably a lot others, is much more ill-defined than they realize. There's a lot more state that a process has than what file descriptors are open in a modern…

Linux posix_spawn is a wrapper around clone and exec. There is no primitive on Linux to create an entirely blank process. This is adequately discussed in the linked LWN post.

Other operating systems either have parallel APIs to fork (e.g. the posix_spawn syscall on macOS) or do not provide fork at all (Windows).

Re: Moving beyond fork() + exec()

#180

Post by pjc50 » Sat, Jun 06, 2026, 7:05 PM UTC

uecker wrote: post #26

The elegance of the fork() + exec() model is that every kind of configuration can be done after the fork using all the usual APIs. Every attempt to replace it with a combined call that I have seen so far seemed fundamentally poorer because it needs to add all configuration options as parameters to the call and then do this in away that you can extend it later and does not become a mess.

The flip side of this is that you have to be aware of the entire state of the process, including everything done in libraries, in order to correctly start a new process.

Quick, what's the highest numbered open file descriptor in the your program?

This gets even worse if you have multiple threads running. Without looking it up, what is the state of all the various synchronization primitives in a forked process?