Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving beyond fork() + exec()

lwn.net

Post reply on HN
Print view View original

191–200 of 358 posts

Re: Moving beyond fork() + exec()

#191

Post by spacechild1 »

ajkjk wrote: post #88

Fork always seemed conceptually terrible even when I first learned about it.. If you want to do one thing (start a process) you should not have to use a mysterious incantation that does a different unrelated thing (forks your process) in order to do it. I am curious about what the best way to handle the example in the article of one process spawning many git subprocesses is. Surely it just doesn't make sense to repea…

Yeah, as someone who originally came from Windows, the fork+exec model never made sense to me. Now I know it's just a historical quirk, but for some reason there are still people who pretend that fork+exec is actually a good thing...

Re: Moving beyond fork() + exec()

#192

Post by thamer »

mrkeen wrote: post #35

> fork() is a relatively expensive system call; it must copy the entire process state (including memory) for the child process. Many optimizations have been made over the years, but a fork is still a fundamentally costly operation. To make things worse, a fork() call is often immediately followed by an exec(), which will discard all of that memory that was so carefully copied for the child. It's weird to leave out a…

Redis is the kind of process where this matters a lot, and while fork() doesn't copy the memory, it still needs to copy the page table. For a process holding tens of GBs of RAM, fork() can take a long time, and there's one every time Redis dumps its .rdb file or rewrites its binary log ("AOF").

Even back in 2012 this blog post showed the high cost of this operation: https://redis.io/blog/testing-fork-time-on-awsxen-infrastruc...

On an m2.xlarge using ~25GB of RAM, fork() took 5.67 seconds. That's a long pause when Redis clients typically experience single-digit msec latency for most operations. Yes, that's only the time needed to copy the page table. It's surprising they don't mention huge pages, it seems like it would be a key consideration here.

No doubt hardware is faster 14 years later, but Redis instances likely use more RAM too. It'd be interesting to see this benchmark revisited.

Re: Moving beyond fork() + exec()

#193

Post by adgjlsfhk1 »

mort96 wrote: post #79
anarazel wrote: post #58

Earlier quoted context omitted.

It is somewhat interesting that the most widely used "big" OS that doesn't use fork, i.e. Windows, has dog slow process creation... I agree that there should be non-fork primitives, I'm just not that sure that performance is the best argument.

The problem with fork isn't really that it's slow. The problem is that if you want it to be not-slow, it locks you into a bunch of OS design decisions: you more or less need a memory subsystem where all writable pages are refcounted and copy-on-write when the refcount is bigger than 1, and you need overcommit. Now these decisions aren't objectively bad , but they have significant trade-offs and it's probably not a go…

One os level thing that is interesting to me is if it would be possible/wise to make an OS based on (concurrent) garbage collection.

Re: Moving beyond fork() + exec()

#194

Post by pjmlp »

mort96 wrote: post #164
pjmlp wrote: post #149

Earlier quoted context omitted.

True, but on Windows the approach is then to use COM servers, which have a faster IPC model, and can even serve multiple clients, depending on how the appartement space is configured.

"Faster IPC model" than what? Faster than writing to and reading from a pipe? Faster than POSIX shared memory?

Than UNIX fork/exec model, or calling into Create Process all the time.

Windows has a more rich set of IPC stuff than POSIX, especially since it has a microkernel like design.

If you are going to say it is everything on the same memory space anyway, it isn't.

Optional on Windows 10, and enforced on Windows 11, Hyper-V is always running, and several components including kernel and driver modules are sandboxed into their little worlds.

Several additional sandboxing changes were announced at BUILD.

Re: Moving beyond fork() + exec()

#195

Post by mort96 »

pjmlp wrote: post #194
mort96 wrote: post #164

Earlier quoted context omitted.

"Faster IPC model" than what? Faster than writing to and reading from a pipe? Faster than POSIX shared memory?

Than UNIX fork/exec model, or calling into Create Process all the time. Windows has a more rich set of IPC stuff than POSIX, especially since it has a microkernel like design. If you are going to say it is everything on the same memory space anyway, it isn't. Optional on Windows 10, and enforced on Windows 11, Hyper-V is always running, and several components including kernel and driver modules are sandboxed into the…

fork/exec is not an IPC model...

Re: Moving beyond fork() + exec()

#196

Post by pjmlp »

knome wrote: post #139
pjmlp wrote: post #70

Earlier quoted context omitted.

Because that OS best practices is to use threads. Traditionally Windows applications that create processes all the time come from UNIX heritage. Contrary to UNIX, Windows NT was designed with threads first mentality, from the get go. While on UNIX they were added after fact, and to this day there are gotchas mixing posix threads with signals, fork and exec.

the only difference between a thread and a process on linux is how many structures they share. the function is identical.

Agreed, however not all UNIXes are like Linux.

Re: Moving beyond fork() + exec()

#197

Post by bluepuma77 »

Animats wrote: post #123
rom1v wrote: post #36

Related to the discussion: "A fork() in the road": https://www.microsoft.com/en-us/research/wp-content/uploads/... > ABSTRACT > The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog t…

> The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. No, it was done that way so that you could launch a program that was too big to fit in memory with the parent program. The original implementation worked by swapping out the forking program to disk on a fork() call. Then, at the moment the program was swapped out but control had not returne…

> It was needed back in the era of really expensive memory.

Well, it seems we are back in an era with really expensive memory.

Re: Moving beyond fork() + exec()

#198

Post by mpyne »

mort96 wrote: post #171
marcosdumay wrote: post #108

Earlier quoted context omitted.

CoW is probably a good idea whether you use fork or not. Or rather, fork is probably a better option than just exec exactly because it can benefit from CoW. At least on systems with virtual addressing. If you want to go into physical addressing, then yes, maybe it's a problem. But Linux will never touch anything with physical addressing, so I don't see what people are complaining about.

CoW is probably a good idea regardless, yeah. Overcommit is more questionable. Regardless, both ought to be argued based on their own merits. It's unfortunate that both are necessary as a consequence of fork().

I don't think fork() mandates overcommit. OpenBSD doesn't seem to even allow overcommit or have an OOM killer, memory allocations that exceed available capacity fail immediately even if the memory is not touched.

Re: Moving beyond fork() + exec()

#200

Post by sanderjd »

lokar wrote: post #52
1718627440 wrote: post #43

Earlier quoted context omitted.

Why? That's a very useful processing primitive.

It’s a hack with many disadvantages. Sometimes a hack is the right answer, but the kernel should it add a primitive for it.

Aren't we discussing just such a primitive?
Post reply on HN