Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving beyond fork() + exec()

lwn.net

Post reply on HN
Print view View original

131–140 of 358 posts

Re: Moving beyond fork() + exec()

#131

Post by keitmo »

epcoa wrote: post #93
zozbot234 wrote: post #78

Earlier quoted context omitted.

Windows was designed with threads-first mentality because on pre-386 machines you don't have viable process memory protection, so your tasks share memory by necessity. This is not a great argument.

This is not true. NT never had fork, was always based on the assumption of an MMU and Dave Cutler was a well known fork hater in the 80s long before this paper came out and made it cool to be so. By the time Windows 95 was out, the baseline was 386 with an MMU. CreateThread was initially designed for NT in 1993 though (which didn’t support pre-386 CPUs).

NT performed unnatural acts to implement fork semantics for the POSIX subsystem.

Re: Moving beyond fork() + exec()

#132

Post by m00x »

FooBarWidget wrote: post #45
mrkeen wrote: post #35

> fork() is a relatively expensive system call; it must copy the entire process state (including memory) for the child process. Many optimizations have been made over the years, but a fork is still a fundamentally costly operation. To make things worse, a fork() call is often immediately followed by an exec(), which will discard all of that memory that was so carefully copied for the child. It's weird to leave out a…

It says state. Copy on write still means it's O(number of page table entries) even if you don't copy the contents. It's a well known issue that forking a program with large virtual memory size is slow.

On modern hardware a cow page copy should only take 1-5ms. Redis forks to save the db to disk and it's been a solid design choice.

I guess it depends on how sensitive your application is to main thread pauses.

Re: Moving beyond fork() + exec()

#133

Post by skydhash »

ggm wrote: post #118

Aesthetically I have no intention of moving beyond. I'm content with my kernels scheduler and how it maps "heavyweight" processes to cores. I do use threaded code. It's significantly harder to write and reason about. (45 years in to a CS career, ageing out) You have to be clever to do better than clever people. Clever people bootstrapped me into fork()/exec() and I know my limits.

I’m using Emacs and various cli tools and while threads are nice to have, they can easily ramp up the complexity of a program beyond what is necessary. I much prefer the boilerplate of setting up a thread pool and tasks queue, rather than dealing with all the await/async syntactic sugar.

Re: Moving beyond fork() + exec()

#134

Post by pjmlp »

PaulDavisThe1st wrote: post #116
pjmlp wrote: post #70

Earlier quoted context omitted.

Because that OS best practices is to use threads. Traditionally Windows applications that create processes all the time come from UNIX heritage. Contrary to UNIX, Windows NT was designed with threads first mentality, from the get go. While on UNIX they were added after fact, and to this day there are gotchas mixing posix threads with signals, fork and exec.

A more accurate way to describe this is that Windows' (NT onward) core execution context model is a bunch of threads that by default share memory, whereas Unixen have a core task context model of a bunch of threads that by default do not share memory. Both systems are implemented using threads as the execution context, but in Unix, the history means that that you fork+exec most of the time, resulting in a two tasks t…

Well, Windows before NT isn't the same design as Windows 16 bit, it only shares the name for all practical purposes, and has more influence from OS/2 than Windows 16 bit.

Which is why I took the effort to explicitly refer to Windows NT on my comment, already expecting some traditional answers from UNIX folks.

Also due to historical reasons POSIX threads are the outcome of every UNIX going their own way implementing threads, finally coming to an agreement years later, with all the plus and minus of relying in POSIX for portable code.

Re: Moving beyond fork() + exec()

#135

Post by asveikau »

The things you can do between fork and exec are sometimes underestimated. Off the top of my head, you can call dup2(), you can set a process group id, probably a few other things.

If you contrast that with win32, where you optionally pack a bunch of initial values into a struct, win32 is a much more narrow, less pleasant, less freeform interface, where it is harder to introduce more features.

But I think there is already posix_spawn to imitate that philosophy on Unix-like OSs.

Re: Moving beyond fork() + exec()

#136

Post by mort96 »

dapperdrake wrote: post #129
mort96 wrote: post #79

Earlier quoted context omitted.

The problem with fork isn't really that it's slow. The problem is that if you want it to be not-slow, it locks you into a bunch of OS design decisions: you more or less need a memory subsystem where all writable pages are refcounted and copy-on-write when the refcount is bigger than 1, and you need overcommit. Now these decisions aren't objectively bad , but they have significant trade-offs and it's probably not a go…

How else does consistency work, then? Only being half facetious here. Maybe you or someone else really has a better take.

What do you mean by consistency here?

Re: Moving beyond fork() + exec()

#137

Post by JdeBP »

yxhuvud wrote: post #103
7jjjjjjj wrote: post #90

Earlier quoted context omitted.

>It feels crazy that we don't have a way to directly express the latter thing Isn't that what posix_spawn is for?

And how do you think posix_spawn is implemented?

This is an oft-overlooked point. An obvious place to look for improving fork+execve is to see whether posix_spawn can be given more efficient kernel mechanisms to be based upon.

And of course that has already been done. On NetBSD, posix_spawn() is a fully-fledged system call and much of the work is done in kernel mode.

* https://blog.netbsd.org/tnf/entry/posix_spawn_syscall_added

Re: Moving beyond fork() + exec()

#138

Post by snozolli »

PaulDavisThe1st wrote: post #116
pjmlp wrote: post #70

Earlier quoted context omitted.

Because that OS best practices is to use threads. Traditionally Windows applications that create processes all the time come from UNIX heritage. Contrary to UNIX, Windows NT was designed with threads first mentality, from the get go. While on UNIX they were added after fact, and to this day there are gotchas mixing posix threads with signals, fork and exec.

A more accurate way to describe this is that Windows' (NT onward) core execution context model is a bunch of threads that by default share memory, whereas Unixen have a core task context model of a bunch of threads that by default do not share memory. Both systems are implemented using threads as the execution context, but in Unix, the history means that that you fork+exec most of the time, resulting in a two tasks t…

whereas Unixen have a core task context model of a bunch of threads that by default do not share memory.

How are those not simply child processes? I don't understand your use of the word 'threads' here.

Does the Unix world not distinguish between threads and processes? In Win32, threads exist within processes, and you can create new threads or child processes.

Re: Moving beyond fork() + exec()

#139

Post by knome »

pjmlp wrote: post #70
anarazel wrote: post #58

Earlier quoted context omitted.

It is somewhat interesting that the most widely used "big" OS that doesn't use fork, i.e. Windows, has dog slow process creation... I agree that there should be non-fork primitives, I'm just not that sure that performance is the best argument.

Because that OS best practices is to use threads. Traditionally Windows applications that create processes all the time come from UNIX heritage. Contrary to UNIX, Windows NT was designed with threads first mentality, from the get go. While on UNIX they were added after fact, and to this day there are gotchas mixing posix threads with signals, fork and exec.

the only difference between a thread and a process on linux is how many structures they share. the function is identical.

Re: Moving beyond fork() + exec()

#140

Post by sunshowers »

pjmlp wrote: post #70
anarazel wrote: post #58

Earlier quoted context omitted.

It is somewhat interesting that the most widely used "big" OS that doesn't use fork, i.e. Windows, has dog slow process creation... I agree that there should be non-fork primitives, I'm just not that sure that performance is the best argument.

Because that OS best practices is to use threads. Traditionally Windows applications that create processes all the time come from UNIX heritage. Contrary to UNIX, Windows NT was designed with threads first mentality, from the get go. While on UNIX they were added after fact, and to this day there are gotchas mixing posix threads with signals, fork and exec.

The problem is that threads are not fault boundaries but processes are. So they're not interchangeable when you care about resilience and misbehaving code.
Post reply on HN