Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving beyond fork() + exec()

lwn.net

Post reply on HN
Print view View original

241–250 of 358 posts

Re: Moving beyond fork() + exec()

#241

Post by thayne »

JdeBP wrote: post #187
PaulDavisThe1st wrote: post #116

Earlier quoted context omitted.

A more accurate way to describe this is that Windows' (NT onward) core execution context model is a bunch of threads that by default share memory, whereas Unixen have a core task context model of a bunch of threads that by default do not share memory. Both systems are implemented using threads as the execution context, but in Unix, the history means that that you fork+exec most of the time, resulting in a two tasks t…

That's actually less accurate, not more. It's a post-hoc revision that conflates Unix with Linux. The Unix model was invented over a decade before the idea of multithreading percolated into mainstream operating systems at all. The reason that Windows NT started as it did, was that OS/2 had come out in 1987, with kernel threads, and the idea of multithreading had taken root. SunOS 5 gained threading, too. Windows NT a…

PaulDavisThe1st is saying that the Unix pattern of forking a process (and not calling exec) was an early form of multi-threading (or multi-processing), but unlike threads in NT and later pthreads, they didn't share memory and communication between them required some form of IPC.

Re: Moving beyond fork() + exec()

#242

Post by dcrazy »

pjmlp wrote: post #234
dcrazy wrote: post #227

Earlier quoted context omitted.

If you want the isolation features of a separate process, you can’t substitute it with a single multithreaded COM server process. .NET tried this with app domains, which are now deprecated.

App Domains were in process, which isn't was I am talking about with outproc COM. Also App Domains are partially back in .NET Core, isolation features aren't there, but code unloading is, via AssemblyLoadContext.

My point is that “just write a COM server” is not an answer to the problem of “I want each work item to be segregated from each other.”

Re: Moving beyond fork() + exec()

#243

Post by up2isomorphism »

rom1v wrote: post #36

Related to the discussion: "A fork() in the road": https://www.microsoft.com/en-us/research/wp-content/uploads/... > ABSTRACT > The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog t…

Not sure if fork is outdated or not, but people calling it a “hack” obviously have pretty bad engineering taste.

Re: Moving beyond fork() + exec()

#244

Post by jkrejcha »

uecker wrote: post #240
jkrejcha wrote: post #237

Earlier quoted context omitted.

I kinda disagree, though I do see the usefulness here. While fork/exec can be useful in some cases, it'd be honestly pretty neat if the APIs took a pidfd argument (maybe with 0 meaning current process). Only program is setuid/setgid binaries I suppose but maybe this case is better handled by special casing `exec`. For example pidfd_t ps = spawn(); // creates a process stopped (kernel does this anyway by default) setu…

Maybe, a few people proposed this. It is a lot better than a single spawn call. But how often would one actually need this? And what are the semantics? Refer arguments (e.g. file descriptors) to the current process or the other one? How are cross-permissions handled? It seems a lot of complexity... Someones proposed a ptrace_syscall which could achieve the same thing.

> But how often would one actually need this?

Well, the idea is that it'd probably be close to the default API for spawning processes (and could even be the bedrock for posix_spawn and friends in libc (and potentially even "simple" fork cases[1])). fork/clone would be the special case

In most cases, most programs don't need special setup. Something like `ptrace_syscall` would also work for this and would be probably the way to do it with the backwards compat limitations of nowadays

ptrace-ability seems to be generally how permissions for this sort of thing are handled in general (see also procfs, process_vm_writev, ptrace, etc). The complication is a little bit around setuid programs but either you could special case execve to imply SIGCONT for setuid or have execve also imply a SIGCONT as well

[1]: Probably would be rare for a compiler to optimize it though

Re: Moving beyond fork() + exec()

#245

Post by asveikau »

loeg wrote: post #229
asveikau wrote: post #218

Earlier quoted context omitted.

I'm not talking about glibc implementation details. I'm talking about how mixing fork(2) with threads creates harmful race conditions. The forked child has only 1 thread in its process. If the parent's threads are holding a lock or are in the middle of mutating a shared data structure, you're fucked, because those threads are no longer running in your child's copy of the address space and will not finish their work.…

Again, you're talking about userspace now. Not kernel-imposed constraints. A userspace program is always free to deadlock itself; fork doesn't change that.

Just want to come back with a simple example.

This means if the program is multi threaded, you cannot rely on calling malloc in the child, because at the time of the fork another thread could have happened to be inside malloc doing manipulations on the global heap.

Which means, practically speaking, "don't allocate memory between fork and exec".

If you want to be overly literal as you have been, you can call mmap and it will give you new pages, but who is really doing that? Not the random shared library code you might want to call into. Hell, even a lot of libc calls malloc.

Which means it's not safe to do a random library call between fork and exec.

See where I'm going with this? That's if your program is multi threaded. If it isn't, these things are most likely fine.

Re: Moving beyond fork() + exec()

#246

Post by jkrejcha »

JdeBP wrote: post #104
amluto wrote: post #99

Earlier quoted context omitted.

Linux has worked pretty hard to optimize fork(). This doesn’t mean that fork() is a good idea. Windows does not historically depend on fork(), so there was no native fork(), so Cygwin kludged it up.

Actually, there is a native fork. There had to be, as POSIX personality support was a part of the Windows NT 3.1 design. What there wasn't was a Win32 form of fork. The Native API for Windows NT allowed it quite straightforwardly.

Iirc Cygwin used to use it but iirc they moved away from it because they said that it was pretty slow

Though actually iirc werfault uses NtCreateUserProcess() to clone processes when writing out crash dumps to this day

Re: Moving beyond fork() + exec()

#247

Post by PaulDavisThe1st »

thayne wrote: post #241
JdeBP wrote: post #187

Earlier quoted context omitted.

That's actually less accurate, not more. It's a post-hoc revision that conflates Unix with Linux. The Unix model was invented over a decade before the idea of multithreading percolated into mainstream operating systems at all. The reason that Windows NT started as it did, was that OS/2 had come out in 1987, with kernel threads, and the idea of multithreading had taken root. SunOS 5 gained threading, too. Windows NT a…

PaulDavisThe1st is saying that the Unix pattern of forking a process (and not calling exec) was an early form of multi-threading (or multi-processing), but unlike threads in NT and later pthreads, they didn't share memory and communication between them required some form of IPC.

Yep, absolutely corrrect. It was true at the lowest level (the semantics of fork) and it was true at the app/platform design level: in Windows you used threads inside a process, on Unix you used multiple communicating processes.

This obviously changed as pthreads came into being, and at this point, I suspect that the typical use for threads-sharing-memory and threads-not-sharing-memory is the same on most platforms.

A reminder that the task_t data structure describes threads and processes not just in Linux, but earlier Unixen also.

Re: Moving beyond fork() + exec()

#248

Post by PaulDavisThe1st »

pjmlp wrote: post #145
snozolli wrote: post #138

Earlier quoted context omitted.

whereas Unixen have a core task context model of a bunch of threads that by default do not share memory. How are those not simply child processes? I don't understand your use of the word 'threads' here. Does the Unix world not distinguish between threads and processes? In Win32, threads exist within processes, and you can create new threads or child processes.

Actually on Windows a process is a thread with additional information. The unit of execution is the thread. On the UNIX world it depends on which UNIX you are talking about. Linux has a similar model to Windows NT nowadays, hence clone() as key primitive. Other UNIXes have different approaches.

I worked on the kernel of DEC Ultrix, Mach/BSD and a couple of other early Unixen. The approach in all the ones I worked on was broadly the same.

Re: Moving beyond fork() + exec()

#250

Post by amluto »

uecker wrote: post #233
dcrazy wrote: post #231

Earlier quoted context omitted.

Nothing about the UNIX shell is reliant on the fork model. Windows processes have stdio handles as well.

A lot of features of UNIX shells are build around pipe and dup and the fork + exec model. One can certainly implement in differently, but it is - like UNIX in general - very nice and elegant.

Help me out here, please. Off the top of my head, the exec command is dependent on exec, except that a spawn + wait implementation would be a mostly okay substitute.

Pipes and redirections don’t need fork + exec. Neither do subshells.

Post reply on HN