Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving beyond fork() + exec()

lwn.net

Post reply on HN
Print view View original

151–160 of 358 posts

Re: Moving beyond fork() + exec()

#151

Post by [deleted] »

pizlonator wrote: post #57
rom1v wrote: post #36

Related to the discussion: "A fork() in the road": https://www.microsoft.com/en-us/research/wp-content/uploads/... > ABSTRACT > The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog t…

Fork is marvelous for the zygote pattern Hard to come up with an optimization that is equally efficient and elegant

[deleted]

Re: Moving beyond fork() + exec()

#152

Post by Someone »

mort96 wrote: post #79
anarazel wrote: post #58

Earlier quoted context omitted.

It is somewhat interesting that the most widely used "big" OS that doesn't use fork, i.e. Windows, has dog slow process creation... I agree that there should be non-fork primitives, I'm just not that sure that performance is the best argument.

The problem with fork isn't really that it's slow. The problem is that if you want it to be not-slow, it locks you into a bunch of OS design decisions: you more or less need a memory subsystem where all writable pages are refcounted and copy-on-write when the refcount is bigger than 1, and you need overcommit. Now these decisions aren't objectively bad , but they have significant trade-offs and it's probably not a go…

> The problem with fork isn't really that it's slow. The problem is that if you want it to be not-slow, it locks you into a bunch of OS design decisions: you more or less need a memory subsystem where all writable pages are refcounted and copy-on-write when the refcount is bigger than 1

It may not be slow, but for the common case where fork is almost immediately followed by exec in the process where fork returns zero fork increases those refcounts and exec almost immediately decreases them again hand does typically unnecessary checks whether refcounts became zero). A combined fork/exec syscall can avoid that work.

On the other hand, a sufficiently powerful combined fork/exec call has to have a lot of parameters that it has to check (whether to inherit open pipes, open files, setting the working directory, etc), and that slows it down.

That can be avoided by having multiple variants of combined fork/exec calls, but you would need lots of them to cover all combinations of flags.

I expect either approach should be faster then having fork, then exec as separate calls, especially when the process calling fork has many resources allocated.

Re: Moving beyond fork() + exec()

#153

Post by foresto »

mort96 wrote: post #79
anarazel wrote: post #58

Earlier quoted context omitted.

It is somewhat interesting that the most widely used "big" OS that doesn't use fork, i.e. Windows, has dog slow process creation... I agree that there should be non-fork primitives, I'm just not that sure that performance is the best argument.

The problem with fork isn't really that it's slow. The problem is that if you want it to be not-slow, it locks you into a bunch of OS design decisions: you more or less need a memory subsystem where all writable pages are refcounted and copy-on-write when the refcount is bigger than 1, and you need overcommit. Now these decisions aren't objectively bad , but they have significant trade-offs and it's probably not a go…

> The problem with fork isn't really that it's slow.

Did someone suggest that it was?

Re: Moving beyond fork() + exec()

#154

Post by trumpdong »

PaulDavisThe1st wrote: post #119
toast0 wrote: post #74

Earlier quoted context omitted.

The zygote pattern[1] is a great optimization to deal with the cost of forking, but IMHO, being able to inexpensively spawn a carefully tailored process regardless of the size and scope of the current process would be better. I would guess it would be a small difference in measurable performance between zygote and a direct clean spawn, but it's one less trick an application needs to do, and it would be very helpful f…

> being able to inexpensively spawn a carefully tailored process regardless of the size and scope of the current process would be better. It's called clone(2)

Which argument to clone starts the process with an empty address space?

Re: Moving beyond fork() + exec()

#155

Post by trumpdong »

m00x wrote: post #132
FooBarWidget wrote: post #45

Earlier quoted context omitted.

It says state. Copy on write still means it's O(number of page table entries) even if you don't copy the contents. It's a well known issue that forking a program with large virtual memory size is slow.

On modern hardware a cow page copy should only take 1-5ms. Redis forks to save the db to disk and it's been a solid design choice. I guess it depends on how sensitive your application is to main thread pauses.

So like 1000-5000s if you have 4GB of data? Over an hour?

Re: Moving beyond fork() + exec()

#156

Post by loeg »

vlovich123 wrote: post #84
pizlonator wrote: post #57

Earlier quoted context omitted.

Fork is marvelous for the zygote pattern Hard to come up with an optimization that is equally efficient and elegant

The paper explicitly covers it that various memory COW/snapshot mechanisms are probably faster and safer than the zygote pattern. As it stands getting the zygote pattern correct and safe is something you have to plan for upfront. You can’t retrofit it which is why the paper mentions it has poor composability. Also the advantages of the zygote pattern can be overstated since the memory sharing benefit is minimal since…

In what sense can you not retrofit the zygote pattern?

Re: Moving beyond fork() + exec()

#157

Post by [deleted] »

epcoa wrote: post #93
zozbot234 wrote: post #78

Earlier quoted context omitted.

Windows was designed with threads-first mentality because on pre-386 machines you don't have viable process memory protection, so your tasks share memory by necessity. This is not a great argument.

This is not true. NT never had fork, was always based on the assumption of an MMU and Dave Cutler was a well known fork hater in the 80s long before this paper came out and made it cool to be so. By the time Windows 95 was out, the baseline was 386 with an MMU. CreateThread was initially designed for NT in 1993 though (which didn’t support pre-386 CPUs).

[deleted]

Re: Moving beyond fork() + exec()

#158

Post by trumpdong »

uecker wrote: post #26

The elegance of the fork() + exec() model is that every kind of configuration can be done after the fork using all the usual APIs. Every attempt to replace it with a combined call that I have seen so far seemed fundamentally poorer because it needs to add all configuration options as parameters to the call and then do this in away that you can extend it later and does not become a mess.

It should be spawn, configure, exec. Configure can be done if the process starts with a ptrace attachment and no threads, so you can force it to do syscalls. Linux doesn't even have a concept of "process with no threads", so it'd probably have to have a dummy thread.

Re: Moving beyond fork() + exec()

#159

Post by Someone »

nvme0n1p1 wrote: post #106
theK wrote: post #94

Earlier quoted context omitted.

Didn't he just say that fork turns out to be comparatively faster to the non-fork samples we get? Ie Linux spawns processes faster than Microsoft's kernels?

We don't have any broadly used non-fork samples. Windows, macOS, and Linux all have fork. So the presence of fork can't be the reason for the performance difference. (Windows's fork is called ZwCreateProcess)

MacOS has posix_spawn. See https://developer.apple.com/library/archive/documentation/Sy... (yes, that’s an iOS man page. MacOS has the call, too, but I couldn’t find the man page online and it looks identical to me)

I don’t know how they implemented it, though. Under the hood, it could do the equivalent of a fork/exec pair.

Re: Moving beyond fork() + exec()

#160

Post by 201984 »

monocasa wrote: post #98
201984 wrote: post #85

Earlier quoted context omitted.

How do you think position independent code can call functions from other .so's without being patched with their addresses? They can't, so even PIC code still has to have a relocation table that gets patched. It's in a different page than the code though, so code does still get reused.

That's not really patching though, any more than any use of function pointers is patching.

There's a part of the .so ELF file (the Global Offset Table aka GOT) that has to be modified with all the addresses of the functions being imported, which of course vary from process to process.

If not patching, what exactly would you call modifying part of the file?

Post reply on HN