Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fork() without exec() is dangerous in large programs

evanjones.ca

Post reply on HN
Print view View original

21–30 of 106 posts

Re: Fork() without exec() is dangerous in large programs

#21

Post by chubot »

eridius wrote: post #10
caf wrote: post #2

In my opinion, it's libraries that secretly use threads under the covers (but still access state shared with the rest of the program like the malloc heap) that are what's dangerous.

I completely disagree. The only part of this that seems dangerous is the fact that it can turn a single-threaded program into a multi-threaded one, and the only real problem there is it makes fork-without-exec unsafe. But fork-without-exec is basically unsafe anyway unless you can absolutely guarantee that nothing has ever spawned a thread. Meanwhile, preventing libraries from using threading under the covers basical…

You can have your cake and eat it too -- there's no throwing away of performance gains. Libraries can use threads, but not directly or "silently". They should take threads as parameters. Then the application knows when they're being used and the application programmer can reason about it.

Re: Fork() without exec() is dangerous in large programs

#22

Post by TickleSteve »

chubot wrote: post #5
caf wrote: post #2

In my opinion, it's libraries that secretly use threads under the covers (but still access state shared with the rest of the program like the malloc heap) that are what's dangerous.

Yes. Libraries should provide algorithms and data structures, and parameterize a threading interface if necessary (though I think it's usually not necessary). Applications can then instantiate their own threads and pass them to the library (i.e. dependency injection.) So if you have 10 different concurrent libraries in an application, then you don't end up with 10 different threading paradigms or 10 different thread…

true...

...this applies not just to threading, but other things that are not in the domain of the library such as memory allocation strategy or communication mechanism.

Libraries should be small, focused and decoupled. This is the way to proper re-usability.

Re: Fork() without exec() is dangerous in large programs

#23

Post by amelius »

chubot wrote: post #21
eridius wrote: post #10

Earlier quoted context omitted.

I completely disagree. The only part of this that seems dangerous is the fact that it can turn a single-threaded program into a multi-threaded one, and the only real problem there is it makes fork-without-exec unsafe. But fork-without-exec is basically unsafe anyway unless you can absolutely guarantee that nothing has ever spawned a thread. Meanwhile, preventing libraries from using threading under the covers basical…

You can have your cake and eat it too -- there's no throwing away of performance gains. Libraries can use threads, but not directly or "silently". They should take threads as parameters. Then the application knows when they're being used and the application programmer can reason about it.

Yes, in theory. In practice, every library developer/maintainer uses different guidelines.

Re: Fork() without exec() is dangerous in large programs

#24

Post by dozzie »

chubot wrote: post #6
pcwalton wrote: post #4

Earlier quoted context omitted.

I dunno--under that definition every concurrent GC is "dangerous".

What libraries need to do GC? That is a language issue and not a library issue. When you are embedding a language like Go in a C program, its runtime becomes a library. But that's the example the proves the rule: there are (or were) well-known problems with that. Libraries shouldn't be doing GC.

> What libraries need to do GC? That is a language issue and not a library issue.

Boehm GC disagrees.

Re: Fork() without exec() is dangerous in large programs

#25

Post by djsumdog »

chubot wrote: post #5
caf wrote: post #2

In my opinion, it's libraries that secretly use threads under the covers (but still access state shared with the rest of the program like the malloc heap) that are what's dangerous.

Yes. Libraries should provide algorithms and data structures, and parameterize a threading interface if necessary (though I think it's usually not necessary). Applications can then instantiate their own threads and pass them to the library (i.e. dependency injection.) So if you have 10 different concurrent libraries in an application, then you don't end up with 10 different threading paradigms or 10 different thread…

I know it's very different in JVM land, but many Akka libraries do something similar where they take am actor system, allowing the developer to share or separate the actor systems.

Re: Fork() without exec() is dangerous in large programs

#26

Post by ithkuil »

xroche wrote: post #11
rdtsc wrote: post #8

Yap. Erlang 19 (latest version) switched to using a smaller spawner executable and spawns all OS processes from there by forking. So it forks something restricted and small not the whole VM. Here are the details of how it works: https://github.com/erlang/otp/blob/a5256e5221aff30f6d2cc7fab... They also claim a 3-5x speedup for launching external commands because of it. So there is a nice performance boost as well. So…

This is not a bad strategy, especially because between the moment you fork() and the child calls exec(), you have a lot of memory regions shared in copy-on-write (because it would be Too costly to actually duplicate the universelle). Which means that busy threads writing in memory in the parent area will trigger minor page faults, possibly impacting performances...

yeah, that's why vfork has been introduced back in the days. See http://man7.org/linux/man-pages/man2/vfork.2.html

Re: Fork() without exec() is dangerous in large programs

#27

Post by the_mitsuhiko »

ahoka wrote: post #18
saynsedit wrote: post #12

Face the fact, fork() is fundamentally flawed!

I would like to see the reasons this comment was down-voted. It is true that fork(2) is simple and elegant design, but it involves a huge accidental complexity and hidden implications like in the original article. There is a reason that Linux has clone(2) underneath.

> It is true that fork(2) is simple and elegant design

What exactly is simple and elegant about it? Have you looked into how much tooling is necessary everywhere in unix to make it work? It's insane. It has a huge footprint and it does not provide standardized APIs to make it work for non covered cases (the best we have is pthread_atfork which is not portable).

Re: Fork() without exec() is dangerous in large programs

#28

Post by amag »

TickleSteve wrote: post #22
chubot wrote: post #5

Earlier quoted context omitted.

Yes. Libraries should provide algorithms and data structures, and parameterize a threading interface if necessary (though I think it's usually not necessary). Applications can then instantiate their own threads and pass them to the library (i.e. dependency injection.) So if you have 10 different concurrent libraries in an application, then you don't end up with 10 different threading paradigms or 10 different thread…

true... ...this applies not just to threading, but other things that are not in the domain of the library such as memory allocation strategy or communication mechanism. Libraries should be small, focused and decoupled. This is the way to proper re-usability.

Like left-pad?

Re: Fork() without exec() is dangerous in large programs

#29

Post by cperciva »

pjmlp wrote: post #20
wruza wrote: post #17

Earlier quoted context omitted.

How many non-unixes do have fork_without_exec()?

Most non-UNIX OSes only have exec(), not fork(). Hence why fork() is just kind-of implemented on their POSIX compatibility layers. In UNIX it is only an issue, because IEEE doesn't want to set in stone in POSIX how threads, signals and processes should interact, due to the way threads were added to UNIX.

Non-unix systems don't have exec; they have spawn.

Re: Fork() without exec() is dangerous in large programs

#30

Post by the_mitsuhiko »

caf wrote: post #2

In my opinion, it's libraries that secretly use threads under the covers (but still access state shared with the rest of the program like the malloc heap) that are what's dangerous.

Libraries should not be disallowed from spawning threads. Threads are inherently the right way to do concurrency for more complex tasks. fork() is a tremendously complex call and has huge implications on all things in the OS.

For many years now I have the rule that new code must not use fork() and I am living a happier life. It's a bad call and it has no use if you have good threading support in the language. There are many programming languages that do not support fork() andy more or heavily restrict the use.

Post reply on HN