Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A fork() in the road

microsoft.com

Post reply on HN
Print view View original

121–130 of 184 posts

Re: A fork() in the road

#121

Post by cesarb »

Gibbon1 wrote: post #80
ralish wrote: post #21

Earlier quoted context omitted.

I'm a bit rusty on this but from memory the overhead is by and large specific to the Win32 environment. Creating a "raw" process is cheap and fast (as you'd reasonably expect), but there's a lot of additional initialisation that needs to occur for a "fully-fledged" Win32 process before it can start executing. Beyond the raw Process and Thread kernel objects, which are represented by EPROCESS + KPROCESS and ETHREAD +…

> created on-demand with the first creation of a GDI or USER object, so presumably CLI apps don't have to pay that price This tickles my brain. I read some blog post bitching that because Windows DLL's are kinda heavy weight it's way easy end up paying that price without realizing it.

It probably was this one: https://randomascii.wordpress.com/2018/12/03/a-not-called-fu...

Re: A fork() in the road

#122

Post by JdeBP »

Gibbon1 wrote: post #80
ralish wrote: post #21

Earlier quoted context omitted.

I'm a bit rusty on this but from memory the overhead is by and large specific to the Win32 environment. Creating a "raw" process is cheap and fast (as you'd reasonably expect), but there's a lot of additional initialisation that needs to occur for a "fully-fledged" Win32 process before it can start executing. Beyond the raw Process and Thread kernel objects, which are represented by EPROCESS + KPROCESS and ETHREAD +…

> created on-demand with the first creation of a GDI or USER object, so presumably CLI apps don't have to pay that price This tickles my brain. I read some blog post bitching that because Windows DLL's are kinda heavy weight it's way easy end up paying that price without realizing it.

As mentioned in this very discussion 2 hours before. (-:

* https://news.ycombinator.com/item?id=19622723

Re: A fork() in the road

#123

Post by olemartinorg »

intea wrote: post #90
ralish wrote: post #21

Earlier quoted context omitted.

I'm a bit rusty on this but from memory the overhead is by and large specific to the Win32 environment. Creating a "raw" process is cheap and fast (as you'd reasonably expect), but there's a lot of additional initialisation that needs to occur for a "fully-fledged" Win32 process before it can start executing. Beyond the raw Process and Thread kernel objects, which are represented by EPROCESS + KPROCESS and ETHREAD +…

The WSL processes are called pico processes. https://blogs.msdn.microsoft.com/wsl/2016/05/23/pico-process...

That was a super interesting read (and view), thank you. I've been in Linux land for almost two decades, but I've also spent a week (or so) porting our Linux-based development environment over to Windows with the help of WSL. This sheds some light on how it actually works. Maybe I'll have to look over it once more armed with this new information and see if I can squash some of those remaining problems with our solution.

Re: A fork() in the road

#124

Post by naasking »

stirfrykitty wrote: post #114
naasking wrote: post #106

Earlier quoted context omitted.

Talk about uncharitable. MS Research produces world-class research. They don't just research Windows, they do research in all operating systems, programming languages and more.

It's not about being "uncharitable". It's about protecting nix from being controlled by outside forces. MS does, indeed. have world-class research, but they are sticking their heads in the nix camp, which some of us don't like. We're not all in this together, despite what some will tell you. Sadly, UNIX (umbrella term here) is not what it was a few years ago. I dearly miss Solaris, for example. Nothing touched it in…

Microsoft Research is not Microsoft. Microsoft Research employs some of the main Haskell developers, and you don't see Haskellers going all conspiracy theory. Research is research, and either the ideas they describe are good and should be adopted, or they're bad and should be ignored.

Re: A fork() in the road

#125

Post by dexen »

temac wrote: post #120
dexen wrote: post #115

Earlier quoted context omitted.

>all the kernel page tables still need to be copied and for a multi-GB process that's nontrivial Only in the pathological case where the large process is backed solely by the 4kb pages. The hardware has long now supported large pages - on x86 since Pentium Pro, if memory serves - and huge pages. The popular OSes (Linux 2.6+ and Windows 2003+) also do support large and huge pages. A 2GB process can easily be three pag…

You want to ditch shared libraries and mmap to map your big processes using GB pages to make fork fast again (despite it not being the main and only drawback)??? OS research might be relevant, and it's good that some people have wild idea, but honestly I doubt this one will go anywhere :P

Ah sorry, only want to ditch the shared libraries; added mmap is a later edition didn't realize it's unclear. Of course mmap is necessary.

Re: A fork() in the road

#126

Post by eesmith »

sanxiyn wrote: post #111
eesmith wrote: post #48

Earlier quoted context omitted.

Yes, certainly. The paper covers many of those limitations. The goal is not "remove", but "seek to remove". The relevant definition of "seek" here is "to make an attempt" says https://www.merriam-webster.com/dictionary/seek . How many of those 1304 Ubuntu packages require fork()? Are there benefits to replacing (say) 1283 of them with posix_spawn()?

Yes, there are benefits to using posix_spawn: It's faster. See Figure 1.

How many of those packages would be improved with a faster spawn mechanism? Who is going to investigate each one? How will they convince upstream to change well-tested code?

Re: A fork() in the road

#127

Post by kevin_b_er »

fanf2 wrote: post #73
speedplane wrote: post #35

Earlier quoted context omitted.

I used to work on a cross-platform project, and spent several weeks trying to figure out why our application ran significantly faster on linux than windows. One major culprit was process creation (another was file creation). I never really uncovered the true reason, but I suspect it had to do with the large number of DLLs that Windows would automatically link if you weren't very careful. Linux, of course, can also lo…

Anti-virus software makes process and file operations a lot slower.

This should not be ignored. Windows machines are a favorite for having lots of heavy anti-virus running on them. They can destroy I/O performance. Windows 10 has a "real time scanner" running by default, but many corporate-IT security teams will add more and more. This alone can seriously slow down windows vs linux.

Re: A fork() in the road

#128

Post by cryptonector »

JdeBP wrote: post #81
kazinator wrote: post #75

Earlier quoted context omitted.

CreateProcess requires an application to initialize from scratch. When you fork, you cheaply inherit the initialized state of the whole application image. Only a few pages that are mutated have to be subject to copy-on-write. Even that copy-on-write is cheaper than calculating the contents of those pages from scratch.

There has been a lot of discussion in recent years about how cheap that "cheaply" really is. * https://news.ycombinator.com/item?id=9653238 * https://news.ycombinator.com/item?id=18071278 * https://news.ycombinator.com/item?id=19622503

Yeah, it's not really cheap at all. However! vfork() is cheap, very very cheap, though, of course, you then have to follow it up with an exec(), and the cost of that on Windows depends on the setup cost of the executable being exec'ed.

Part of the problem is the DLLs, as many have mentioned, and also the fact that each statically links in its own CRT (C run-time). The shared C run-time MSFT is working on should help here. As should more lazy loading and setup.

Re: A fork() in the road

#129

Post by cryptonector »

rgovostes wrote: post #51

On macOS, fork() is a bit weird: https://opensource.apple.com/source/Libc/Libc-997.90.3/sys/f... Many frameworks are backed by XPC services, where the parent process has a socket-like connection to a backend server. After forking, the child would have no valid connection to the server. The fork() function establishes a new connection in the child for libSystem, to allow Unix programs to port easily to macOS, but othe…

fork() is generally unsafe for that reason, and OS X is only special in this regard in that it has more of these hidden C library handles that can blow up on the child-side of fork(). vfork()+exec()-or-_exit() is much safer.

Re: A fork() in the road

#130

Post by cryptonector »

saagarjha wrote: post #11
swiftcoder wrote: post #8

Fork has really weird semantics, and a lot of fun gotchas around managing resources. Good riddance?

> Good riddance? Regardless of this paper, I don't see its use declining significantly any time soon.

Developers should use posix_spawn() as much as possible.
Post reply on HN