Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Midipix: Posix for Windows

midipix.org

Post reply on HN
Print view View original

81–90 of 92 posts

Re: Midipix: Posix for Windows

#81

Post by ised »

JoeAltmaier wrote: post #78
ised wrote: post #76

Was POSIX ever really intended to apply to a non-UNIX OS, e.g., VMS? The Windows kernel is based on the VMS kernel, right? And NTFS is based on the VMS filesystem? http://en.wikipedia.org/wiki/David_cutler I would have been happy with VMS on the PC. Instead we got Windoze. How much of our lives has this monstrosity wasted? Just let it die. Do daemontools' supervise and svscan need fork()?

Posix was an API standard, implementable anywhere. The very first POSIX implementation I assisted with, and it ran on CTOS which had a message-passing kernel and built-in networking. Nothing like Unix or Windows.

Somehow it just became strongly associated with UNIX?

Or UNIX became the embodiment of the concept of "run anywhere"?

Re: Midipix: Posix for Windows

#82

Post by wfunction »

poizan42 wrote: post #58
wfunction wrote: post #17

Earlier quoted context omitted.

And where is the copy-on-write happening here? If I recall correctly, the problem was that attempting to reproduce fork()'s copy-on-write behavior on Windows resulted in a massive performance hit. (Which, IIRC, contributes to the slowness of Cygwin.) EDIT: Oh, I'd skipped the part above "here's how it works"... that just says exactly the same thing I said above.

It is possibly to fork on Windows by using ZwCreateProcess directly, however the win32 subsystem and every single thirdparty library you are using does not expect this. If you are only using the NT native api then it should work just fine, but you are prevented from interacting with the win32 environment.

> if you are only using the NT native API then it should work just fine

I believe I had trouble executing any instructions in the new process at all. If you can make it work I'd like to see your code, otherwise I'm skeptical.

Re: Midipix: Posix for Windows

#83

Post by wfunction »

ddevault wrote: post #55
dalias wrote: post #54

Earlier quoted context omitted.

No, OOM killer is completely orthogonal to this and is a consequence of not doing correct commit accounting. With strict commit accounting turned on (vm.overcommit_memory=2) fork will correctly fail when there is not sufficient physical backing. But you're correct that fork+exec is a bad model.

The OOM killer exists because of memory overcommitment, which exists because of fork/exec. The justification for overcommitment is that a big-ass process might fork just to do an exec immediately afterwards. If that is the case, then it would be lame to error out the fork because there's not enough room for a second copy of it. But if it doesn't actually exec, then suddenly there's not actually as much memory as it t…

Thanks for sharing this, I wasn't aware of the relationship between the two.

Re: Midipix: Posix for Windows

#84

Post by wfunction »

TickleSteve wrote: post #24
wfunction wrote: post #22

Earlier quoted context omitted.

> What are you doing that involves lots of forking? Running scripts? Also, try enumerating the files in a large directory hierarchy and compare it with native Windows speed (or Linux speed), it's not even comparable.

fair enough, it probably wont match native for that. Personally tho, speed has always been sufficient for me to never notice. I use it most days, but I don't do comparisons.

"Won't match native" is quite the understatement.

It's not a question of a 40% speed difference, more like a > 400% speed difference last time I checked.

Re: Midipix: Posix for Windows

#85

Post by poizan42 »

wfunction wrote: post #82
poizan42 wrote: post #58

Earlier quoted context omitted.

It is possibly to fork on Windows by using ZwCreateProcess directly, however the win32 subsystem and every single thirdparty library you are using does not expect this. If you are only using the NT native api then it should work just fine, but you are prevented from interacting with the win32 environment.

> if you are only using the NT native API then it should work just fine I believe I had trouble executing any instructions in the new process at all. If you can make it work I'd like to see your code, otherwise I'm skeptical.

AFAIK this is how fork() is implemented in SUA, but I haven't tried it myself. You could load the SUA subsystem dll into IDA Pro and see how they actually do it.

Re: Midipix: Posix for Windows

#86

Post by wfunction »

mpixorg wrote: post #63
the8472 wrote: post #43

Earlier quoted context omitted.

> For one or the other reason, fork() has become in many discussion of posix on windows something of a fetish. how about this one? fopen + mmap + unlink Afaik windows APIs forbid deleting mmaped files.

With FILE_SHARE_DELETE there shouldn't be any particular problem, but I certainly want to test this with the posix flags you had in mind. Do you happen to have a minimal example that you deem problematic on Windows?

If you delete the backing file of (or perhaps even close its handle -- I don't remember) a memory-mapped file, you get VERY bizarre behavior in Windows. I don't remember what it was, but I remember I did it and the behavior I got was quite nonsensical.

Re: Midipix: Posix for Windows

#87

Post by dalias »

dalias wrote: post #48
mpixorg wrote: post #42

The project's pre-pre-alpha will be out very soon (in a week or less) and will already give a nice taste of its design and speed. The primary focus thus far has been major challenges (toolchain, process creation and initialization, signals, glue layer between posix system call layer and libc, etc.). This means that while you will be able to test functions such as fork(), execve(), mmap(), or fopen() with a utf-8 path…

I have a sequence of tweets here summarizing basic measurements I did on musl's posix_spawn versus fork+exec: https://twitter.com/RichFelker/status/602313644026761216 There are still plenty of applications that use fork semantically, to keep the same process image in the child, which benefit somewhat from a fast fork. But most places where fork affects performance now are things that are already pessimized by using f…

To clarify, the numbers in my tweets are measured on Linux, comparing musl libc's posix_spawn (with CLONE_VM) to plain fork+exec (which should be independent of libc). I just posted the test program on our mailing list: http://www.openwall.com/lists/musl/2015/06/04/1

Re: Midipix: Posix for Windows

#88

Post by wfunction »

poizan42 wrote: post #85
wfunction wrote: post #82

Earlier quoted context omitted.

> if you are only using the NT native API then it should work just fine I believe I had trouble executing any instructions in the new process at all. If you can make it work I'd like to see your code, otherwise I'm skeptical.

AFAIK this is how fork() is implemented in SUA, but I haven't tried it myself. You could load the SUA subsystem dll into IDA Pro and see how they actually do it.

I believe the POSIX subsystem has some kind of support from the native API that prevents you from writing your own random subsystem and expecting it to work; I don't remember what it was though (it's been a few years).

Re: Midipix: Posix for Windows

#89

Post by poizan42 »

wfunction wrote: post #88
poizan42 wrote: post #85

Earlier quoted context omitted.

AFAIK this is how fork() is implemented in SUA, but I haven't tried it myself. You could load the SUA subsystem dll into IDA Pro and see how they actually do it.

I believe the POSIX subsystem has some kind of support from the native API that prevents you from writing your own random subsystem and expecting it to work; I don't remember what it was though (it's been a few years).

There's some special handling when loading the image where it does different things depending on the value of the subsystem field, but I don't know whether there's any special handling in the kernel that can't be duplicated by using the native api. But the NT kernel does very little when it comes to initializing new processes, most of the initialization is done in user space by ntdll, so it seems unlikely.

Anyways the cygwin guys claim to have forked processes with ZwCreateProcess, but just had problems with getting it to work with the win32 subsystem: http://www.cygwin.com/ml/cygwin-developers/2011-04/msg00034....

Also this guy seems to have managed to do it: http://stackoverflow.com/questions/10657699/cant-use-createp..., but it hang when he called a win32 function (CreateProcess) from the child.

Re: Midipix: Posix for Windows

#90

Post by quotemstr »

poizan42 wrote: post #89
wfunction wrote: post #88

Earlier quoted context omitted.

I believe the POSIX subsystem has some kind of support from the native API that prevents you from writing your own random subsystem and expecting it to work; I don't remember what it was though (it's been a few years).

There's some special handling when loading the image where it does different things depending on the value of the subsystem field, but I don't know whether there's any special handling in the kernel that can't be duplicated by using the native api. But the NT kernel does very little when it comes to initializing new processes, most of the initialization is done in user space by ntdll, so it seems unlikely. Anyways th…

> had problems with getting it to work with the win32 subsystem

Keep in mind that the whole _point_ of Cygwin is to interact with the Windows world. If you want a POSIX sandbox, you can use a VM with fewer headaches and better performance.

Post reply on HN