Live data from Hacker News

Midipix: Posix for Windows

midipix.org

61–70 of 92 posts

Re: Midipix: Posix for Windows

#61
post #59

Earlier quoted context omitted.

The OOM killer exists because of memory overcommitment, which exists because of fork/exec. The justification for overcommitment is that a big-ass process might fork just to do an exec immediately afterwards. If that is the case, then it would be lame to error out the fork because there's not enough room for a second copy of it. But if it doesn't actually exec, then suddenly there's not actually as much memory as it t…

No, fork is only one path that can lead to overcommit. Allocation of new memory as COW references to a zero page, and COW writable MAP_PRIVATE mappings of files (such as the writable LOAD segments of any executable or library file) also lead to overcommit unless you do proper commit accounting. Any system that does not need to do detailed commit accounting to avoid overcommit is basically wasting the fact that it has…

I never meant to suggest that fork is the only path that leads to overcommitment, but that fork/exec generally insists on overcommitment for reasonable use.

Re: Midipix: Posix for Windows

#62
post #49

Interix did this, and then they were shelved. http://en.wikipedia.org/wiki/Interix

Interix is a lot different because it requires installing a system component, which requires administrator privileges. Midipix produces applications that (at least as I understand it) run on basically any NT-based Windows with no special privileges.

Sure, but to echo userbinator's comment above, compatibility layers are limited. Interix was architected to run directly on top of HAL.

Re: Midipix: Posix for Windows

#63
post #43
post #42

The project's pre-pre-alpha will be out very soon (in a week or less) and will already give a nice taste of its design and speed. The primary focus thus far has been major challenges (toolchain, process creation and initialization, signals, glue layer between posix system call layer and libc, etc.). This means that while you will be able to test functions such as fork(), execve(), mmap(), or fopen() with a utf-8 path…

> For one or the other reason, fork() has become in many discussion of posix on windows something of a fetish. how about this one? fopen + mmap + unlink Afaik windows APIs forbid deleting mmaped files.

With FILE_SHARE_DELETE there shouldn't be any particular problem, but I certainly want to test this with the posix flags you had in mind. Do you happen to have a minimal example that you deem problematic on Windows?

Re: Midipix: Posix for Windows

#64
post #9

Deploying Linux apps on Windows Server is easier these days (Hyper-V). If you really need decent POSIX OS on Windows just use virtualiztion.

Things relying on cuda/opencl generally don't virtualize well, especially not with windows hosts. With Xen you could do GPU passthrough things, but even that isn't easy to setup.

Re: Midipix: Posix for Windows

#65
post #63
post #43

Earlier quoted context omitted.

> For one or the other reason, fork() has become in many discussion of posix on windows something of a fetish. how about this one? fopen + mmap + unlink Afaik windows APIs forbid deleting mmaped files.

With FILE_SHARE_DELETE there shouldn't be any particular problem, but I certainly want to test this with the posix flags you had in mind. Do you happen to have a minimal example that you deem problematic on Windows?

has been a while since I tried it. it was basically

open rw, set length, mmap from two 2 processes to have shared memory, unlink & close.

expected behavior is to have a chunk of shared memory without the tempfile in the directory tree

Re: Midipix: Posix for Windows

#66
No word about debugging. How are they going to port, say, gdb? Are they going to implement ptrace, although it doesn't conform to any standard? If so, it also raises the question of WHICH flavor of ptrace?

Also, POSIX has a number of optional features, like realtime and queued signals. Which features exactly will be supported?

Re: Midipix: Posix for Windows

#67
I work for Microsoft. As you can imagine, this is a problem we have devoted a considerable amount of energy to thinking about. A lot of this discussion has revolved around comparisons to previous solutions (particularly Cygwin and Interix/SUA), but I think it's worth backing up and thinking about what the fundamental limitations of implementing something approaching POSIX compliance in userspace. I will try to tell the story more or less as it is told internally by the people who built these things from scratch.

To be transparent, I am sort of worried that the authors have told us in this thread that they think the number of applications that "really need" `fork` is small, and that the discussions about POSIX subsystems on Windows are overindexed on discussions about `fork` without justification.

The truth of the matter is that the semantics of `fork` infect every API that creates process state. Every library, every syscall that creates process state will have a clear answer for what happens when a process that is using that bit of process state calls `fork`. This includes file and socket management, IPC stuff, threads, signals, and so on. To make `fork` work properly on Windows, you absolutely need to replicate what UNIX does here, or you will break a lot of apps, because make no mistake, a lot of apps depend on these semantics to work correctly. `fork` is not just a function that occasionally gets called and sometimes needs to be fast for people who are calling it. It is core to the semantics of the POSIX API, and if you don't treat it as such you are in for a bad time later.

Perhaps more worrying, though, is that there is a serious impedence mismatch between the UNIX and Windows models of asynchrony. Particularly in the case of sockets and signals, this difference is immense, and I am extremely skeptical we will find a good (or even close to production-worthy) solution in usermode. Maybe these good folks have found something I have missed; to me the approach just seems doomed.

And of course, this is all just the start of the problem. It is a much longer trudge to solve the "real" problem, which is immense. The original Interix POSIX subsystem in NT 3.5 (I'm told) had a `fork` that was just barely enough to pass the 1003.1-1990 validation suite, and fell over quickly if you pushed much harder. But they didn't have to mess with the `fork` implementation to turn that 67kloc into the more robust and conformant SFU 3.x; it was mostly a long tail of things extrernal to fork that just needed to be ironed out.

On top of that, to really have `fork` interop, you would likely need to redefine every Win32 API so it did the "right thing" under UNIXy things like signal delivery and fork, and that is a truly, terrifyingly tall order.

Re: Midipix: Posix for Windows

#68
post #67

I work for Microsoft. As you can imagine, this is a problem we have devoted a considerable amount of energy to thinking about. A lot of this discussion has revolved around comparisons to previous solutions (particularly Cygwin and Interix/SUA), but I think it's worth backing up and thinking about what the fundamental limitations of implementing something approaching POSIX compliance in userspace. I will try to tell t…

These are some good observations. Note that on modern POSIX, calling anything but async-signal-safe functions after forking in a multi-threaded process results in undefined behavior. I think it's totally reasonable to consider any programs using the WinAPI (not just POSIX functions provided through midipix) as being formally multi-threaded, and to consider the WinAPI non-AS-safe. I'm not 100% sure what midipix is doing in this regard, but my advice on the project (I'm the primary author/maintainer of musl libc, which midipix is using) has been not to worry about making arbitrary functions work after fork, but only supporting the things that POSIX requires to work. My view in general is that fork should be phased out, but posix_spawn is not sufficiently powerful yet to replace all uses of fork+exec -- it can't do advanced uid changes, setsid, resource limits, etc., and posix_spawn can't be used effectively from an async signal context because the API (attributes and file actions) inherently involves allocation.

Re: Midipix: Posix for Windows

#69

That's an interesting name choice that might make it hard for me to remember in the future, do you happen to know why they chose it? In any case, I welcome a posix interface for windows, it would be a cool tool to have, especially in the case of cross-platform utilities.

Any reason Cygwin isnt suitable? Thats the goto project for the type of stuff.

The biggest problem with Cygwin is that programs linked with Cygwin inherit global state from from a Cygwin installation on the system they're running on. If you want to produce a Windows program that just runs on any system you install it on using Cygwin, it will work right for most users, but if a power user who has Cygwin installed and has their own custom mounts, options (like different binary/text mode settings), etc. tries to run your program, it might break spectacularly. This makes Cygwin a really poor choice for making binaries you want to distribute as standalone programs.

Aside from that, Cygwin tries to hard to be a complete Unix environment on Windows, whereas midipix just gives you enough to use interfaces that were standardized in POSIX as a reasonable, uniform API for all operating systems to provide. Some functions go beyond that, but you don't have to use them. And even some things that are mandatory in POSIX are optional in midipix; as I understand it, you can choose at build time whether you want the overhead of being able to support tty devices (and the associated semantics like job control, signals from the controlling tty, etc.).

Post reply on HN