Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Midipix: Posix for Windows

midipix.org

Post reply on HN
Print view View original

71–80 of 92 posts

Re: Midipix: Posix for Windows

#71

Post by cygx »

dalias wrote: post #50
hsivonen wrote: post #12

Having fopen() that takes utf-8 on Windows is a huge deal. I already want to investigate using this for Firefox on Windows for this reason alone.

Yes, this is the really big deal that everyone focused on fork and mmap semantics and other details is overlooking. Having midipix as the means of producing Windows versions of cross-platform software like Firefox should make it possible to remove a lot of ugly #ifdeffery and/or whole "portability layers" that are avoiding the standard functions like fopen because of a lack of Unicode support on Windows.

I assume midipix uses the same approach as Scheme 48, Racket and Rust[1] to deal with ill-formed UTF-16?

[1] https://simonsapin.github.io/wtf-8/

Re: Midipix: Posix for Windows

#72

Post by mpixorg »

antics wrote: post #67

I work for Microsoft. As you can imagine, this is a problem we have devoted a considerable amount of energy to thinking about. A lot of this discussion has revolved around comparisons to previous solutions (particularly Cygwin and Interix/SUA), but I think it's worth backing up and thinking about what the fundamental limitations of implementing something approaching POSIX compliance in userspace. I will try to tell t…

There are two main scenarios for fork(), and the lack of distinction between them is what I suspect led to some of the articulated worries.

The first scenario, in the spirit of traditional unix, consists of the sequence fork+execve. This scenraio is supported in midipix, yet requires that the (optional) subsystem performs execve on behalf of the forkee. As noted earlier, if all the child does is call execve, then application authors should consider switching to posix_spawn not only due to its better portability, but also for performance reasons.

The second, and arguably more interesting scenario, takes place when the child brnaches away from the parent process, performs (in parallel) a designated portion of a larger task, and then exits. This second scenario has been thoroughly tested and works as expected with respect to IPC, I/O, signal delivery, and nested forking.

In either of the above scenarios, there is no expectation that the child could interact with csrss, create or operate on GDI handles, or otherwise use the WIN32 API directly. Then again, it is expected that a process that calls fork() will not have any of the GUI libraries loaded at the time the call is made. Any limitation here, if any, has little to no relevance to existing applications that depend of fork, but only to what I fondly call the "fetishization of fork" in discussions about posix-to-windows portability. Applications that truly depend on fork were written with neither WIN32 nor GDI in mind, and therefore will not break due to such windows-specific interfaces not being available (as an aside, an X11 application calling fork() from within its event loop is an equally bad design regardless of whether the operating system supports that or not).

Looking forward to further cooperation! I hope this addresses at least some of the questions you have raised.

Re: Midipix: Posix for Windows

#73

Post by e12e »

TickleSteve wrote: post #16
wfunction wrote: post #15

This project sounds too good to be true. I would be quite astonished if they can make fork() work seamlessly without an amazingly high performance cost. In my experience the only way to do such a thing is to use NtCreateProcess(), but that function itself seems impossible for anyone besides Microsoft to use correctly. (I have tried numerous times and failed, and so have many others.)

[ https://www.cygwin.com/faq.html#faq.api.fork ] Here's how it works: Parent initializes a space in the Cygwin process table for child. Parent creates child suspended using Win32 CreateProcess call, giving the same path it was invoked with itself. Parent calls setjmp to save its own context and then sets a pointer to this in the Cygwin shared memory area (shared among all Cygwin tasks). Parent fills in the child's .d…

After reading that, the idea behind colinux[1] doesn't seem quite as crazy. Doesn't look like it's been ported to 64bit, though :(

[1] http://www.colinux.org/

Re: Midipix: Posix for Windows

#74

Post by crpatino »

ElijahLynn wrote: post #25

My first response after reading the headline... But why?

If you create a product that runs on client's hardware, you want to be able to run consistently in as many platforms as the majority of your potential customer's. You are leaving money on the table if you don't.

If a subset of those platforms is Linux + (Commercial) Unixes + Windows, the Windows part is going to take twice as long and cost you thrice as much to get in line with the other 2... unless you have some clever solution like the one proposed here.

Re: Midipix: Posix for Windows

#75

Post by mpixorg »

cygx wrote: post #71
dalias wrote: post #50

Earlier quoted context omitted.

Yes, this is the really big deal that everyone focused on fork and mmap semantics and other details is overlooking. Having midipix as the means of producing Windows versions of cross-platform software like Firefox should make it possible to remove a lot of ugly #ifdeffery and/or whole "portability layers" that are avoiding the standard functions like fopen because of a lack of Unicode support on Windows.

I assume midipix uses the same approach as Scheme 48, Racket and Rust[1] to deal with ill-formed UTF-16? [1] https://simonsapin.github.io/wtf-8/

Actually no... the application makes all calls using utf-8, and is expected to provide it in a well-formed manner so that the system call layer could convert it to utf-16. In the reverse route, where utf-16 is read by the system call layer and then converted to utf-8 (getdents(2) and friends), it is expected that file names be in well-formed utf-16. For a file-system volume to have ill-formed utf-16 name entries would make for an interesting case... have not encountered that yet, but will certainly look into that.

Re: Midipix: Posix for Windows

#76

Post by ised »

Was POSIX ever really intended to apply to a non-UNIX OS, e.g., VMS?

The Windows kernel is based on the VMS kernel, right?

And NTFS is based on the VMS filesystem?

http://en.wikipedia.org/wiki/David_cutler

I would have been happy with VMS on the PC.

Instead we got Windoze. How much of our lives has this monstrosity wasted? Just let it die.

Do daemontools' supervise and svscan need fork()?

Re: Midipix: Posix for Windows

#77

Post by mpixorg »

edwintorok wrote: post #28

Maybe I'm missing something obvious but where do I download the installer? Or do I have to (cross-)build it all from source using mingw-w64?

The general design allows for building the toolchain in its entirety (libc, libcgcc, libstdc++) independent of the runtime components. The idea behind this is that changes to the toolchain will fairly soon become rare [but see below], whereas changes to the runtime will be very frequent at least in the first couple of years.

Building the cross-compiler requires a native compiler and a shell environemnt that is capable of building gcc. The build process is trivial, and thus far has been tested on several Linux flavors (just asked that this also gets tested on BSD and OSX). Building gcc in an msys/cygwin environemnt is always a bit more tricky, so we have not spent much time trying that. Instead, we are working hard to become self-hosted as soon as possible.

+ building the cross-compiler:

git clone git://midipix.org/cbb/cbb-gcc && cd cbb-gcc && ./cbb-midipix-cross-gcc.sh

yep, that's that. This will use $HOME/temp as a temporary folder, and install the toolchain to $HOME/midipix. Make sure you add $HOME/midipix/bin to your path in order to use the toolchain. As with all other gcc builds, you need to have the usual dependencies on the build system (gmp,mpfr,mpc,libelf,texinfo) and a working shell environment.

+ pre-pre-alpha, radical changes: some major changes to the toolchain are underway. If you find it hard to believe that building the cross-toolchain is that easy please go ahead and run the above commands from a nearby shell, but please also rebuild it in a day or two (look for a commit message mentioning "automatic creation of GOT entries" :))

+ as a kind reminder, cross-compiling is just for building applications; testing them can only be done with the runtime library, which is not out yet.

Re: Midipix: Posix for Windows

#78

Post by JoeAltmaier »

ised wrote: post #76

Was POSIX ever really intended to apply to a non-UNIX OS, e.g., VMS? The Windows kernel is based on the VMS kernel, right? And NTFS is based on the VMS filesystem? http://en.wikipedia.org/wiki/David_cutler I would have been happy with VMS on the PC. Instead we got Windoze. How much of our lives has this monstrosity wasted? Just let it die. Do daemontools' supervise and svscan need fork()?

Posix was an API standard, implementable anywhere. The very first POSIX implementation I assisted with, and it ran on CTOS which had a message-passing kernel and built-in networking. Nothing like Unix or Windows.

Re: Midipix: Posix for Windows

#79

Post by mpixorg »

mpixorg wrote: post #77
edwintorok wrote: post #28

Maybe I'm missing something obvious but where do I download the installer? Or do I have to (cross-)build it all from source using mingw-w64?

The general design allows for building the toolchain in its entirety (libc, libcgcc, libstdc++) independent of the runtime components. The idea behind this is that changes to the toolchain will fairly soon become rare [but see below], whereas changes to the runtime will be very frequent at least in the first couple of years. Building the cross-compiler requires a native compiler and a shell environemnt that is capabl…

Almost forgot to mention: the current cross-compiler is based on gcc-4.6.4 since that is the last modern gcc which does not depend on C++, and is therefore easier and faster to build and run. Porting subsequent gcc versions, and likewise clang and cparser, is a high priority, and will follow the initial release of the runtime components.

Re: Midipix: Posix for Windows

#80

Post by TickleSteve »

dalias wrote: post #69
TickleSteve wrote: post #13

Earlier quoted context omitted.

Any reason Cygwin isnt suitable? Thats the goto project for the type of stuff.

The biggest problem with Cygwin is that programs linked with Cygwin inherit global state from from a Cygwin installation on the system they're running on. If you want to produce a Windows program that just runs on any system you install it on using Cygwin, it will work right for most users, but if a power user who has Cygwin installed and has their own custom mounts, options (like different binary/text mode settings)…

It used to be like that... not anymore.

Cygwin works fine with multiple installations these days [https://cygwin.com/faq/faq.html#faq.using.multiple-copies]

The thing that wont work is if you try to mix and match a dll from one installation with binaries from another, but I think you can agree that that situation is fair. You just need your paths setup correctly.

Post reply on HN