How different is the 32-bit Windows application environment from the 64-bit Linux application environment on x64 systems, as far as how system calls are invoked goes, and how memory is accessed?
We had a somewhat similar issue of wanting to run code from a different, narrower, system, although the systems involved were not anywhere near as different as Windows and Linux, back in the mid '80s at ISC. We were doing the 386 port of System V Release 3 under contract from AT&T. Part of the contract required that we add the ability to run binaries from the 286 ports of System V Release 2 and System V Release 3 [1].
The approach was similar to what Wine does, if my limited understanding of Wine is correct. We wrote a program, i286emul, which could be pointed at a 286 program. It would allocate memory for the 286 program and load the program into that memory.
386 Unix was using a simple paging memory model. The 386 still used the selector/offset system even when using paging, but the selectors were loaded once when a program started and not touched again. I think it was using what, in 16-bit land, would be called "small model" (one code segment, one data/stack segment), but with all the segments set to 4 GB, and the real memory management done via that paging system that operated below the output of the selector/offset system.
We had to add a system call (or maybe it was a new command added to an existing ioctl...I don't remember for sure) that allowed i286emul to ask the kernel to set up some 16-bit selectors referencing the parts of the i286emul 32-bit address space at which it had loaded the 16-bit program, or at which it had allocated memory for the 16-bit program's data and stack.
That let i286emul load a 16-bit program into i286emul's address space, get the selectors set up, and then it could load the selectors and jump into the 16-bit program.
Eventually, the 16-bit program would try to do a system call. Fortunately, 16-bit 286 Unix and 32-bit 386 Unix happened to use different mechanisms for invoking system calls, so that when 16-bit tried to do a 286 Unix system call it would be an illegal instruction or a segfault or something like that (I don't remember the exact mechanism it used), instead of something that would try to invoke a 386 Unix system call. (As far as I know, this was pure luck that the people who decided on the 386 system call interface mechanism picked something different from the 286 system call interface mechanism).
So, to handle system calls we just had to have i286emul handle the fault, recognize that it was caused by the 286 program trying to do a system call, and emulate that call. I don't remember if that was as simple as installing a signal handler for the fault, or if we needed some kernel help.
The only other kernel change I remember was making exec recognize 286 binaries, and turn the exec into an i286emul exec, with the path to the 286 program passed as an additional argument.
Later, when AT&T and Microsoft made a deal to add Xenix binary compatibility to 386 Unix, we got the contract for that, leading to x286emul, which was very similar to i286emul. There were a couple of minor areas in which Xenix and Unix differed that we could not take care of in user mode, so had to add a kernel flag [2] to have it slightly change some behavior for x286emul.
How hard would it be to take a similar approach with 32-bit Windows on 64-bit Linux? That is, make it so that a Linux process can reasonably mix 32-bit and 64-code, and give it some control over how 32-bit code sees the processes address space, and then implement a version of Wine that can handle 32-bit Windows programs translating their system calls to 64-bit Linux calls? The issues I can see include:
1. What is actually different between 32-bit code execution and 64-code execution as far as the kernel goes? Is there some flag that marks 32-bit/64-bit, or is it mostly that 32-bit code is simply going to use addressing mode that refer to the double word registers instead of the quad word registers?
2. We greatly benefited on i286emul/x286emul by the very different memory models of the two systems. I'd expect 32-bit Windows and 64-bit Unix to have very similar models which could be a problem...but I'd expect 32-bit Windows and 32-bit Linux to be even closer, and Wine dealt with that, so I'm not sure 32-bit/64-bit would add much difficulty.
3. We also benefited from 286 Unix and Xenix being similar to 386 Unix, so mostly 286 system calls mapped directly to corresponding 386 system calls, with just a little translation of arguments and return codes. Windows and Linux are very different. But Wine is already dealing with that, so as with memory model it is not obvious that 32-bit vs. 64-bit makes much difference.
[1] ...not that there were actually any System V Release 3 on the 286 programs anyone cared about, because as far as I know that port was never completed. We were doing that one for AT&T, too, but there were too many assumptions about memory management baked into the 3B2 source that were not right for the very different memory environment of the 286, and maybe halfway through the port AT&T realized that Release 3 on the 286 was going to suck unless we did some major rewriting rather than just porting, and the 386 was so much better than on one was going to give a damn about running Release 3 on a 286 even if it did not suck, and dropped the port.
[2] That stupid flag led to the most annoying meeting I have ever attended. For x286emul, we were just doing the user mode code. Microsoft was doing any kernel changes we needed. They said an issue arose with our request for this flag that could not be resolved by email or phone. Me and the other guy who were writing x286emul had to fly from LAX to SEATAC, go all the way over to Redmond, to resolve this thing in person.
We got to the meeting, and they presented the issue: should controlling the flag be via a new system call, or via a flag on the system call (or ioctl...I never can remember which it was) that is used for some of the other 286 emulation control? We gave out preference on that, and were sent home. How the heck could that not have been handled by email or phone?