About the Rosetta Translation Environment • Hacker News Board

Re: About the Rosetta Translation Environment

#81

Post by DCKing » Tue, Jun 23, 2020, 3:30 PM UTC

Angostura wrote: post #59

DCKing wrote: post #40

Rosetta's goal is to support legacy Mac apps. It's quite finely scoped, but I hope Apple will go above this and make it available as part of their virtualization framework. This way e.g. Parallels and Docker could use this to provide some way to tap in probably the fastest way to run x86 on ARM Macs. This would mean Rosetta would go beyond its stated scope, and certainly go beyond what the relatively short-lived Rose…

FWIW they showed the latest Tombraider game running on Rosetta on a dev Mac said to be using the ARM chip from the high-end iPad. It ran pretty well. Not sure if I would quite count this as AAA

I don't think that was running on Rosetta 2's dynamic recompiler, but instead used Rosetta 2's other stated feature of being able to translate existing apps at install time. In the demo they state that they downloaded the game from the Mac App store.

Certainly static recompilation yields somewhat-close-to-native performance when it is available, but that is not possible for x86 "VMs".

Re: About the Rosetta Translation Environment

#82

Post by makomk » Tue, Jun 23, 2020, 3:31 PM UTC

beervirus wrote: post #24

jayyhu wrote: post #17

>Rosetta translates all x86_64 instructions, but it doesn’t support the execution of some newer instruction sets and processor features, such as AVX, AVX2, and AVX512 vector instructions. No AVX will probably mean that the vast majority of pro/graphics intensive apps won't work out of the box with Rosetta.

Do those programs blindly try to execute AVX instructions without checking whether they’re supported? What happens if you're just running on an older intel processor?

All well-behaving software will of course check if AVX is supported before trying to use it, but at least in some cases that just means they'll refuse to install or run on systems that don't have it: https://www.pro-tools-expert.com/production-expert-1/2019/9/... (Developing parallel code paths and keeping them working indefinitely is a lot of extra work, and AVX has been around for close to a decade at this point.)

Re: About the Rosetta Translation Environment

#83

Post by numpad0 » Tue, Jun 23, 2020, 3:36 PM UTC

cpuguy83 wrote: post #19

marcinzm wrote: post #4

>However, Rosetta doesn’t translate the following executables: Virtual Machine apps that virtualize x86_64 computer platforms How will this impact docker? Does this mean you can't run x86 docker containers on new Apple laptops?

Apple mentioned they are specifically "working with Docker to support 'these things' in coming months". Confirmed with some Docker folks they are working on "something". All very nondescript, but they said they can't talk about it yet.

So I had to look this up: Docker do not rely on VT-x. If kernel has cgroups it runs.

Hypothetically it could run natively on Windows NT, Linux, Darwin, Solaris, etc if it ported for appropriate APIs. However, most Docker containers contain x86_64 Linux binaries, so the runtime environment has to be binary compatible with x86_64 Linux.

Thus, instead of going through the pain similar to WSL1, Docker people just spin up a midsize Linux VM, run remote controlled Docker daemon in it and call that as the "Docker for [PLATFORM]". This will suffice because use case of Docker is to debug on a laptop and deploy to GNU/Linux servers.

So if ARM macOS runs an x86_64 Ubuntu VM it also means Docker runs.

Re: About the Rosetta Translation Environment

#84

Post by qubex » Tue, Jun 23, 2020, 3:38 PM UTC

Bigpet wrote: post #44

godzillabrennus wrote: post #38

The Apple transition to ARM for me is sad. There was so much positivity around the Intel transition. It opened up the Mac platform. Now it’s going back into a closed black box.

I think I'm missing something. How is ARM closing things up? Does Apple have a bunch of processor extensions that make it incompatible with other ARM processors? Or do you think they'll take the chance to kill boot-camp or something?

The “ARM world” is almost inconceivably varied by x86 standards. There’s not even an agreed-upon boot procedure (the Raspberrry Pi’s CPU famously gets booted by the GPU). Efforts to standardise this in the server space are only just getting off the ground with the Arm Server Base Boot Requirements version 1.2 being ratified last year. I doubt you’ll find Apple subscribing to anything that they won’t develop in-house.

This is, after all, the company that developed the Lightening connector and sticks on its iPhones to it despite the whole industry shifting to USB-C.

Basically the OS situation on Macs post-ARM is probably going to look like the Linux situation on iDevices currently: occasionally somebody manages to text-mode-boot an iPhone 7 into a profoundly unstable hacked-up kernel and there is much merriment and rejoicing, but really... macs are going to become rather exotic (on the inside).

Have you ever wondered about your iPad’s boot sequence? Because you’ve never had to think about it. Notice how you can’t install versions of i(Pad)OS that are no longer signed? You think Apple will allow booting unsigned images? And all the Secure Enclave infrastructure they’ve built up?

Nah.

Re: About the Rosetta Translation Environment

#85

Post by api » Tue, Jun 23, 2020, 3:38 PM UTC

lsllc wrote: post #53

mkl wrote: post #41

> Rosetta can translate most Intel-based apps, including apps that contain just-in-time (JIT) compilers. How on Earth does it do that? If executable code is being generated at runtime, it's going to be x86_64 binary machine code still (there are too many ways to generate valid machine code, and it won't know right away whether you're JITting, or cross compiling and actually want x86_64), so Rosetta would need to dete…

It probably just hooks mmap() (or mprotect()) and looks for when the JIT sets PROT_EXEC on memory it hasn't translated yet. Here's a tutorial for a simple JIT that uses mmap() with PROT_EXEC to write machine code to before executing: https://github.com/spencertipping/jit-tutorial

"Just"

This is some serious black magic, especially if it actually works. Even if it only works some of the time I give it an A for effort, since they could easily have just said "JITs will not work with Rosetta." Most applications do not contain JITs.

My guess is there are enough important Mac apps that contain JITs of some form to merit this work, probably high performance math and graphics kernels that JIT processing pipelines or something like that. I wonder if some pro studio editing applications do this.

Re: About the Rosetta Translation Environment

#86

Post by cpuguy83 » Tue, Jun 23, 2020, 3:39 PM UTC

marcinzm wrote: post #10

EtienneK wrote: post #8

Earlier quoted context omitted.

I would assume you’d only be able to run ARM Docker containers.

Which is less than ideal if you're using x86 servers since you no longer have the same dev containers as production containers.

I don't think people should consider the vm Docker For Mac uses anything close to resembling production as is.

Re: About the Rosetta Translation Environment

#87

Post by andoma » Tue, Jun 23, 2020, 3:39 PM UTC

I wonder how it handles the stricter memory ordering from x86? For example, this code:

  void store_value_and_unlock(int *p, int *l, int v)
  {
    *p = v;
    __sync_lock_release(l, 0);
  }

on x86_64 compiles to:

   mov     dword ptr [rdi], edx
   mov     dword ptr [rsi], 0
   ret

vs on arm64:

   str     w2, [x0]
   stlr    wzr, [x1]
   ret

Notice how the second store on ARM is a store with release semantics to ensure correct memory ordering as it was intended in the original C code. This information is lost as it's not needed on x86 which guarantees (among other things) that stores are visible in-order from all other threads.

Re: About the Rosetta Translation Environment

#88

Post by selsta » Tue, Jun 23, 2020, 3:42 PM UTC

perardi wrote: post #63

dsabanin wrote: post #3

From the page: What Can't Be Translated? Rosetta can translate most Intel-based apps, including apps that contain just-in-time (JIT) compilers. However, Rosetta doesn’t translate the following executables: * Kernel extensions * Virtual Machine apps that virtualize x86_64 computer platforms Some people are not going to be happy about this. Edit: But I personally am okay with that.

Apple wants kexts gone. They announced the deprecation last year, and this is another chance to force the change. (And for good reasons, if my layman's understanding of the security is at all correct.) [ https://developer.apple.com/support/kernel-extensions/](http...

Apple ARM will support kexts.

https://developer.apple.com/documentation/apple_silicon/inst...

Re: About the Rosetta Translation Environment

#89

Post by lsllc » Tue, Jun 23, 2020, 3:45 PM UTC

monocasa wrote: post #57

lsllc wrote: post #53

Earlier quoted context omitted.

It probably just hooks mmap() (or mprotect()) and looks for when the JIT sets PROT_EXEC on memory it hasn't translated yet. Here's a tutorial for a simple JIT that uses mmap() with PROT_EXEC to write machine code to before executing: https://github.com/spencertipping/jit-tutorial

Since they don't enforce W^X you need to go a little deeper than that since some JITs reuse the same memory later for new traces. So what you'll do is silently enforce W^X to trap code modifications, but not propagate those traps to the emulated code.

Right, if you look at the example JIT project I posted, they start by creating the memory with mmap() and PROT_EXEC before generating any code. So yes, you'd need to trap subsequent writes to allow for retranslation [I assume they have hooks in the Darwin kernel for this].

Re: About the Rosetta Translation Environment

#90

Post by needusername » Tue, Jun 23, 2020, 3:47 PM UTC

api wrote: post #85

lsllc wrote: post #53

Earlier quoted context omitted.

It probably just hooks mmap() (or mprotect()) and looks for when the JIT sets PROT_EXEC on memory it hasn't translated yet. Here's a tutorial for a simple JIT that uses mmap() with PROT_EXEC to write machine code to before executing: https://github.com/spencertipping/jit-tutorial

"Just" This is some serious black magic, especially if it actually works. Even if it only works some of the time I give it an A for effort, since they could easily have just said "JITs will not work with Rosetta." Most applications do not contain JITs. My guess is there are enough important Mac apps that contain JITs of some form to merit this work, probably high performance math and graphics kernels that JIT process…

> Most applications do not contain JITs.

Java apps and JavaScript apps (Electron) come to mind.