Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero-Copy GPU Inference from WebAssembly on Apple Silicon

abacusnoir.com

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#31

Post by porridgeraisin »

trueno wrote: post #25
notepad0x90 wrote: post #14

Earlier quoted context omitted.

I don't know, to me your sentiment sounds a lot like how back in the day they used to say "you can't just use a calculator all the time, use your brain and show the work on pen and paper". humans have been using tools to communicate since pre-history. language itself is one tool of communication invented to supersede body-language and grunting and noises. the thought and idea is theirs, it was communicated. Would it…

Calculators have never been the medium in which we communicate our human experience and knowledge transfer. Calculators aren't part of the social fabric or culture. Very 2d extrapolation that somehow resulted in an alleged parallel. Language is woven deeply into civilization and our histories & been a part of our species literal survival against the most unforgiving odds/environments. Using what is effectively a ghos…

It's not that deep man, it's just a blog post about some software library. There's no civilisational communication going on here, relax. This whole thing will become irrelevant in a few decades before the end of our lifespans. It's just never that deep.

Why does it matter if it's their thought or not. If you currently care about GPU inference from webassembly on apple silicon, you can use this article. That's really about it.

Now if you care about GPU inference from wasm on apple silicon, and you found problems with this articles content, then great, comment about it. If you say that the problem with the content is due to the usual surface level slop LLMs belt out, then great complain about LLMs. But your comment didn't say anything about gpu inference from wasm on apple silicon.

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#33

Post by itamos »

On one side it sounds promising to exploit shared memory properties to speed up inference. But on the other hand, the well established inference engines are perhaps already well optimized to overlap compute and communication efficiently. In this case the host-device copies are likely not a problem to tackle.

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#34

Post by fho »

eis wrote: post #30
nl wrote: post #26

I'm pretty sure this is just "yes (parts of), memory control in WASM works"[1]. The whole Apple Silicon thing is (in this case) just added details that don't actually matter. [1] https://github.com/WebAssembly/memory-control/blob/main/prop...

Apple Silicon uses unified memory where the CPU and GPU use the exact same memory and no copies from RAM to VRAM are needed. The article opens with mentioning just that and indeed it is the whole point of the article.

I am always a bit baffled why Apple gets credited with this. Unified memory has been a thing for decades. I can still load the biggest models on my 10th gen Intel Core CPU and the integrated GPU can run inference.

The difference being that modern integrated GPU are just that much faster and can run inference at tolerable speeds.

(Plus NPUs being a thing now, but that also started much earlier. Thr 10th gen Intel Core architecture already had instructions to deal with "AI" workloads... just very preliminary)

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#35

Post by ben-schaaf »

littlecranky67 wrote: post #28
fulafel wrote: post #27

> Apple Silicon changes the physics. The CPU and GPU share the same physical memory (Apple's Unified Memory Architecture) ... no bus! Beware the reality distortion field: This is of course how it's worked on most x86 machines for a long time. And also on most Macs when they were using Intel chips.

Why did all my x86 onboard iGPU reserve a fixed amount of RAM on boot, inaccessible to the OS? Why do dGPU bring their own VRAM and how to directly manipulate it from the CPU without copying?

Correct me if I'm wrong, but that reserved memory is for the framebuffer? The iBoot bootloader also reserves some memory for the framebuffer.

dGPUs bring their own VRAM because it's a different type of memory, allowing them to get higher performance than they could with DDR. The M4 Max requires 128GB of LPDDR5X to reach its ~500GB/s bandwidth. The RX Vega 64 had that same bandwidth in 2017 with just 8GB of HBM2.

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#37

Post by mirekrusin »

fho wrote: post #34
eis wrote: post #30

Earlier quoted context omitted.

Apple Silicon uses unified memory where the CPU and GPU use the exact same memory and no copies from RAM to VRAM are needed. The article opens with mentioning just that and indeed it is the whole point of the article.

I am always a bit baffled why Apple gets credited with this. Unified memory has been a thing for decades. I can still load the biggest models on my 10th gen Intel Core CPU and the integrated GPU can run inference. The difference being that modern integrated GPU are just that much faster and can run inference at tolerable speeds. (Plus NPUs being a thing now, but that also started much earlier. Thr 10th gen Intel Core…

That’s shared, not unified, it’s partitioned where cpu and gpu copies are managed by driver. Lunar lake (2024) is getting closer but still not as tightly integrated as apple and capped to 32GB only (Apple has up to 512GB). AMD ryzen ai max is closer to Apple but still 3 times slower memory.

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#38

Post by fc417fc802 »

ben-schaaf wrote: post #35
littlecranky67 wrote: post #28

Earlier quoted context omitted.

Why did all my x86 onboard iGPU reserve a fixed amount of RAM on boot, inaccessible to the OS? Why do dGPU bring their own VRAM and how to directly manipulate it from the CPU without copying?

Correct me if I'm wrong, but that reserved memory is for the framebuffer? The iBoot bootloader also reserves some memory for the framebuffer. dGPUs bring their own VRAM because it's a different type of memory, allowing them to get higher performance than they could with DDR. The M4 Max requires 128GB of LPDDR5X to reach its ~500GB/s bandwidth. The RX Vega 64 had that same bandwidth in 2017 with just 8GB of HBM2.

Nope, the reserved memory is what's available to use from the various APIs (VK, GL, etc). More recently there's OS support for flexible on demand allocation by the GPU driver.

Of course the APIs have allowed you to make direct use of pointers to CPU memory for something like a decade. However that requires maintaining two separate code paths because doing so while running on a dGPU is _extremely_ expensive.

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#39

Post by fc417fc802 »

mirekrusin wrote: post #37
fho wrote: post #34

Earlier quoted context omitted.

I am always a bit baffled why Apple gets credited with this. Unified memory has been a thing for decades. I can still load the biggest models on my 10th gen Intel Core CPU and the integrated GPU can run inference. The difference being that modern integrated GPU are just that much faster and can run inference at tolerable speeds. (Plus NPUs being a thing now, but that also started much earlier. Thr 10th gen Intel Core…

That’s shared, not unified, it’s partitioned where cpu and gpu copies are managed by driver. Lunar lake (2024) is getting closer but still not as tightly integrated as apple and capped to 32GB only (Apple has up to 512GB). AMD ryzen ai max is closer to Apple but still 3 times slower memory.

Shared vs unified is merely a driver implementation detail. Regardless, in practice (IIUC) data is still going to be copied if you perform a transfer using a graphics API because the driver has no way of knowing what the host might do with the pointed-to memory after the transfer.

If you make use of host pointers and run on an iGPU no copy will take place.

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#40

Post by fho »

mirekrusin wrote: post #37
fho wrote: post #34

Earlier quoted context omitted.

I am always a bit baffled why Apple gets credited with this. Unified memory has been a thing for decades. I can still load the biggest models on my 10th gen Intel Core CPU and the integrated GPU can run inference. The difference being that modern integrated GPU are just that much faster and can run inference at tolerable speeds. (Plus NPUs being a thing now, but that also started much earlier. Thr 10th gen Intel Core…

That’s shared, not unified, it’s partitioned where cpu and gpu copies are managed by driver. Lunar lake (2024) is getting closer but still not as tightly integrated as apple and capped to 32GB only (Apple has up to 512GB). AMD ryzen ai max is closer to Apple but still 3 times slower memory.

My last serious GPU programming was with OpenCL. And if my memory does not fail me the API was quite specific about copying and/or sharing memory on a shared memory system.

I am pretty sure that my old 10th gen CPU/GPU combo has the ability to use the "unified"/zero-copy access mode for the GPU.

Post reply on HN