Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero-Copy GPU Inference from WebAssembly on Apple Silicon

abacusnoir.com

Post reply on HN
Print view View original

41–50 of 59 posts

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#41

Post by jhayward »

bensyverson wrote: post #6
rvz wrote: post #4

Earlier quoted context omitted.

This sort of obvious pattern is an instant AI dead give-away that I keep on seeing in hundreds of blogs and code posted on this site: "Here is X - it makes Y" "That's not X, it's Y." "...no this, no that, no X, no Y." Another way of telling via code is by deducing the experience of the author if they became an expert of a different language since...yesterday. There will be a time where it will be problematic for thos…

I think the days of on-site interviews with whiteboard tests may be drawing to a close faster than you suspect

Well, there is a long tradition of "testing" developer candidates by asking them to exhibit skills in tasks that they never, ever, do in their work. Like whiteboard coding.

It doesn't have a great success record.

I personally would rather they exhibited expert skills in using tools, and expressing their design insight as a part of that skillset.

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#43

Post by eis »

fho wrote: post #34
eis wrote: post #30

Earlier quoted context omitted.

Apple Silicon uses unified memory where the CPU and GPU use the exact same memory and no copies from RAM to VRAM are needed. The article opens with mentioning just that and indeed it is the whole point of the article.

I am always a bit baffled why Apple gets credited with this. Unified memory has been a thing for decades. I can still load the biggest models on my 10th gen Intel Core CPU and the integrated GPU can run inference. The difference being that modern integrated GPU are just that much faster and can run inference at tolerable speeds. (Plus NPUs being a thing now, but that also started much earlier. Thr 10th gen Intel Core…

I don't think people are crediting Apple with inventing unified memory - I certainly did not. There have been similar systems for decades. What Apple did is popularize this with widely available hardware with GPUs that don't totally suck for inference in combination with RAM that has decent speed at an affordable price. You either had iGPUs which were slow (plus not exactly the fastest DDR memory) but at least sitting on the same die or you had fast dGPUs which had their own limited amount of VRAM. So the choice was between direct memory access but not powerfull or powerfull but strangled by having to go through the PCIE subsystem to access RAM.

The article is talking about one particular optimization that one can implement with Apple Silicon and I at least wasn't aware that it is now possible to do so from WebAssembly - so to completely dismiss it as if it had nothing to do with Apple Silicon is imho not fair.

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#44

Post by pjmlp »

eis wrote: post #43
fho wrote: post #34

Earlier quoted context omitted.

I am always a bit baffled why Apple gets credited with this. Unified memory has been a thing for decades. I can still load the biggest models on my 10th gen Intel Core CPU and the integrated GPU can run inference. The difference being that modern integrated GPU are just that much faster and can run inference at tolerable speeds. (Plus NPUs being a thing now, but that also started much earlier. Thr 10th gen Intel Core…

I don't think people are crediting Apple with inventing unified memory - I certainly did not. There have been similar systems for decades. What Apple did is popularize this with widely available hardware with GPUs that don't totally suck for inference in combination with RAM that has decent speed at an affordable price. You either had iGPUs which were slow (plus not exactly the fastest DDR memory) but at least sittin…

Back in the 8 and 16 bit home computer days, or game consoles for that matter it was popular enough already.

And yes things like the Amiga Blitter, arcade or console graphics units were already baby GPUs.

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#47

Post by littlecranky67 »

fulafel wrote: post #29
littlecranky67 wrote: post #28

Earlier quoted context omitted.

Why did all my x86 onboard iGPU reserve a fixed amount of RAM on boot, inaccessible to the OS? Why do dGPU bring their own VRAM and how to directly manipulate it from the CPU without copying?

To the first question: blame Windows I guess. But even on older chips, GPU code could access memory allocated on the CPU side so this didn't cap the amount of data your GPGPU code could crunch.

I remember this was mostly a BIOS setting how much memory to allocate for iGPU - and once set in the BIOS, that memory was not accessible to the underlying OS (besides GPU I/O).

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#48

Post by kimixa »

fc417fc802 wrote: post #38
ben-schaaf wrote: post #35

Earlier quoted context omitted.

Correct me if I'm wrong, but that reserved memory is for the framebuffer? The iBoot bootloader also reserves some memory for the framebuffer. dGPUs bring their own VRAM because it's a different type of memory, allowing them to get higher performance than they could with DDR. The M4 Max requires 128GB of LPDDR5X to reach its ~500GB/s bandwidth. The RX Vega 64 had that same bandwidth in 2017 with just 8GB of HBM2.

Nope, the reserved memory is what's available to use from the various APIs (VK, GL, etc). More recently there's OS support for flexible on demand allocation by the GPU driver. Of course the APIs have allowed you to make direct use of pointers to CPU memory for something like a decade. However that requires maintaining two separate code paths because doing so while running on a dGPU is _extremely_ expensive.

As someone that's worked on GPU drivers for shared memory systems for over 15 years, supporting hardware that was put on the market over 20 years ago, and they've "always" (in my experience) been able to dynamically assign memory pages to the GPU.

The "reserved" memory is more about the guaranteed minimum to allow the thing to actually light up, and sometimes specific hardware blocks had more limited requirements (e.g. the display block might require contiguous physical addresses, or the MMU data/page tables themselves) so we would reserve a chunk to ensure they can actually be allocated with those requirements. But they tended to be a small proportion of the total "GPU Memory used".

Sure, sharing the virtual address space is less well supported, but the total amount of memory the GPU can use is flexible at runtime.

Re: Zero-Copy GPU Inference from WebAssembly on Apple Silicon

#50

Post by tancop »

jedisct1 wrote: post #46

Doesn't work on web browsers, only with one headless runtime, one one CPU architecture. What's even the point of using webassembly here?

loading third party agents in a sandbox with full custom model support. right now you need to either run that code directly (super dangerous) use a vm/container (slow and complicated) or a interpreter like lua (language bound, slow and weak security). wasm is perfect for this, its almost native speed, built for security and language neutral. onnx and coreml are secure but they can only do the actual model not all the code around it.
Post reply on HN