Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust’s Standard Library on the GPU

vectorware.com

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Rust’s Standard Library on the GPU

#51

Post by zozbot234 »

solaarphunk wrote: post #48

I've been building something similar (GPU-native OS research project) and wanted to share a mental model shift that unlocked things for me. The question "why run CPU code on GPU when GPU cores are slower?" assumes you're running ONE program. But GPUs execute in SIMD wavefronts of 32 threads - and here's the trick: each of those 32 lanes can run a DIFFERENT process. Same instruction, different data. Calculator on lane…

If you're running vastly different processes in different ALU lanes, the single master "program" that comprises them all is effectively an interpreter. And then it's hard to have the exact same control flow lead to vastly different effects in different processes, especially once you account for branches. This works well for inference batches since those are essentially about straight-line processing, but not much else.

Re: Rust’s Standard Library on the GPU

#52

Post by JonChesterfield »

codedokode wrote: post #21

I think it is possible to run CPU code on GPU (including the whole OS), because GPU has registers, memory, arithmetic and branch instructions, and that should be enough. However, it will be able to use only several cores from many thousands because GPU cores are effectively wide SIMD cores, grouped into the clusters, and CPU-style code would use only single SIMD lane. Am I wrong?

Merely mislead by marketing. The x64 arch has 512bit registers and a hundred or so cores. The gpu arch has 1024bit registers and a few hundred SMs or CUs, being the thing equivalent to an x64 core.

The software stacks running on them are very different but the silicon has been converging for years.

Re: Rust’s Standard Library on the GPU

#53

Post by JonChesterfield »

solaarphunk wrote: post #48

I've been building something similar (GPU-native OS research project) and wanted to share a mental model shift that unlocked things for me. The question "why run CPU code on GPU when GPU cores are slower?" assumes you're running ONE program. But GPUs execute in SIMD wavefronts of 32 threads - and here's the trick: each of those 32 lanes can run a DIFFERENT process. Same instruction, different data. Calculator on lane…

It'll go much faster if you give each process a warp instead of a thread. That means each process has its own IP and set of vector registers, and when your editor takes a different branch to your browser, no cost.

Re: Rust’s Standard Library on the GPU

#54

Post by fulafel »

saagarjha wrote: post #45
fulafel wrote: post #27

Earlier quoted context omitted.

GPUs having have thousands of cores is just a silly marketing newspeak. They rebranded SIMD lanes "cores". For eaxmple Nvidia 5000 series GPUs have 50-170 SMs which are the equivalent of cpu cores there. So a more than desktops, less than bigger server CPUs. By this math each avx-512 cpu core has 16-64 "gpu cores".

Each SM can typically schedule 4 warps so it’s more like 400 “cores” each with 1024-bit SIMD instructions. If you look at it this way, they clearly outclass CPU architectures.

This level corresponds to SMT in CPUs I gather. So you can argue your 192 core EPYC server cpu has 384 "vCPUs" since execution resources per core are overprovisioned and when execution blocks waiting for eg memory another thread can run in its place. As Intel and AMD only do 2-way SMT this doesn't make the numbers go up as much.

The single GPU warp is both beefier and wimpier than the SMT thread: they're in-order barely superscalar, whereas on CPU side it's wide superscalar big-window OoO brainiac. But on the other hand the SM has wider SIMD execution resources and there's enough througput for several warps without blocking.

A major difference is how the execution resources are tuned to the expected workloads. CPU's run application code that likes big low latency caches and high single thread performance on branchy integer code, but it doesn't pay to put in execution resources for maximizing AVX-512 FP math instructions per cycle or increasing memory bandwidth indefinitely.

Re: Rust’s Standard Library on the GPU

#55

Post by saagarjha »

fulafel wrote: post #54
saagarjha wrote: post #45

Earlier quoted context omitted.

Each SM can typically schedule 4 warps so it’s more like 400 “cores” each with 1024-bit SIMD instructions. If you look at it this way, they clearly outclass CPU architectures.

This level corresponds to SMT in CPUs I gather. So you can argue your 192 core EPYC server cpu has 384 "vCPUs" since execution resources per core are overprovisioned and when execution blocks waiting for eg memory another thread can run in its place. As Intel and AMD only do 2-way SMT this doesn't make the numbers go up as much. The single GPU warp is both beefier and wimpier than the SMT thread: they're in-order bar…

Right, but the CPU does not have a matrix multiply unit or high bandwidth memory.

Re: Rust’s Standard Library on the GPU

#56

Post by fulafel »

saagarjha wrote: post #55
fulafel wrote: post #54

Earlier quoted context omitted.

This level corresponds to SMT in CPUs I gather. So you can argue your 192 core EPYC server cpu has 384 "vCPUs" since execution resources per core are overprovisioned and when execution blocks waiting for eg memory another thread can run in its place. As Intel and AMD only do 2-way SMT this doesn't make the numbers go up as much. The single GPU warp is both beefier and wimpier than the SMT thread: they're in-order bar…

Right, but the CPU does not have a matrix multiply unit or high bandwidth memory.

Yep. But from the point of view of running CPU-style code on GPUs (eg Rust std lib) and how the "thousands of cores" fiction relates those are less relevant.

And for GenAI matrix math there's of course all the non-gpu acceleration features in various shapes and forms, like the on-chip edge tpu on G phones or Intel and Apple's name things that are both called AMX.

Re: Rust’s Standard Library on the GPU

#57

Post by radarsat1 »

storystarling wrote: post #38
radarsat1 wrote: post #36

Earlier quoted context omitted.

I don't know what the pros are doing but I'd be a bit shocked if it isn't already done this way in real production systems. And it doesn't feel like porting the standard library is necessary for this, it's just some logic.

Raw CUDA works for the heavy lifting but I suspect it gets messy once you implement things like grammar constraints or beam search. You end up with complex state machines during inference and having standard library abstractions seems pretty important to keep that logic from becoming unmaintainable.

I was thinking mainly about the standard AR loop, yes I can see that grammars would make it a bit more complicated especially when considering batching.

Re: Rust’s Standard Library on the GPU

#58

Post by [deleted] »

saagarjha wrote: post #55
fulafel wrote: post #54

Earlier quoted context omitted.

This level corresponds to SMT in CPUs I gather. So you can argue your 192 core EPYC server cpu has 384 "vCPUs" since execution resources per core are overprovisioned and when execution blocks waiting for eg memory another thread can run in its place. As Intel and AMD only do 2-way SMT this doesn't make the numbers go up as much. The single GPU warp is both beefier and wimpier than the SMT thread: they're in-order bar…

Right, but the CPU does not have a matrix multiply unit or high bandwidth memory.

[deleted]

Re: Rust’s Standard Library on the GPU

#59

Post by codedokode »

dancek wrote: post #23
codedokode wrote: post #21

I think it is possible to run CPU code on GPU (including the whole OS), because GPU has registers, memory, arithmetic and branch instructions, and that should be enough. However, it will be able to use only several cores from many thousands because GPU cores are effectively wide SIMD cores, grouped into the clusters, and CPU-style code would use only single SIMD lane. Am I wrong?

This seems correct to me. Of course you'd need to build a CPU emulator to run CPU code. A single GPU core is apparently about 100x slower than a single CPU core. With emulation a 1000x slowdown might be expected. So with a lot of handwaving, expect performance similar to a 4 MHz processor. Obviously code designed for a GPU is much faster. You could probably build a reasonable OS that runs on the GPU.

You don't need an emulator, you can compile into GPU machine code.
Post reply on HN