Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust’s Standard Library on the GPU

vectorware.com

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Rust’s Standard Library on the GPU

#31

Post by storystarling »

shihab wrote: post #19

To the author (or anyone from vectorware team), can you please give me, admittedly a skeptic, a motivating example of a "GPU-native" application? That is, where does it truly make a difference to dispatch non-parallel/syscalls etc from GPU to CPU instead of dispatching parallel part of a code from CPU to GPU? From the "Announcing VectorWare" page: > Even after opting in, the CPU is in control and orchestrates work on…

The killer app here is likely LLM inference loops. Currently you pay a PCIe latency penalty for every single token generated because the CPU has to handle the sampling and control logic. Moving that logic to the GPU and keeping the whole generation loop local avoids that round trip, which turns out to be a major bottleneck for interactive latency.

Re: Rust’s Standard Library on the GPU

#32

Post by tucnak »

storystarling wrote: post #31
shihab wrote: post #19

To the author (or anyone from vectorware team), can you please give me, admittedly a skeptic, a motivating example of a "GPU-native" application? That is, where does it truly make a difference to dispatch non-parallel/syscalls etc from GPU to CPU instead of dispatching parallel part of a code from CPU to GPU? From the "Announcing VectorWare" page: > Even after opting in, the CPU is in control and orchestrates work on…

The killer app here is likely LLM inference loops. Currently you pay a PCIe latency penalty for every single token generated because the CPU has to handle the sampling and control logic. Moving that logic to the GPU and keeping the whole generation loop local avoids that round trip, which turns out to be a major bottleneck for interactive latency.

Turns out how? Where are the numbers?

Re: Rust’s Standard Library on the GPU

#33

Post by storystarling »

tucnak wrote: post #32
storystarling wrote: post #31

Earlier quoted context omitted.

The killer app here is likely LLM inference loops. Currently you pay a PCIe latency penalty for every single token generated because the CPU has to handle the sampling and control logic. Moving that logic to the GPU and keeping the whole generation loop local avoids that round trip, which turns out to be a major bottleneck for interactive latency.

Turns out how? Where are the numbers?

It is less about the raw transfer speed and more about the synchronization and kernel launch overheads. If you profile a standard inference loop with a batch size of 1 you see the GPU spending a lot of time idle waiting for the CPU to dispatch the next command. That is why optimizations like CUDA graphs exist, but moving the control flow entirely to the device is the cleaner solution.

Re: Rust’s Standard Library on the GPU

#34

Post by zozbot234 »

pixelpoet wrote: post #28

GPUs aren't fast because they run standard CPU code with magic pixie dust, they're fast because they're specialised vector processors running specialised vector code. Cuda can also do C++ new, delete and virtual functions and exception handling and all the rest. And if you use that stuff, you're basically making an aeroplane flap its wings, with all the performance implications that come with such an abomination. inb…

Nope, the magic pixie dust language that was supposed to run Python-like code on GPU was Mojo /s

this is really just leveraging Rust's existing, unique fit across HPC/numerics, embedded programming, low-level systems programming and even old retro-computing targets, and trying to expand that fit to the GPU by leveraging broad characteristics that are quite unique to Rust and are absolutely relevant among most/all of those areas.

The real GPU pixie dust is called "lots of slow but efficient compute units", "barrel processing", "VRAM/HBM" and "non-flat address space(s) with explicit local memories". And of course "wide SIMD+SPMD[0]" which is the part you already mentioned and is in fact somewhat harder to target other than in special cases (though neural inference absolutely relies on it!). But never mind that. A lot of existing CPU code that's currently bottlenecked on memory access throughput will absolutely benefit from being seamlessly ran on GPU.

[0] SPMD is the proper established name for what people casually call SIMT

Re: Rust’s Standard Library on the GPU

#35

Post by zozbot234 »

fulafel wrote: post #27
codedokode wrote: post #21

I think it is possible to run CPU code on GPU (including the whole OS), because GPU has registers, memory, arithmetic and branch instructions, and that should be enough. However, it will be able to use only several cores from many thousands because GPU cores are effectively wide SIMD cores, grouped into the clusters, and CPU-style code would use only single SIMD lane. Am I wrong?

GPUs having have thousands of cores is just a silly marketing newspeak. They rebranded SIMD lanes "cores". For eaxmple Nvidia 5000 series GPUs have 50-170 SMs which are the equivalent of cpu cores there. So a more than desktops, less than bigger server CPUs. By this math each avx-512 cpu core has 16-64 "gpu cores".

170 compute units is still a crapload of em for a non-server platform with non-server platform requirements. so the broad "lots of cores" point is still true, just highly overstated as you said. plus those cores are running the equivalent of n-way SMT processing, which gives you an even higher crapload of logical threads. AND these logical threads can also access very wide SIMD when relevant, which even early Intel E-cores couldn't. All of that absolutely matters.

Re: Rust’s Standard Library on the GPU

#36

Post by radarsat1 »

storystarling wrote: post #31
shihab wrote: post #19

To the author (or anyone from vectorware team), can you please give me, admittedly a skeptic, a motivating example of a "GPU-native" application? That is, where does it truly make a difference to dispatch non-parallel/syscalls etc from GPU to CPU instead of dispatching parallel part of a code from CPU to GPU? From the "Announcing VectorWare" page: > Even after opting in, the CPU is in control and orchestrates work on…

The killer app here is likely LLM inference loops. Currently you pay a PCIe latency penalty for every single token generated because the CPU has to handle the sampling and control logic. Moving that logic to the GPU and keeping the whole generation loop local avoids that round trip, which turns out to be a major bottleneck for interactive latency.

I don't know what the pros are doing but I'd be a bit shocked if it isn't already done this way in real production systems. And it doesn't feel like porting the standard library is necessary for this, it's just some logic.

Re: Rust’s Standard Library on the GPU

#37

Post by ukoki »

shihab wrote: post #19

To the author (or anyone from vectorware team), can you please give me, admittedly a skeptic, a motivating example of a "GPU-native" application? That is, where does it truly make a difference to dispatch non-parallel/syscalls etc from GPU to CPU instead of dispatching parallel part of a code from CPU to GPU? From the "Announcing VectorWare" page: > Even after opting in, the CPU is in control and orchestrates work on…

Not OP but I'm currently make a city-builder computer game with a large procedurally-generated world. The terrain height at any point in the world is defined by function that takes a small number of constant parameters, and the horizontal position in the world, to give the height of the terrain at that position.

I need the heights on the GPU so I can modify the terrain meshes to fit the terrain. I need the heights on the CPU so I can know when the player is clicking the terrain and where to place things.

Rather than generating a heightmap on the CPU and passing a large heightmap texture to the GPU I have implemented the identical height generating functions in rust (CPU) and webgl (GPU). As you might imagine, its very easy for these to diverge and so I have to maintain a large set of tests that verify that generated heights are identical between implementations.

Being able to write this implementation once and run it on the CPU and GPU would give me much better guarantees that the results will be the same. (although necause of architecture differences and floating point handling they the results will never be perfect, but I just need them to be within an acceptable tolerance)

Re: Rust’s Standard Library on the GPU

#38

Post by storystarling »

radarsat1 wrote: post #36
storystarling wrote: post #31

Earlier quoted context omitted.

The killer app here is likely LLM inference loops. Currently you pay a PCIe latency penalty for every single token generated because the CPU has to handle the sampling and control logic. Moving that logic to the GPU and keeping the whole generation loop local avoids that round trip, which turns out to be a major bottleneck for interactive latency.

I don't know what the pros are doing but I'd be a bit shocked if it isn't already done this way in real production systems. And it doesn't feel like porting the standard library is necessary for this, it's just some logic.

Raw CUDA works for the heavy lifting but I suspect it gets messy once you implement things like grammar constraints or beam search. You end up with complex state machines during inference and having standard library abstractions seems pretty important to keep that logic from becoming unmaintainable.

Re: Rust’s Standard Library on the GPU

#39

Post by xmcqdpt2 »

ukoki wrote: post #37
shihab wrote: post #19

To the author (or anyone from vectorware team), can you please give me, admittedly a skeptic, a motivating example of a "GPU-native" application? That is, where does it truly make a difference to dispatch non-parallel/syscalls etc from GPU to CPU instead of dispatching parallel part of a code from CPU to GPU? From the "Announcing VectorWare" page: > Even after opting in, the CPU is in control and orchestrates work on…

Not OP but I'm currently make a city-builder computer game with a large procedurally-generated world. The terrain height at any point in the world is defined by function that takes a small number of constant parameters, and the horizontal position in the world, to give the height of the terrain at that position. I need the heights on the GPU so I can modify the terrain meshes to fit the terrain. I need the heights on…

That's a good application but likely not one requiring a full standard library on the GPU? Procedurally generated data on GPU isn't uncommon AFAIK. It wasn't when I was dabbling in GPGPU stuff ~10 years ago.

If you wrote in open cl, or via intel libraries, or via torch or arrayfire or whatever, you could dispatch it to both CPU and GPU at will.

Re: Rust’s Standard Library on the GPU

#40

Post by xmcqdpt2 »

elromulous wrote: post #25
bigyabai wrote: post #24

Earlier quoted context omitted.

Given enough time, we'll all loop back around to the Xeon Phi: https://en.wikipedia.org/wiki/Xeon_Phi

It was ahead of its time! When I was in grad school I tried getting my hands on a phi, it seemed impossible.

Xeon Phi was so cool. I wanted to use the ones we had so much... but couldn't find any applications that would benefit enough to make it worth the effort. I guess that's why it died lol.
Post reply on HN