Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust’s Standard Library on the GPU

vectorware.com

Post reply on HN
Print view View original

41–50 of 59 posts

Re: Rust’s Standard Library on the GPU

#41

Post by tucnak »

storystarling wrote: post #33
tucnak wrote: post #32

Earlier quoted context omitted.

Turns out how? Where are the numbers?

It is less about the raw transfer speed and more about the synchronization and kernel launch overheads. If you profile a standard inference loop with a batch size of 1 you see the GPU spending a lot of time idle waiting for the CPU to dispatch the next command. That is why optimizations like CUDA graphs exist, but moving the control flow entirely to the device is the cleaner solution.

I'm not convinced. (A bit of advice: if you wish to make a statement about performance, always start by measuring things. Then when somebody asks you for proof/data, you would already have it.) If what you're saying were true, it would be a big deal, except unfortunately it isn't.

Dispatch has overheads, but it's largely insignificant. Where it otherwise would be significant:

1. Fused kernels exist

2. CUDA graphs (and other forms of work-submission pipelining) exist

Re: Rust’s Standard Library on the GPU

#42

Post by the__alchemist »

spease wrote: post #30
rl3 wrote: post #22

Earlier quoted context omitted.

> What would be cool is the dream that's been building for decades about parallel computing abstractions where you write what looks like normal single-threaded CPU code, but it automagically works on SIMD instructions or GPU. I've had that same dream at various points over the years, and prior to AI my conclusion was that it was untenable barring a very large, world-class engineering team with truckloads of money. I'…

There was a library for Rust called “faster” which worked similarly to Rayon, but for SIMD. The simpleminded way to do what you’re saying would be to have the compiler create separate PTX and native versions of a Rayon structure, and then choose which to invoke at runtime.

Why past tense? I would use that if it truly acted like Rayon! I.e minimal friction.

Re: Rust’s Standard Library on the GPU

#43

Post by saagarjha »

tucnak wrote: post #41
storystarling wrote: post #33

Earlier quoted context omitted.

It is less about the raw transfer speed and more about the synchronization and kernel launch overheads. If you profile a standard inference loop with a batch size of 1 you see the GPU spending a lot of time idle waiting for the CPU to dispatch the next command. That is why optimizations like CUDA graphs exist, but moving the control flow entirely to the device is the cleaner solution.

I'm not convinced. (A bit of advice: if you wish to make a statement about performance, always start by measuring things. Then when somebody asks you for proof/data, you would already have it.) If what you're saying were true, it would be a big deal, except unfortunately it isn't. Dispatch has overheads, but it's largely insignificant. Where it otherwise would be significant: 1. Fused kernels exist 2. CUDA graphs (an…

CUDA graphs are pretty slow at synchronizing things.

Re: Rust’s Standard Library on the GPU

#44

Post by cong-or »

What’s the latency on a hostcall? A PCIe round-trip for something like File::open is fine—that’s slow I/O anyway. But if a println! from GPU code blocks on a host round-trip every time, that completely changes how you’d use it for debugging.

Is there device-side buffering, or does each write actually wait for the host?

Re: Rust’s Standard Library on the GPU

#45

Post by saagarjha »

fulafel wrote: post #27
codedokode wrote: post #21

I think it is possible to run CPU code on GPU (including the whole OS), because GPU has registers, memory, arithmetic and branch instructions, and that should be enough. However, it will be able to use only several cores from many thousands because GPU cores are effectively wide SIMD cores, grouped into the clusters, and CPU-style code would use only single SIMD lane. Am I wrong?

GPUs having have thousands of cores is just a silly marketing newspeak. They rebranded SIMD lanes "cores". For eaxmple Nvidia 5000 series GPUs have 50-170 SMs which are the equivalent of cpu cores there. So a more than desktops, less than bigger server CPUs. By this math each avx-512 cpu core has 16-64 "gpu cores".

Each SM can typically schedule 4 warps so it’s more like 400 “cores” each with 1024-bit SIMD instructions. If you look at it this way, they clearly outclass CPU architectures.

Re: Rust’s Standard Library on the GPU

#46

Post by saagarjha »

LegNeato wrote: post #16
koyote wrote: post #7

Are there any details around how the round-trip and exchange of data (CPU GPU) is implemented in order to not be a big (partially-hidden) performance hit? e.g. this code seems like it would entirely run on the CPU? print!("Enter your name: "); let _ = std::io::stdout().flush(); let mut name = String::new(); std::io::stdin().read_line(&mut name).unwrap(); But what if we concatenated a number to the string that was cal…

We use the cuda device allocator for allocations on the GPU via Rust's default allocator.

Have you considered “allocating” out of shared memory instead?

Re: Rust’s Standard Library on the GPU

#47

Post by e12e »

There's indeed an actual toggle at the top to switch on pedantic mode, FYI.

But still a bit odd that the article doesn't show assembly/cuda/opencl output of the compiler - nor show an example of anything parallel - like maybe a vector search, Mandelbrot calculation or something like that?

Something like this 2019 article on cuda for Julia:

https://nextjournal.com/sdanisch/julia-gpu-programming

Re: Rust’s Standard Library on the GPU

#48

Post by solaarphunk »

I've been building something similar (GPU-native OS research project) and wanted to share a mental model shift that unlocked things for me.

The question "why run CPU code on GPU when GPU cores are slower?" assumes you're running ONE program. But GPUs execute in SIMD wavefronts of 32 threads - and here's the trick: each of those 32 lanes can run a DIFFERENT process. Same instruction, different data. Calculator on lane 0, text editor on lane 1, file indexer on lane 2. No divergence, legal SIMD, full utilization. Suddenly you're not running "slow CPU code on GPU" - you're running 32 independent programs in parallel on hardware designed for exactly this pattern.

The win isn't throughput for compute-heavy code. It's eliminating CPU roundtrips for interactive stuff. Every kernel launch, every synchronization, every "GPU done, back to CPU, dispatch next thing" adds latency. A persistent kernel that polls for input, updates state, and renders - all without returning to CPU - changes the responsiveness equation entirely.

  A few things to try at home if you're curious:
1. Write a Metal/CUDA kernel with while(true) and an atomic shutdown flag. See how long it runs. (Spoiler: indefinitely, if you do it right)

2. Put 32 different "process states" in a buffer and have each SIMD lane execute instructions for its own process. Watch all 32 make progress simultaneously.

3. Measure the latency from "input event" to "pixel on screen" with CPU orchestration vs GPU polling an input queue directly. The difference surprised me.

The persistent kernel thing has a nasty gotcha though - ALL 32 threads must participate in the while loop. If you do if (tid != 0) return; then while(true), it'll work for a few million iterations then hard-lock. Ask me how I know.

Re: Rust’s Standard Library on the GPU

#49

Post by moron4hire »

ukoki wrote: post #37
shihab wrote: post #19

To the author (or anyone from vectorware team), can you please give me, admittedly a skeptic, a motivating example of a "GPU-native" application? That is, where does it truly make a difference to dispatch non-parallel/syscalls etc from GPU to CPU instead of dispatching parallel part of a code from CPU to GPU? From the "Announcing VectorWare" page: > Even after opting in, the CPU is in control and orchestrates work on…

Not OP but I'm currently make a city-builder computer game with a large procedurally-generated world. The terrain height at any point in the world is defined by function that takes a small number of constant parameters, and the horizontal position in the world, to give the height of the terrain at that position. I need the heights on the GPU so I can modify the terrain meshes to fit the terrain. I need the heights on…

There are GPU-based picking algorithms. You really should not have to maintain parallel data generation systems on both the GPU and CPU just to support picking. Maybe you have a different issue that would require it, but picking alone shouldn't be it.

Re: Rust’s Standard Library on the GPU

#50

Post by nicman23 »

shihab wrote: post #19

To the author (or anyone from vectorware team), can you please give me, admittedly a skeptic, a motivating example of a "GPU-native" application? That is, where does it truly make a difference to dispatch non-parallel/syscalls etc from GPU to CPU instead of dispatching parallel part of a code from CPU to GPU? From the "Announcing VectorWare" page: > Even after opting in, the CPU is in control and orchestrates work on…

in large sim systems p2p comms and not having to involve the cpu in any way - because the cpu is doing work as well and you do not want to have the cpu to sync every result if it is partial

one example is pme decomposition in gromacs.

Post reply on HN