Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async/Await on the GPU

vectorware.com

Post reply on HN
Print view View original

11–20 of 61 posts

Re: Async/Await on the GPU

#11

Post by textlapse »

LegNeato wrote: post #7
textlapse wrote: post #5

What's the performance like? What would the benefits be of converting a streaming multiprocessor programming model to this?

We aren't focused on performance yet (it is often workload and executor dependent, and as the post says we currently do some inefficient polling) but Rust futures compile down to state machines so they are a zero-cost abstraction. The anticipated benefits are similar to the benefits of async/await on CPU: better ergonomics for the developer writing concurrent code, better utilization of shared/limited resources, fewe…

warp is expensive - essentially it's running a 'don't run code' to maintain SIMT.

GPUs are still not practically-Turing-complete in the sense that there are strict restrictions on loops/goto/IO/waiting (there are a bunch of band-aids to make it pretend it's not a functional programming model).

So I am not sure retrofitting a Ferrari to cosplay an Amazon delivery van is useful other than for tech showcase?

Good tech showcase though :)

Re: Async/Await on the GPU

#12

Post by firefly2000 »

LegNeato wrote: post #8
firefly2000 wrote: post #6

Is this Nvidia-only or does it work on other architectures?

Currently NVIDIA-only, we're cooking up some Vulkan stuff in rust-gpu though.

Does the lack of forward progress guarantees (ITS) on other architectures pose challenges for async/await?

Re: Async/Await on the GPU

#13

Post by the__alchemist »

Et tu, GPU?

I am, bluntly, sick of Async taking over rust ecosystems. Embedded and web/HTTP have already fallen. I'm optimistic this won't take hold in GPU; well see. Async splits the ecosystem. I see it as the biggest threat to Rust staying a useful tool.

I use rust on the GPU for the following: 3d graphics via WGPU, cuFFT via FFI, custom kernels via Cudarc, and ML via Burn and Candle. Thankfully these are all Async-free.

Re: Async/Await on the GPU

#14

Post by jmalicki »

zozbot234 wrote: post #3

I'm not quite seeing the real benefit of this. Is the idea that warps will now be able to do work-stealing and continuation-stealing when running heterogenous parallel workloads? But that requires keeping the async function's state in GPU-wide shared memory, which is generally a scarce resource.

A ton of GPU workloads require leaving large amounts of RAM resident on the GPU and running computation with some new data from the CPU.

Re: Async/Await on the GPU

#15

Post by xiphias2 »

Really cool experiment (the whole company).

Training pipelines are full of data preparation that are first written on CPU then moving to GPU and always thinking of what to keep on CPU and what to put on GPU, when is it worth to create a tensor, or should it be tiling instead. I guess your company is betting on solving problems like this (and async-await is needed for serving inference requests directly on the GPU for example).

My question is a little bit different: how do you want to handle the SIMD question: should a rust function be running on the warp as a machine with 32 long arrays as data types, or always ,,hope'' for autovectorization to work (especially with Rust's iter library helpers).

Re: Async/Await on the GPU

#16

Post by pjmlp »

zozbot234 wrote: post #3

I'm not quite seeing the real benefit of this. Is the idea that warps will now be able to do work-stealing and continuation-stealing when running heterogenous parallel workloads? But that requires keeping the async function's state in GPU-wide shared memory, which is generally a scarce resource.

This is already happening in C++, NVidia is the one pushing the senders/receivers proposal, which is one of the possible co-routine runtimes to be added into C++ standard library.

Re: Async/Await on the GPU

#17

Post by zozbot234 »

textlapse wrote: post #11
LegNeato wrote: post #7

Earlier quoted context omitted.

We aren't focused on performance yet (it is often workload and executor dependent, and as the post says we currently do some inefficient polling) but Rust futures compile down to state machines so they are a zero-cost abstraction. The anticipated benefits are similar to the benefits of async/await on CPU: better ergonomics for the developer writing concurrent code, better utilization of shared/limited resources, fewe…

warp is expensive - essentially it's running a 'don't run code' to maintain SIMT. GPUs are still not practically-Turing-complete in the sense that there are strict restrictions on loops/goto/IO/waiting (there are a bunch of band-aids to make it pretend it's not a functional programming model). So I am not sure retrofitting a Ferrari to cosplay an Amazon delivery van is useful other than for tech showcase? Good tech s…

I think you're conflating GPU 'threads' and 'warps'. GPU 'threads' are SIMD lanes that are all running with the exact same instructions and control flow (only with different filtering/predication), whereas GPU warps are hardware-level threads that run on a single compute unit. There's no issue with adding extra "don't run code" when using warps, unlike GPU threads.

Re: Async/Await on the GPU

#18

Post by Cieric »

xiphias2 wrote: post #15

Really cool experiment (the whole company). Training pipelines are full of data preparation that are first written on CPU then moving to GPU and always thinking of what to keep on CPU and what to put on GPU, when is it worth to create a tensor, or should it be tiling instead. I guess your company is betting on solving problems like this (and async-await is needed for serving inference requests directly on the GPU for…

I'm not even sure a 32 wide array would be good either since on AMD warps are 64 wide. I wouldn't go fully towards auto vectorization with though.

Re: Async/Await on the GPU

#19

Post by GZGavinZhao »

One concern I have is that this async/await approach is not "AOT"-enough like the Triton approach, in the sense that you know how to most efficiently schedule the computations on which warps since you know exactly what operations you'll be performing at compile time.

Here with the async/await approach, it seems like there needs to be manual book-keeping at runtime to know what has finished, what has not, and _then_ consider which warp should we put this new computation in. Do you anticipate that there will be measurable performance difference?

Re: Async/Await on the GPU

#20

Post by zozbot234 »

Cieric wrote: post #18
xiphias2 wrote: post #15

Really cool experiment (the whole company). Training pipelines are full of data preparation that are first written on CPU then moving to GPU and always thinking of what to keep on CPU and what to put on GPU, when is it worth to create a tensor, or should it be tiling instead. I guess your company is betting on solving problems like this (and async-await is needed for serving inference requests directly on the GPU for…

I'm not even sure a 32 wide array would be good either since on AMD warps are 64 wide. I wouldn't go fully towards auto vectorization with though.

Warp SIMD-width should be a build-time constant. You'd be using a variable-length vector-like interface that gets compiled down to a specified length as part of building the code.
Post reply on HN