Warp specialization is an abomination that should be killed and I'm glad this could be an alternative.
I hope they can minimize the bookkeeping costs because I don't see it gain traction in AI if it hurts big kernels performance.
Async/Await on the GPU
31–40 of 61 posts
Re: Async/Await on the GPU
#32Earlier quoted context omitted.
I think you're conflating GPU 'threads' and 'warps'. GPU 'threads' are SIMD lanes that are all running with the exact same instructions and control flow (only with different filtering/predication), whereas GPU warps are hardware-level threads that run on a single compute unit. There's no issue with adding extra "don't run code" when using warps, unlike GPU threads.
My understanding of warp ( https://docs.nvidia.com/cuda/cuda-programming-guide/01-intro... ) is that you are essentially paying the cost of taking both the branches. I understand with newer GPUs, you have clever partitioning / pipelining in such a way block A takes branch A vs block B that takes branch B with sync/barrier essentially relying on some smart 'oracle' to schedule these in a way that still fits in the SIM…
That applies inside a single warp, notice the wording:
> In SIMT, all threads in the warp are executing the same kernel code, but each thread may follow different branches through the code. That is, though all threads of the program execute the same code, threads do not need to follow the same execution path.
This doesn't say anything about dependencies of multiple warps.
Re: Async/Await on the GPU
#33[flagged]
Re: Async/Await on the GPU
#34Earlier quoted context omitted.
My understanding of warp ( https://docs.nvidia.com/cuda/cuda-programming-guide/01-intro... ) is that you are essentially paying the cost of taking both the branches. I understand with newer GPUs, you have clever partitioning / pipelining in such a way block A takes branch A vs block B that takes branch B with sync/barrier essentially relying on some smart 'oracle' to schedule these in a way that still fits in the SIM…
That applies inside a single warp, notice the wording: > In SIMT, all threads in the warp are executing the same kernel code, but each thread may follow different branches through the code. That is, though all threads of the program execute the same code, threads do not need to follow the same execution path. This doesn't say anything about dependencies of multiple warps.
It's definitely possible, I am not arguing against that.
I am just saying it's not as flexible/cost-free as you would on a 'normal' von Neumann-style CPU.
I would love to see Rust-based code that obviates the need to write CUDA kernels (including compiling to different architectures). It feels icky to use/introduce things like async/await in the context of a GPU programming model which is very different from a traditional Rust programming model.
You still have to worry about different architectures and the streaming nature at the end of the day.
I am very interested in this topic, so I am curious to learn how the latest GPUs help manage this divergence problem.
Re: Async/Await on the GPU
#35Et tu, GPU? I am, bluntly, sick of Async taking over rust ecosystems. Embedded and web/HTTP have already fallen. I'm optimistic this won't take hold in GPU; well see. Async splits the ecosystem. I see it as the biggest threat to Rust staying a useful tool. I use rust on the GPU for the following: 3d graphics via WGPU, cuFFT via FFI, custom kernels via Cudarc, and ML via Burn and Candle. Thankfully these are all Async…
I don't see the utility of async on the GPU.
> Async splits the ecosystem. I see it as the biggest threat to Rust staying a useful tool.
Someone somewhere convinced you there is a async coloring problem. That person was wrong, async is an inherent property of some operations. Adding it as a type level construct gives visibility to those inherent behaviors, and with that more freedom in how you compose them.
Re: Async/Await on the GPU
#36Earlier quoted context omitted.
Currently NVIDIA-only, we're cooking up some Vulkan stuff in rust-gpu though.
I don't have anything to offer but my encouragement, but there are _dozens_ of ROCm enjoyers out there. In years prior I wouldn't have even bothered, but it's 2026 and AMD's drivers actually come with a recent version of torch that 'just works' on windows. Anything is possible :)
Thank you! We're small so have to focus. If anyone from AMD wants to reach out, happy to chat.
Re: Async/Await on the GPU
#37[flagged]
[dead]
Re: Async/Await on the GPU
#38Love this.
You mention futures are cooperative and GPUs lack interrupts, but GPU warps already have a hardware scheduler that preempts at the instruction level. ARe you intentionally working above that layer, or do you see a path to a fture executor that hooks into warp scheduling more directly to get preemptive-like behavior?
Re: Async/Await on the GPU
#39[flagged]
I had a longer, snarkier response to this the (as I'm writing) top comment on this thread. I spent longer than I'd like to have trying to decode what insight you were sharing here (what exactly is inverted in the GPU/CPU summaries you give?) until I browsed your comment history and saw what looks like a bunch of AI-generated comments (sometimes less than a minute apart from each other) and realized I was trying to decode slop.
This one's especially clear because you reference "the cases shayonj mentioned", but shayonj's comment[1] doesn't mention any use cases, but it does make a comparison to "NVIDIA's stdexec", which seems like might have gotten mixed into what your model was trying to say in the preceding paragraph?
This is really annoying. Please stop.
Re: Async/Await on the GPU
#40Et tu, GPU? I am, bluntly, sick of Async taking over rust ecosystems. Embedded and web/HTTP have already fallen. I'm optimistic this won't take hold in GPU; well see. Async splits the ecosystem. I see it as the biggest threat to Rust staying a useful tool. I use rust on the GPU for the following: 3d graphics via WGPU, cuFFT via FFI, custom kernels via Cudarc, and ML via Burn and Candle. Thankfully these are all Async…
I don't see the utility of async on the GPU. > Async splits the ecosystem. I see it as the biggest threat to Rust staying a useful tool. Someone somewhere convinced you there is a async coloring problem. That person was wrong, async is an inherent property of some operations. Adding it as a type level construct gives visibility to those inherent behaviors, and with that more freedom in how you compose them.
itd be interesting to see a setup where there's only async and you have to specify when you actually want to block on a result.
flip the colouring problem on its head