Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async/Await on the GPU

vectorware.com

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Async/Await on the GPU

#51

Post by winwang »

lmeyerov wrote: post #47
the__alchemist wrote: post #13

Et tu, GPU? I am, bluntly, sick of Async taking over rust ecosystems. Embedded and web/HTTP have already fallen. I'm optimistic this won't take hold in GPU; well see. Async splits the ecosystem. I see it as the biggest threat to Rust staying a useful tool. I use rust on the GPU for the following: 3d graphics via WGPU, cuFFT via FFI, custom kernels via Cudarc, and ML via Burn and Candle. Thankfully these are all Async…

For graphistry at least, I care less about the surface syntax (async/await) and more about getting gpu-side work stealing, dynamic task scheduling, etc. Our code is written at such a much higher level that these are primitives needed by our runtime, not most of our developers & users. Imagine SQL, cypher, etc on GPUs and our implementations of those being able to use the gpu-side libraries when coordinating 1M+ threa…

Is it the performance benefits? Or being able to write concurrent code much more expressively? Though I suppose the latter might imply the former.

Re: Async/Await on the GPU

#52

Post by lmeyerov »

winwang wrote: post #51
lmeyerov wrote: post #47

Earlier quoted context omitted.

For graphistry at least, I care less about the surface syntax (async/await) and more about getting gpu-side work stealing, dynamic task scheduling, etc. Our code is written at such a much higher level that these are primitives needed by our runtime, not most of our developers & users. Imagine SQL, cypher, etc on GPUs and our implementations of those being able to use the gpu-side libraries when coordinating 1M+ threa…

Is it the performance benefits? Or being able to write concurrent code much more expressively? Though I suppose the latter might imply the former.

Performance.

Our code looks like pure pandas (fancier SQL) wrapped as HTTP service (arrow instead of json), so the expressivity is more of a step backwards. We already did the work of turning awkward irregular code into relational pipelines that GPUs love.

Our problems are:

- Multi-tenancy. Our users get to time share GPUs, so when getting many GPU tasks big & small, we want them co-scheduled across the many GPUs & their many cores. GPUs are already more cost effective per Watt than CPUs, but we think we can 2x+ here, which is significant.

- Constant overheads. One job can be deep, with many operations, so round-tripping each step of the control plane, think each SQL subexpression, CPUGPU is silly and adds up. Small jobs are dominated by embarrassing overheads that are precluding certain use cases. We are thinking of doing CPU hot paths to avoid this, but rather just fix the GPU path.

Re: Async/Await on the GPU

#53

Post by [deleted] »

8note wrote: post #40
notnullorvoid wrote: post #35

Earlier quoted context omitted.

I don't see the utility of async on the GPU. > Async splits the ecosystem. I see it as the biggest threat to Rust staying a useful tool. Someone somewhere convinced you there is a async coloring problem. That person was wrong, async is an inherent property of some operations. Adding it as a type level construct gives visibility to those inherent behaviors, and with that more freedom in how you compose them.

itd be interesting to see a setup where there's only async and you have to specify when you actually want to block on a result. flip the colouring problem on its head

[deleted]

Re: Async/Await on the GPU

#54

Post by Panzerschrek »

AS I know GPUs execute code pretty fast as long as all threads in a warp go the same execution path. Branching causes performance degradation. But executing exactly the same code for multiple coroutines seems for me to be practically impossible. So, can good performance be reached with such approach at all?

Re: Async/Await on the GPU

#55

Post by ablob »

Panzerschrek wrote: post #54

AS I know GPUs execute code pretty fast as long as all threads in a warp go the same execution path. Branching causes performance degradation. But executing exactly the same code for multiple coroutines seems for me to be practically impossible. So, can good performance be reached with such approach at all?

The short answer is yes. The post literally has an example of co-routines (think C-style: possible, but ugly). The difference here is how easy it is to write. I'd wager the question is not if it can be achieved, but for which use cases it can be ergonomic.

Re: Async/Await on the GPU

#57

Post by bob1029 »

LegNeato wrote: post #26
GZGavinZhao wrote: post #19

One concern I have is that this async/await approach is not "AOT"-enough like the Triton approach, in the sense that you know how to most efficiently schedule the computations on which warps since you know exactly what operations you'll be performing at compile time. Here with the async/await approach, it seems like there needs to be manual book-keeping at runtime to know what has finished, what has not, and _then_ c…

Doing things at compile time / AOT is almost always better for perf. We believe async/await and futures enables more complex programs and doing things you couldn't easily do on the GPU before. Less about performance and more about capability (though we believe async/await perf will be better in some cases, time will tell).

> Doing things at compile time / AOT is almost always better for perf

https://devblogs.microsoft.com/dotnet/bing-on-dotnet-8-the-i...

Re: Async/Await on the GPU

#58

Post by zozbot234 »

Panzerschrek wrote: post #54

AS I know GPUs execute code pretty fast as long as all threads in a warp go the same execution path. Branching causes performance degradation. But executing exactly the same code for multiple coroutines seems for me to be practically impossible. So, can good performance be reached with such approach at all?

GPU 'threads' are SIMD lanes and this post does not discuss those. It's about running multiple GPU 'warps', i.e. hardware threads with separate control flow and code execution.

(Beyond that, "executing the same code" on multiple instances of a single coroutine ought to be sometimes possible on an opportunistic basis.)

Re: Async/Await on the GPU

#60

Post by NewsaHackO »

magicalist wrote: post #43
NewsaHackO wrote: post #42

Earlier quoted context omitted.

I see this accusation a lot, and admittedly, I defended someone who later on was shown to use AI to generate comments, but I am still missing a motivation for this. Is your argument that he is using AI to copyedit his posts, or that he is asking AI to write a response to a random thread that looks insightful? Because I cannot fathom why someone would ever do that.

I have no idea what their motivation is and no idea if they're using an LLM to tune their prose or write comments whole cloth (considering the four recent comments, each two paragraphs, within 2.5 minutes, though, I'm guessing fully generated). I was just annoyed enough by spending a couple of minutes trying to decode what had the semblance of something interesting that I felt compelled to write my response :) There…

I agree. I am just wondering aloud what they could possibly gain from this. Just really bizarre behavior.
Post reply on HN