Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia adds native Python support to CUDA

thenewstack.io

Post reply on HN
Print view View original

161–170 of 196 posts

Re: Nvidia adds native Python support to CUDA

#161

Post by ImprobableTruth »

alanfranz wrote: post #160
ImprobableTruth wrote: post #103

Earlier quoted context omitted.

The reason is that the usage is completely different from coroutine based async. With GPUs you want to queue _as many async operations as possible_ and only then synchronize. That is, you would have a program like this (pseudocode): b = foo(a) c = bar(b) d = baz(c) synchronize() With coroutines/async await, something like this b = await foo(a) c = await bar(b) d = await baz(c) would synchronize after every step, bein…

Well you can and should create multiple coroutine/tasks and then gather them. If you replace cuda with network calls, it’s exactly the same problem. Nothing to do with asyncio.

No, that's a different scenario. In the one I gave there's explicitly a dependency between requests. If you use gather, the network requests would be executed in parallel. If you have dependencies they're sequential by nature because later ones depend on values of former ones.

The 'trick' for CUDA is that you declare all this using buffers as inputs/outputs rather than values and that there's automatic ordering enforcement through CUDA's stream mechanism. Marrying that with the coroutine mechanism just doesn't really make sense.

Re: Nvidia adds native Python support to CUDA

#162

Post by saagarjha »

almostgotcaught wrote: post #144
dragonwriter wrote: post #113

Earlier quoted context omitted.

> i'm just able to actually read and comprehend what i'm reading rather than perform hype: The evidence of that is lacking. > so the article is about cuda-core, not whatever you think it's about cuda.core (a relatively new, rapidly developing, library whose entire API is experimental) is one of several things (NVMath is another) mentioned in the article, but the newer and as yet unreleased piece mentioned in the arti…

> No, as is is fairly explicit in the next line after the one you quote, it is about the Nvidia CUDA Python toolchain using in-process compilation rather than relying on shelling out to out-of-process command-line compilers for CUDA code. my guy what i am able to read, which you are not, is the source and release notes. i do not need to read tweets and press releases because i know what these things actually are. her…

cuTile is basically Nvidia’s Triton (no, not that Triton, OpenAI’s Triton) competitor. It takes your Python code and generates kernels at runtime. CUTLASS has a new Python interface that does the same thing.

Re: Nvidia adds native Python support to CUDA

#163

Post by saagarjha »

nickysielicki wrote: post #102
wiredfool wrote: post #15

Earlier quoted context omitted.

Curious what the timing would be if it included the memory transfer time, e.g. matricies = [np.random(...) for _ in range] time_start = time.time() cp_matricies = [cp.array(m) for m in matrices] add_(cp_matricies) sync time_end = time.time()

I don’t mean to call you or your pseudocode out specifically, but I see this sort of thing all the time, and I just want to put it out there: PSA: if you ever see code trying to measure timing and it’s not using the CUDA event APIs, it’s fundamentally wrong and is lying to you. The simplest way to be sure you’re not measuring noise is to just ban the usage of any other timing source. Definitely don’t add unnecessary…

I mean you can definitely use it in a pinch if you know what you’re doing. But yes the event APIs are better.

Re: Nvidia adds native Python support to CUDA

#164

Post by saagarjha »

ashvardanian wrote: post #90

CuTile, in many ways, feels like a successor to OpenAI's Triton... And not only are we getting tile/block-level primitives and TileIR, but also a proper SIMT programming model in CuPy, which I don't think enough people noticed even at this year's GTC. Very cool stuff! That said, there were almost no announcements or talks related to CPUs, despite the Grace CPUs being announced quite some time ago. It doesn't feel lik…

I mean, it doesn’t really make sense to unify them. CPUs and GPUs have very different performance characteristics and you design for them differently depending on what they let you do. There’s obviously a common ground where you can design mostly good interfaces to do things ok (I’ll argue PyTorch is that) but it’s not really reasonable to write an algorithm that is hobbled on CPUs for no reason because it assumes that synchronizing between execution contexts is super expensive.

Re: Nvidia adds native Python support to CUDA

#165

Post by saagarjha »

ferguess_k wrote: post #21
gymbeaux wrote: post #12

This is huge. Anyone who was considering AMD + ROCm as an alternative to NVIDIA in the AI space isn’t anymore. I’m one of those people who can’t (won’t) learn C++ to the extent required to effectively write code for GPU execution…. But to have a direct pipeline to the GPU via Python. Wow. The efficiency implications are huge, not just for Python libraries like PyTorch, but also anything we write that runs on an NVIDI…

Just curious why can't AMD do the same thing?

They did; they are active contributors to OpenAI’s Triton compiler which has a very similar execution model.

Re: Nvidia adds native Python support to CUDA

#166

Post by saagarjha »

taminka wrote: post #31
the__alchemist wrote: post #8

Rust support next? RN I am manually [de]serializing my data structures as byte arrays to/from the kernels. It would be nice to have truly shared data structures like CUDA gives you in C++!

even putting aside how rust ownership semantics map poorly onto gpu programming, ml researchers will never learn rust, this will never ever happen...

I don’t think this is true. It seems to me more that nobody has put in a serious effort to make a nice interface build using Rust.

Re: Nvidia adds native Python support to CUDA

#167

Post by saagarjha »

no_wizard wrote: post #152

What makes Python such a target for these kind of things? I've noticed alot of projects add Python support like this. Does the Python codebase allow for it to compile down to different targets easier than others?

There’s a lot of existing Python code in this space and many ML researchers are comfortable in Python.

Re: Nvidia adds native Python support to CUDA

#168

Post by Barrin92 »

airstrike wrote: post #76
whycome wrote: post #52

Earlier quoted context omitted.

Would you say Python is a good language to learn as a beginner?

As someone who spent nearly a decade with Python, I'd say 90% of people will answer "yes", so I'd like to offer a different perspective. IMHO if you want to pick it up for a couple toy projects just to get a feel of what coding is like, then by all means try it out. But eventually you'll benefit tremendously from exploring other languages. Python will teach you a lot of bad habits. You will feel like you know what yo…

>Knowing what I know now, I wish Rust existed when I started out so that it could have been my first language

No offense but I don't think this makes any sense (or only if you take the first part of that sentence literally). It's like jumping into Calculus 3 to introduce a kid to maths. From a teaching standpoint, if you're a beginner, you can't even understand what problem Rust solves. Someone who doesn't know what manual memory management, a heap and a stack is should not be handed a borrow checker.

You can either start from the top, the old school way, teach a lisp or python as a more modern alternative and teach people symbolic computing, or you can start with C and teach people from the bottom up how computers work, but frankly throwing you into a language that basically exists to solve problems professional C++ developers have in large projects is kind of wild

Re: Nvidia adds native Python support to CUDA

#169

Post by jms55 »

nickysielicki wrote: post #102
wiredfool wrote: post #15

Earlier quoted context omitted.

Curious what the timing would be if it included the memory transfer time, e.g. matricies = [np.random(...) for _ in range] time_start = time.time() cp_matricies = [cp.array(m) for m in matrices] add_(cp_matricies) sync time_end = time.time()

I don’t mean to call you or your pseudocode out specifically, but I see this sort of thing all the time, and I just want to put it out there: PSA: if you ever see code trying to measure timing and it’s not using the CUDA event APIs, it’s fundamentally wrong and is lying to you. The simplest way to be sure you’re not measuring noise is to just ban the usage of any other timing source. Definitely don’t add unnecessary…

Never used CUDA, but I'm guessing these map to the same underlying stuff as timestamp queries in graphics APIs, yes?

Re: Nvidia adds native Python support to CUDA

#170

Post by jms55 »

ashvardanian wrote: post #90

CuTile, in many ways, feels like a successor to OpenAI's Triton... And not only are we getting tile/block-level primitives and TileIR, but also a proper SIMT programming model in CuPy, which I don't think enough people noticed even at this year's GTC. Very cool stuff! That said, there were almost no announcements or talks related to CPUs, despite the Grace CPUs being announced quite some time ago. It doesn't feel lik…

> And not only are we getting tile/block-level primitives and TileIR

As someone working on graphics programming, it always frustrates me to see so much investment in GPU APIs _for AI_, but almost nothing for GPU APIs for rendering.

Block level primitives would be great for graphics! PyTorch-like JIT kernels programmed from the CPU would be great for graphics! ...But there's no money to be made, so no one works on it.

And for some reason, GPU APIs for AI are treated like an entirely separate thing, rather than having one API used for AI and rendering.

Post reply on HN