Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Triton: Open-Source GPU Programming for Neural Networks

openai.com

Post reply on HN
Print view View original

71–80 of 116 posts

Re: Triton: Open-Source GPU Programming for Neural Networks

#71

Post by N1H1L »

dragontamer wrote: post #11
croes wrote: post #4

Too bad it's CUDA Sooner or later this will become a problem because you are depending on the benevolence of a single manufacturer.

AMD's ROCm 4.0 now supports cooperative groups, which is probably one of the last major holdouts for CUDA compatibility. There's still the 64-wavefront (for AMD CDNA cards) instead of 32-wavefronts (for CUDA). But AMD even has 4x4 half-float matrix multiplication instructions in ROCm (for MI100, the only card that supports the matrix-multiplication / tensor instructions) --------- I think CUDA vs OpenCL is over. ROCm…

Two reasons also. Some of the fastest research supercomputers in the US have been nVIDIA based. Sitting at a national lab, it's laughable how much more support there is from nVIDIA. Getting a DGX box is easy - there is nothing comparable from AMD's side.

And the less said about Intel it's better. Smart, motivated people I know have left Argonne recently because of the Aurora shitshow.

Re: Triton: Open-Source GPU Programming for Neural Networks

#72

Post by N1H1L »

dragontamer wrote: post #25
meragrin_ wrote: post #24

Earlier quoted context omitted.

> The main issue with AMD is that they're cutting support for their older cards. No, their main issue is not properly supporting ROCm in general. No Windows support at all? It still feels like they don't know whether they want to continue investing in ROCm long term.

I'd assume that they're gonna support ROCm as long as the Frontier deployment at Oak Ridge National Labs is up. ORNLs isn't exactly a customer you want to piss off.

Yes. I am hopeful Frontier will shake things up.

Re: Triton: Open-Source GPU Programming for Neural Networks

#73

Post by N1H1L »

dragontamer wrote: post #59
gnufx wrote: post #51

Earlier quoted context omitted.

Yes, I agree with the basic point. Where is the spec for El Capitain, then? I don't remember ever finding it, and I rather assumed it would wait for possible lessons from the Frontier hardware. It's possibly worth saying that these things presumably are competitive with nvlink-based multi-GPU, and appear to support unified memory, but I don't know how that compares with ac922s. Not that UM is relevant for what I know…

Wow. This is quite frustrating. I have plenty of links, but they're all from non-primary source newsites, like servethehome or (worse) wcftech. Those are more "casual" blogs that aren't worth quoting IMO and are closer to rumormills (even if they sometimes have good information). I can't find a single spec from AMD's website or ORNL, or the Department of Energy (primary sources: the ones we know would tell the truth…

Try this

https://www.olcf.ornl.gov/frontier/

Re: Triton: Open-Source GPU Programming for Neural Networks

#74

Post by lsb »

boulos wrote: post #5

Folks might find the author’s research paper [1] while at Harvard more informative. This is a great high-level description, but if you want more detail, I recommend the paper. [1] https://dl.acm.org/doi/abs/10.1145/3315508.3329973

That's http://www.eecs.harvard.edu/~htk/publication/2019-mapl-tille... for those of us outside the paywall

Re: Triton: Open-Source GPU Programming for Neural Networks

#75

Post by queuebert »

lsb wrote: post #74
boulos wrote: post #5

Folks might find the author’s research paper [1] while at Harvard more informative. This is a great high-level description, but if you want more detail, I recommend the paper. [1] https://dl.acm.org/doi/abs/10.1145/3315508.3329973

That's http://www.eecs.harvard.edu/~htk/publication/2019-mapl-tille... for those of us outside the paywall

Always wait for the second link.

Re: Triton: Open-Source GPU Programming for Neural Networks

#77

Post by make3 »

JDDunn9 wrote: post #58

> CPUs and AMD GPUs are not supported at the moment, but we welcome community contributions aimed at addressing this limitation. That's disappointing. My biggest frustration is that every ML library (Pytorch, Keras, etc) is tied to CUDA/Nvidia, so I take a huge performance hit when running them on my Mac.

not pytorch anymore. there's support now

Re: Triton: Open-Source GPU Programming for Neural Networks

#78

Post by nl »

enos_feedler wrote: post #67
dragontamer wrote: post #63

Earlier quoted context omitted.

> In my understanding, the architecture of a GPU and CPU are so fundamentally different that it doesn’t reallly make sense. Yes and no. You find surprising bits of code that can be shared. Read through a high-performance GPU project like "GPU perft" ( https://github.com/ankan-ban/perft_gpu/blob/master/chess.h ). The "perft" problem is the problem of counting the number of board positions reachable in X moves. (Perft(…

Sounds similar to business logic shared between and client and server web app. Thanks I just wonder if this alone is enough to make a unified programming environment the preferred way. On the web we had Meteor.js which tried this approach to unify client and server with javascript. The shared code was of similar types. Meteor never became the preferred way to write web apps. Im wondering if the same is true for GPU +…

This analogy is insufficient to capture what is going on.

In web dev, the client and server are doing fundamentally different things (eg UX vs data storage) and while sharing business logic is interesting it doesn't solve a huge problem.

CPU vs GPU is vaugly analogous to different web browsers on the client in the old days when they were very different. They have different performance characteristics, so shims worked but had performance impact. But they were widely used because it was too hard to do yourself.

Re: Triton: Open-Source GPU Programming for Neural Networks

#79

Post by anon_tor_12345 »

N1H1L wrote: post #71
dragontamer wrote: post #11

Earlier quoted context omitted.

AMD's ROCm 4.0 now supports cooperative groups, which is probably one of the last major holdouts for CUDA compatibility. There's still the 64-wavefront (for AMD CDNA cards) instead of 32-wavefronts (for CUDA). But AMD even has 4x4 half-float matrix multiplication instructions in ROCm (for MI100, the only card that supports the matrix-multiplication / tensor instructions) --------- I think CUDA vs OpenCL is over. ROCm…

Two reasons also. Some of the fastest research supercomputers in the US have been nVIDIA based. Sitting at a national lab, it's laughable how much more support there is from nVIDIA. Getting a DGX box is easy - there is nothing comparable from AMD's side. And the less said about Intel it's better. Smart, motivated people I know have left Argonne recently because of the Aurora shitshow.

E.g. Pavan lol

Re: Triton: Open-Source GPU Programming for Neural Networks

#80

Post by nl »

N1H1L wrote: post #73
dragontamer wrote: post #59

Earlier quoted context omitted.

Wow. This is quite frustrating. I have plenty of links, but they're all from non-primary source newsites, like servethehome or (worse) wcftech. Those are more "casual" blogs that aren't worth quoting IMO and are closer to rumormills (even if they sometimes have good information). I can't find a single spec from AMD's website or ORNL, or the Department of Energy (primary sources: the ones we know would tell the truth…

Try this https://www.olcf.ornl.gov/frontier/

Still doesn't have specs I can see!
Post reply on HN