Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Triton: Open-Source GPU Programming for Neural Networks

openai.com

Post reply on HN
Print view View original

41–50 of 116 posts

Re: Triton: Open-Source GPU Programming for Neural Networks

#41

Post by einpoklum »

dragontamer wrote: post #11
croes wrote: post #4

Too bad it's CUDA Sooner or later this will become a problem because you are depending on the benevolence of a single manufacturer.

AMD's ROCm 4.0 now supports cooperative groups, which is probably one of the last major holdouts for CUDA compatibility. There's still the 64-wavefront (for AMD CDNA cards) instead of 32-wavefronts (for CUDA). But AMD even has 4x4 half-float matrix multiplication instructions in ROCm (for MI100, the only card that supports the matrix-multiplication / tensor instructions) --------- I think CUDA vs OpenCL is over. ROCm…

> ROCm from AMD has its restrictions, but... it really is easier to program than OpenCL

That really not the point. OpenCL is a standard that - at least in principle - is supposed to be supported on multiple platforms by multiple vendors. ROCm is AMD-only, and even that is questionable since it didn't exist 6 or 7 years ago, and who knows - they might drop it like they changed their earlier focus.

Also, CUDA has a much richer ecosystem than AMD ROCm (I'm sad to say; as I'm not a fan of NVIDIA).

Re: Triton: Open-Source GPU Programming for Neural Networks

#42

Post by my123 »

einpoklum wrote: post #41
dragontamer wrote: post #11

Earlier quoted context omitted.

AMD's ROCm 4.0 now supports cooperative groups, which is probably one of the last major holdouts for CUDA compatibility. There's still the 64-wavefront (for AMD CDNA cards) instead of 32-wavefronts (for CUDA). But AMD even has 4x4 half-float matrix multiplication instructions in ROCm (for MI100, the only card that supports the matrix-multiplication / tensor instructions) --------- I think CUDA vs OpenCL is over. ROCm…

> ROCm from AMD has its restrictions, but... it really is easier to program than OpenCL That really not the point. OpenCL is a standard that - at least in principle - is supposed to be supported on multiple platforms by multiple vendors. ROCm is AMD-only, and even that is questionable since it didn't exist 6 or 7 years ago, and who knows - they might drop it like they changed their earlier focus. Also, CUDA has a muc…

Reminder that ROCm is still not supported on RDNA and RDNA2.

Re: Triton: Open-Source GPU Programming for Neural Networks

#43

Post by einpoklum »

croes wrote: post #21
shubuZ wrote: post #8

Earlier quoted context omitted.

Which other hardware vendor provides the level of performance that Nvidia's GPU provide? Wasnt the benevolence on single (or couple) manufacturer(s) true in 90s, 2020s?

It's not about performance but open standards. Remember Oracle vs. Google, at some time in the future NVidia could decide to get money out of CUDA.

Well, yes and no. Neither AMD nor NVIDIA are committed to open, libre, standards - they've both been "betraying" OpenCL (just like Apple and Intel, I should mention). So we have AMD's not-open (?) ROCm and NVIDIA's not-open CUDA.

Also, if we only cared about open standard we'd simply not use a GPU at all and settle RISC-V chips :-)

Re: Triton: Open-Source GPU Programming for Neural Networks

#44

Post by dragontamer »

einpoklum wrote: post #41
dragontamer wrote: post #11

Earlier quoted context omitted.

AMD's ROCm 4.0 now supports cooperative groups, which is probably one of the last major holdouts for CUDA compatibility. There's still the 64-wavefront (for AMD CDNA cards) instead of 32-wavefronts (for CUDA). But AMD even has 4x4 half-float matrix multiplication instructions in ROCm (for MI100, the only card that supports the matrix-multiplication / tensor instructions) --------- I think CUDA vs OpenCL is over. ROCm…

> ROCm from AMD has its restrictions, but... it really is easier to program than OpenCL That really not the point. OpenCL is a standard that - at least in principle - is supposed to be supported on multiple platforms by multiple vendors. ROCm is AMD-only, and even that is questionable since it didn't exist 6 or 7 years ago, and who knows - they might drop it like they changed their earlier focus. Also, CUDA has a muc…

> OpenCL is a standard that - at least in principle - is supposed to be supported on multiple platforms by multiple vendors.

As was C++AMP (which was actually pretty good IMO as a language). Just because its a standard doesn't mean its going to be used.

OpenCL 2.0 was very poorly implemented: almost no one used any of its advanced features. To the point that OpenCL 3.0 is resetting from OpenCL 1.2.

Only Intel really supported OpenCL 2.1 or OpenCL 2.2. The entire OpenCL 2.x branch for years was squandered with tepid responses from NVidia and AMD (yes, AMD had better OpenCL 2.0 support. But it's debugger didn't work, all of the code was tested on OpenCL 1.2 only. No one cared)

I dare say that OpenCL 2.x was about as "standardized" and respected as C++ AMP. Just because its an open standard doesn't mean that its actually useful. Any serious OpenCL programmer stuck with OpenCL 1.2, including both AMD and NVidia OpenCL programmers.

Re: Triton: Open-Source GPU Programming for Neural Networks

#47

Post by king_magic »

6gvONxR4sf7o wrote: post #29
thebruce87m wrote: post #14

Unfortunate name clash with NVIDIAs Triton Inference Server: https://developer.nvidia.com/nvidia-triton-inference-server

The author commented on reddit about that ( https://www.reddit.com/r/MachineLearning/comments/otdpkx/n_i... ) > PS: The name Triton was coined in mid-2019 when I released my PhD paper on the subject ( http://www.eecs.harvard.edu/~htk/publication/2019-mapl-tille... ). I chose not to rename the project when the Triton inference server came out a year later since it's the only thing that ties my helpful PhD advisors to…

The author is unfortunately wrong. NVIDIA's Triton was referenced in marketing material as far back as 2018. https://developer.nvidia.com/blog/nvidia-serves-deep-learnin...

Re: Triton: Open-Source GPU Programming for Neural Networks

#49

Post by jpf0 »

dragontamer wrote: post #35
jpf0 wrote: post #34

Earlier quoted context omitted.

LLVM has an openMP implementation

The task-based parallelism in LLVM leaves much to be desired however. Ideally, you'd want a more efficient implementation. But yeah, good enough to play with. But maybe not good enough to achieve high levels of performance. The SIMD stuff is probably simple enough to implement... maybe I should checkout how well LLVM works with OMP SIMD keywords.

Can you comment on experience (or contact me) regarding implementation efficiency? We have recently implemented task-based parallelism in the J language with openMP[0]. Improvements or critiques are appreciated. SIMD instructions there have been coded directly rather than via pragmas.

[0] https://www.monument.ai/m/parallel

Re: Triton: Open-Source GPU Programming for Neural Networks

#50

Post by Athas »

dragontamer wrote: post #11
croes wrote: post #4

Too bad it's CUDA Sooner or later this will become a problem because you are depending on the benevolence of a single manufacturer.

AMD's ROCm 4.0 now supports cooperative groups, which is probably one of the last major holdouts for CUDA compatibility. There's still the 64-wavefront (for AMD CDNA cards) instead of 32-wavefronts (for CUDA). But AMD even has 4x4 half-float matrix multiplication instructions in ROCm (for MI100, the only card that supports the matrix-multiplication / tensor instructions) --------- I think CUDA vs OpenCL is over. ROCm…

> ROCm from AMD has its restrictions, but... it really is easier to program than OpenCL. Its a superior model: having a single language that supports both CPU and GPU code is just easier than switching between C++ and OpenCL (where data-structures can't be shared as easily).

ROCm isn't a programming language or programming model - it's a collection of tools and libraries. Which language are you specifically referencing here? HIP?

Post reply on HN