Triton: Open-Source GPU Programming for Neural Networks • Hacker News Board

Re: Triton: Open-Source GPU Programming for Neural Networks

#51

Post by gnufx » Wed, Jul 28, 2021, 9:08 PM UTC

dragontamer wrote: post #33

gnufx wrote: post #31

Earlier quoted context omitted.

Is Frontier known to be using M100? I haven't been following along, but it was trailed as using purpose-built GPUs (presumably also for Sierra).

Frontier is going to be an AMD-GPU for sure. Whether its MI100 or the next generation MI200 is still not really known and subject to rumors. I'm currently under the assumption that its MI100 however, but now that you press me on that detail I admit that I'm somewhat assuming. Frontier will be using ROCm in any case. El Capitan (Lawrence Rivermore National Labs) is MI100 and specified as such. So I'll "retreat" my arg…

Yes, I agree with the basic point. Where is the spec for El Capitain, then? I don't remember ever finding it, and I rather assumed it would wait for possible lessons from the Frontier hardware.

It's possibly worth saying that these things presumably are competitive with nvlink-based multi-GPU, and appear to support unified memory, but I don't know how that compares with ac922s. Not that UM is relevant for what I know of usage on our system....

Re: Triton: Open-Source GPU Programming for Neural Networks

#52

Post by dragontamer » Wed, Jul 28, 2021, 9:09 PM UTC

Athas wrote: post #50

dragontamer wrote: post #11

Earlier quoted context omitted.

AMD's ROCm 4.0 now supports cooperative groups, which is probably one of the last major holdouts for CUDA compatibility. There's still the 64-wavefront (for AMD CDNA cards) instead of 32-wavefronts (for CUDA). But AMD even has 4x4 half-float matrix multiplication instructions in ROCm (for MI100, the only card that supports the matrix-multiplication / tensor instructions) --------- I think CUDA vs OpenCL is over. ROCm…

> ROCm from AMD has its restrictions, but... it really is easier to program than OpenCL. Its a superior model: having a single language that supports both CPU and GPU code is just easier than switching between C++ and OpenCL (where data-structures can't be shared as easily). ROCm isn't a programming language or programming model - it's a collection of tools and libraries. Which language are you specifically referenci…

Yeah, HIP specifically. That's the part of ROCm that is aiming to give a CUDA-like environment.

Re: Triton: Open-Source GPU Programming for Neural Networks

#53

Post by gnufx » Wed, Jul 28, 2021, 9:14 PM UTC

dragontamer wrote: post #32

gnufx wrote: post #30

Earlier quoted context omitted.

Why not a standard? OpenMP is more than pretty much C(++) and Fortran, and has offload inspired by the needs of the Sierra supercomputer.

OpenMP 4.5+ looks very promising to me (particularly the "simd" keyword associated with for-loops). But open-source implementations of OpenMP are somewhat lackluster... at least last time I checked it out. Maybe its time I revisit it. I've always thought the OpenMP spec was being written by highly competent programmers. They seem to "get" what is needed. But the question is if I can get my hands on any OpenMP impleme…

I should really find time to do comparisons with the compilers to hand, including XL, on the NAS benchmarks, as I've never seen that, though it must have been done. I think we have a "community" version of XL, i.e. no support, like basically everything else. I wasn't aware there was anything much wrong with GCC and libgomp or libomp, but then I haven't measured.

Re: Triton: Open-Source GPU Programming for Neural Networks

#54

Post by gnufx » Wed, Jul 28, 2021, 9:18 PM UTC

dragontamer wrote: post #35

jpf0 wrote: post #34

Earlier quoted context omitted.

LLVM has an openMP implementation

The task-based parallelism in LLVM leaves much to be desired however. Ideally, you'd want a more efficient implementation. But yeah, good enough to play with. But maybe not good enough to achieve high levels of performance. The SIMD stuff is probably simple enough to implement... maybe I should checkout how well LLVM works with OMP SIMD keywords.

In LLVM or in libomp? I don't know what omp simd is likely to get you over autovectorization. I know of cases where it was thought necessary (-fopenmp-simd, without -fopenmp) but wasn't with recent GCC.

Re: Triton: Open-Source GPU Programming for Neural Networks

#55

Post by dragontamer » Wed, Jul 28, 2021, 9:26 PM UTC

jpf0 wrote: post #49

dragontamer wrote: post #35

Earlier quoted context omitted.

The task-based parallelism in LLVM leaves much to be desired however. Ideally, you'd want a more efficient implementation. But yeah, good enough to play with. But maybe not good enough to achieve high levels of performance. The SIMD stuff is probably simple enough to implement... maybe I should checkout how well LLVM works with OMP SIMD keywords.

Can you comment on experience (or contact me) regarding implementation efficiency? We have recently implemented task-based parallelism in the J language with openMP[0]. Improvements or critiques are appreciated. SIMD instructions there have been coded directly rather than via pragmas. [0] https://www.monument.ai/m/parallel

I can't say that my critiques are based off of personal experience. But mostly about microbenchmarks I've read that other people have talked about. I am probably a bit out of date, since its been a while since I last played with OpenMP.

I'm looking at the benchmarks I used to look at, and they're all from 2014 or earlier. So maybe I really should double-check modern implementations. We all know GCC 4.x and LLVM 3.x are an eternity ago, so I probably should revisit their performance.

For example: https://www.phoronix.com/scan.php?page=article&item=llvm_cla...

And back then, it was pretty well known that OpenMP implementations were slower than commercial (such as Intel ICC or IBM's OpenMP implementation).

Re: Triton: Open-Source GPU Programming for Neural Networks

#56

Post by CuriousCosmic » Wed, Jul 28, 2021, 9:29 PM UTC

peytoncasper wrote: post #3

riyadparvez wrote: post #2

I am confused. Is it another competitor of Tensorflow, JAX, and Pytorch? Or something else?

I believe this is more of an optimization layer to be utilized by libraries like Tensorflow and JAX. More of a simplification of the interaction with traditional CUDA instructions. I imagine these libraries and possibly some users would implement libraries on top of this language and reap some of the optimization benefit without having to maintain low-level CUDA specific code.

Im curious how it would compare to Halide Lang. They both seem to be targetting the same problem.

Re: Triton: Open-Source GPU Programming for Neural Networks

#57

Post by dragontamer » Wed, Jul 28, 2021, 9:41 PM UTC

gnufx wrote: post #54

dragontamer wrote: post #35

Earlier quoted context omitted.

The task-based parallelism in LLVM leaves much to be desired however. Ideally, you'd want a more efficient implementation. But yeah, good enough to play with. But maybe not good enough to achieve high levels of performance. The SIMD stuff is probably simple enough to implement... maybe I should checkout how well LLVM works with OMP SIMD keywords.

In LLVM or in libomp? I don't know what omp simd is likely to get you over autovectorization. I know of cases where it was thought necessary (-fopenmp-simd, without -fopenmp) but wasn't with recent GCC.

Autovectorization has issues with function calls.

"#pragma omp declare simd" applies over a function call, which then allows that function to be used inside of a "#pragma omp for simd" loop.

A few keywords here and there really help the autovectorizer achieve closer to CUDA-like environments (like... actually having your SIMD code extend "through" a function call, so you can start splitting up the work a bit better).

EDIT: Here's an example from Intel's ICC: https://software.intel.com/content/www/us/en/develop/documen...

Re: Triton: Open-Source GPU Programming for Neural Networks

#58

Post by JDDunn9 » Wed, Jul 28, 2021, 9:44 PM UTC

> CPUs and AMD GPUs are not supported at the moment, but we welcome community contributions aimed at addressing this limitation.

That's disappointing. My biggest frustration is that every ML library (Pytorch, Keras, etc) is tied to CUDA/Nvidia, so I take a huge performance hit when running them on my Mac.

Re: Triton: Open-Source GPU Programming for Neural Networks

#59

Post by dragontamer » Wed, Jul 28, 2021, 9:44 PM UTC

gnufx wrote: post #51

dragontamer wrote: post #33

Earlier quoted context omitted.

Frontier is going to be an AMD-GPU for sure. Whether its MI100 or the next generation MI200 is still not really known and subject to rumors. I'm currently under the assumption that its MI100 however, but now that you press me on that detail I admit that I'm somewhat assuming. Frontier will be using ROCm in any case. El Capitan (Lawrence Rivermore National Labs) is MI100 and specified as such. So I'll "retreat" my arg…

Yes, I agree with the basic point. Where is the spec for El Capitain, then? I don't remember ever finding it, and I rather assumed it would wait for possible lessons from the Frontier hardware. It's possibly worth saying that these things presumably are competitive with nvlink-based multi-GPU, and appear to support unified memory, but I don't know how that compares with ac922s. Not that UM is relevant for what I know…

Wow. This is quite frustrating.

I have plenty of links, but they're all from non-primary source newsites, like servethehome or (worse) wcftech. Those are more "casual" blogs that aren't worth quoting IMO and are closer to rumormills (even if they sometimes have good information).

I can't find a single spec from AMD's website or ORNL, or the Department of Energy (primary sources: the ones we know would tell the truth on this matter).

The closest I got is this: https://www.amd.com/system/files/documents/oak-ridge-nationa...

Which is an MI100 test that ORNLs did in "preparation" for Frontier. It doesn't seem like Frontier nor El Capitan have had their specs officially released yet.

Re: Triton: Open-Source GPU Programming for Neural Networks

#60

Post by ganoushoreilly » Wed, Jul 28, 2021, 10:01 PM UTC

JDDunn9 wrote: post #58

> CPUs and AMD GPUs are not supported at the moment, but we welcome community contributions aimed at addressing this limitation. That's disappointing. My biggest frustration is that every ML library (Pytorch, Keras, etc) is tied to CUDA/Nvidia, so I take a huge performance hit when running them on my Mac.

I'm really surprised AMD hasn't pushed much harder into the space given their aggressive targeting of the data center with Epyc. I too use mostly Macs during the day and any ML projects are always relegated to Nvidia machines in my rack / amazon.