Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Triton: Open-Source GPU Programming for Neural Networks

openai.com

Post reply on HN
Print view View original

31–40 of 116 posts

Re: Triton: Open-Source GPU Programming for Neural Networks

#31

Post by gnufx »

dragontamer wrote: post #27
shubuZ wrote: post #8

Earlier quoted context omitted.

Which other hardware vendor provides the level of performance that Nvidia's GPU provide? Wasnt the benevolence on single (or couple) manufacturer(s) true in 90s, 2020s?

AMD's MI100 is slightly faster than NVidia A100 for double-precision FLOPs at slightly lower costs. Good enough for Oak Ridge National Labs (Frontier Supercomputer), to say the least. NVidia is faster at 4x4 16-bit matrix multiplications (common in Tensor / Deep Learning stuff), but MI100 still has 4x4 16-bit matrix multiplication instructions and acceleration. Its not far behind, and the greater 64-bit FLOPs is enou…

Is Frontier known to be using M100? I haven't been following along, but it was trailed as using purpose-built GPUs (presumably also for Sierra).

Re: Triton: Open-Source GPU Programming for Neural Networks

#32

Post by dragontamer »

gnufx wrote: post #30
dragontamer wrote: post #19

Earlier quoted context omitted.

If you need CPU-side SIMD, then try ispc: https://ispc.github.io/ Its pretty much the OpenCL-model, except it compiles into AVX2 code / AVX512 code. Very similar to CUDA / OpenCL style programming. Its not single-source like CUDA, but it largely accomplishes the programming model IMO.

Why not a standard? OpenMP is more than pretty much C(++) and Fortran, and has offload inspired by the needs of the Sierra supercomputer.

OpenMP 4.5+ looks very promising to me (particularly the "simd" keyword associated with for-loops). But open-source implementations of OpenMP are somewhat lackluster... at least last time I checked it out.

Maybe its time I revisit it. I've always thought the OpenMP spec was being written by highly competent programmers. They seem to "get" what is needed. But the question is if I can get my hands on any OpenMP implementation to actually play with.

Not all of us can afford IBM's compiler suite!

Re: Triton: Open-Source GPU Programming for Neural Networks

#33

Post by dragontamer »

gnufx wrote: post #31
dragontamer wrote: post #27

Earlier quoted context omitted.

AMD's MI100 is slightly faster than NVidia A100 for double-precision FLOPs at slightly lower costs. Good enough for Oak Ridge National Labs (Frontier Supercomputer), to say the least. NVidia is faster at 4x4 16-bit matrix multiplications (common in Tensor / Deep Learning stuff), but MI100 still has 4x4 16-bit matrix multiplication instructions and acceleration. Its not far behind, and the greater 64-bit FLOPs is enou…

Is Frontier known to be using M100? I haven't been following along, but it was trailed as using purpose-built GPUs (presumably also for Sierra).

Frontier is going to be an AMD-GPU for sure. Whether its MI100 or the next generation MI200 is still not really known and subject to rumors. I'm currently under the assumption that its MI100 however, but now that you press me on that detail I admit that I'm somewhat assuming. Frontier will be using ROCm in any case.

El Capitan (Lawrence Rivermore National Labs) is MI100 and specified as such. So I'll "retreat" my argument to that more easily verified position. Still a top supercomputer in the world, and more is known about El Capitan than Frontier.

Re: Triton: Open-Source GPU Programming for Neural Networks

#34

Post by jpf0 »

dragontamer wrote: post #32
gnufx wrote: post #30

Earlier quoted context omitted.

Why not a standard? OpenMP is more than pretty much C(++) and Fortran, and has offload inspired by the needs of the Sierra supercomputer.

OpenMP 4.5+ looks very promising to me (particularly the "simd" keyword associated with for-loops). But open-source implementations of OpenMP are somewhat lackluster... at least last time I checked it out. Maybe its time I revisit it. I've always thought the OpenMP spec was being written by highly competent programmers. They seem to "get" what is needed. But the question is if I can get my hands on any OpenMP impleme…

LLVM has an openMP implementation

Re: Triton: Open-Source GPU Programming for Neural Networks

#35

Post by dragontamer »

jpf0 wrote: post #34
dragontamer wrote: post #32

Earlier quoted context omitted.

OpenMP 4.5+ looks very promising to me (particularly the "simd" keyword associated with for-loops). But open-source implementations of OpenMP are somewhat lackluster... at least last time I checked it out. Maybe its time I revisit it. I've always thought the OpenMP spec was being written by highly competent programmers. They seem to "get" what is needed. But the question is if I can get my hands on any OpenMP impleme…

LLVM has an openMP implementation

The task-based parallelism in LLVM leaves much to be desired however. Ideally, you'd want a more efficient implementation.

But yeah, good enough to play with. But maybe not good enough to achieve high levels of performance. The SIMD stuff is probably simple enough to implement... maybe I should checkout how well LLVM works with OMP SIMD keywords.

Re: Triton: Open-Source GPU Programming for Neural Networks

#36

Post by 37ef_ced3 »

shubuZ wrote: post #9

I have found writing CUDA code is much simpler than writing correct multi-threaded AVX2/AVX-512 code.

Use a domain-specific compiler to generate custom, stand-alone, massively multi-threaded AVX-512 inference C code:

https://NN-512.com

The generated code is easily twice as fast as TensorFlow's AVX-512 kernels (Intel's oneAPI).

Re: Triton: Open-Source GPU Programming for Neural Networks

#37

Post by einpoklum »

gnufx wrote: post #30
dragontamer wrote: post #19

Earlier quoted context omitted.

If you need CPU-side SIMD, then try ispc: https://ispc.github.io/ Its pretty much the OpenCL-model, except it compiles into AVX2 code / AVX512 code. Very similar to CUDA / OpenCL style programming. Its not single-source like CUDA, but it largely accomplishes the programming model IMO.

Why not a standard? OpenMP is more than pretty much C(++) and Fortran, and has offload inspired by the needs of the Sierra supercomputer.

It's much more limited in expressivity than OpenCL/CUDA, IIANM.

Re: Triton: Open-Source GPU Programming for Neural Networks

#38

Post by ipsum2 »

xmaayy wrote: post #22

I wonder if this can be used for graphics programming. Shaders are notoriously hard to write correctly and this seems like it might provide an easier gateway than OpenGLSL

Taichi is a similar project focused on graphics: https://taichi.graphics/

Re: Triton: Open-Source GPU Programming for Neural Networks

#39

Post by anon_tor_12345 »

jpf0 wrote: post #12
blueblisters wrote: post #6

Earlier quoted context omitted.

So is this similar to XLA?

XLA is domain-specific compiler for linear algebra. Triton generates and compiles an intermediate representation for tiled computation. This IR allows more general functions and also claims higher performance. obligatory reference to the family of work: https://github.com/merrymercy/awesome-tensor-compilers

Without reading the paper, I think you have it a little backwards - the IR doesn't itself allow for more general functions. More general functions are possible (in theory) because the frontend (this Triton language) is decoupled from the backend (CUDA) through the IR as an interface. In this way the Triton IR is no less domain specific than XLA (because both are IRs that represent sequences of operators that run on GPU (or TPU or whatever). I guess in theory Triton could be eschewing all of eg cuDNN but most likely it's not as NVIDIA's closed source kernels perform best on their closed source hardware.

Edit: should've read the post before commenting. Looks like they are in fact using LLVM's PTX backend (ie generating cuda kernels from scratch). Kudos to them

Re: Triton: Open-Source GPU Programming for Neural Networks

#40

Post by fzimmermann89 »

So the code looks (apart from pointers) similiar to numba which feels much closer to numpy/pytorch high level code. Are there huge advantages in the triton model compared to numba that I don't see? Or is there a big performance gap? For me numba was always the easiest way to get some new idea running on cuda, and most of the time it was fast enough..

Did anybody find performance comparison between numba and triton?

Post reply on HN