Triton: Open-Source GPU Programming for Neural Networks • Hacker News Board

Re: Triton: Open-Source GPU Programming for Neural Networks

#111

Post by pjmlp » Fri, Jul 30, 2021, 6:31 AM UTC

zozbot234 wrote: post #18

dragontamer wrote: post #11

Earlier quoted context omitted.

AMD's ROCm 4.0 now supports cooperative groups, which is probably one of the last major holdouts for CUDA compatibility. There's still the 64-wavefront (for AMD CDNA cards) instead of 32-wavefronts (for CUDA). But AMD even has 4x4 half-float matrix multiplication instructions in ROCm (for MI100, the only card that supports the matrix-multiplication / tensor instructions) --------- I think CUDA vs OpenCL is over. ROCm…

The new contest is not CUDA vs. OpenCL but CUDA vs. Vulkan Compute. As support for Vulkan in hardware becomes more widespread, it makes more and more sense to just standardize on it for all workloads. The programming model is quite different between the two (kernels vs. shaders) and OpenCL 2.x has quite a few features that are not in Vulkan, but the latest version of OpenCL has downgraded many of these to extensions.

Given the state of Vulkan tooling the contest is already won, NVidia doesn't need to worry about anything.

Re: Triton: Open-Source GPU Programming for Neural Networks

#112

Post by enos_feedler » Fri, Jul 30, 2021, 10:13 PM UTC

nl wrote: post #109

enos_feedler wrote: post #92

Earlier quoted context omitted.

I don’t think this is accurate. CPU and GPU are not trying to execute the same code with different performance characteristics. GPU and CPU are fundamentally doings different things. In fact the analogy for shared business logic works perfectly

No, they are trying to execute exactly the same code. To quote the parent: You can see that the fundamentals of bitboard manipulation (using the 64-bit number to represent the 64-squares of a chessboard) remain the same whether or not you're on a CPU or GPU. In this case, you want the code to be shared between the two sides. Why write the code twice? Both CPUs and GPUs are very good at 64-bit integer manipulation. Th…

The exact same business logic (helper functions), yes of course. My point is the high level algorithm is going to be fundmanetally different. Just like the high level algorithm on the browser (render and handle UI interaction) is different than the fundamental server-side algorithm (render html from database requests), and yet there might be shared helper functions. Does this make sense?

Re: Triton: Open-Source GPU Programming for Neural Networks

#113

Post by nl » Sun, Aug 01, 2021, 1:03 AM UTC

enos_feedler wrote: post #112

nl wrote: post #109

Earlier quoted context omitted.

No, they are trying to execute exactly the same code. To quote the parent: You can see that the fundamentals of bitboard manipulation (using the 64-bit number to represent the 64-squares of a chessboard) remain the same whether or not you're on a CPU or GPU. In this case, you want the code to be shared between the two sides. Why write the code twice? Both CPUs and GPUs are very good at 64-bit integer manipulation. Th…

The exact same business logic (helper functions), yes of course. My point is the high level algorithm is going to be fundmanetally different. Just like the high level algorithm on the browser (render and handle UI interaction) is different than the fundamental server-side algorithm (render html from database requests), and yet there might be shared helper functions. Does this make sense?

No, the algorithm is exactly the same.

For example, in neural networks the key algorithm is differentiation. That is implemented the same way on both CPUs and GPUs.

Re: Triton: Open-Source GPU Programming for Neural Networks

#114

Post by 37ef_ced3 » Mon, Aug 02, 2021, 8:43 PM UTC

johndough wrote: post #104

37ef_ced3 wrote: post #102

Earlier quoted context omitted.

Here is a guy who worked it out: https://github.com/jonatron/test_nn512 That's a full ResNet50 conversion from the corresponding Caffe protobuf. He also wrote a small script to convert TensorFlow networks. GCC warns about perfectly valid, correct C. It even warns about switch case fallthough. All of the generated code is fully correct. Turn the warnings off. Please don't copy and paste from the website! Download the…

Thanks, that code does indeed work! And it is significantly faster than PyTorch (15 vs 60 milliseconds). Maybe I will use it in the future if AVX512 should become more widespread.

The new NN-512 release (version 35) changes the build instructions to suppress GCC's warnings while building the generated code (-w switch):

  // To build an object file:
  // gcc -c -w -std=c99 -pthread -Ofast -mavx512f Example1.c

Thanks for your feedback.

Re: Triton: Open-Source GPU Programming for Neural Networks

#115

Post by N1H1L » Mon, Aug 02, 2021, 9:05 PM UTC

dragontamer wrote: post #105

N1H1L wrote: post #88

Earlier quoted context omitted.

Aurora was supposed to be up and running in early 2018. It seems like that it won't be functional even in 2021. This is by the way, such an years long delay has never happened when it comes to US Leadership Computing Facilities. Intel, last I heard has written off close to 300 million dollars on Aurora.

Do you know if there's any post-mortems about what went wrong there? I mean, I have the gist. Intel bet big on Xeon Phi, but that didn't seem to offer enough performance. Intel then switched over to this GPU-strategy (see Intel Xe), but that required them to rework virtually everything from scratch. No one said anything in my previous paragraph. But its blatantly obvious: the Xeon Phi was being advertised very strong…

My friends were at more junior levels so I don't know the full details too. However know this, a leadership class computing purchase by a national lab is a very complicated thing. Many people have their hands in the decision making pie - all the way up to Secretary of Energy.

Secondly, DOE has a strategy (mostly rightly in my opinion) of not putting all its' computation eggs in a single companies' basket. Thus national labs compete between themselves as do companies. Notice the cadence of computing purchases: IBM/Nvidia -> Intel/Intel -> AMD/AMD.

Thirdly, Intel had promised that using OneAPI existing GPU optimized simulation codes could be translated to Intel GPUs with minimum effort. The idea had merit, back in 2016-2017. I have a CUDA based simulation code - it's almost a matter of recompiling to Intel, with some minimal effort on my part. That it didn't work out - well hindsight is 20/20

Re: Triton: Open-Source GPU Programming for Neural Networks

#116

Post by enos_feedler » Fri, Aug 06, 2021, 3:58 AM UTC

nl wrote: post #113

enos_feedler wrote: post #112

Earlier quoted context omitted.

The exact same business logic (helper functions), yes of course. My point is the high level algorithm is going to be fundmanetally different. Just like the high level algorithm on the browser (render and handle UI interaction) is different than the fundamental server-side algorithm (render html from database requests), and yet there might be shared helper functions. Does this make sense?

No, the algorithm is exactly the same. For example, in neural networks the key algorithm is differentiation. That is implemented the same way on both CPUs and GPUs.

differentiation isn't really the key algorithm. In fact, I believe that happens at compile time, not run-time. So not only is not key, it isn't even happening at run-time.