Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Triton: Open-Source GPU Programming for Neural Networks

openai.com

Post reply on HN
Print view View original

101–110 of 116 posts

Re: Triton: Open-Source GPU Programming for Neural Networks

#101

Post by king_magic »

0x008 wrote: post #99
king_magic wrote: post #47

Earlier quoted context omitted.

The author is unfortunately wrong. NVIDIA's Triton was referenced in marketing material as far back as 2018. https://developer.nvidia.com/blog/nvidia-serves-deep-learnin...

Are we sure that marketing article was not changed after the fact? The Nvidia Triton Release notes show 'Starting in release 20.03, TensorRT Inference Server is now called Triton Inference Server.' [0] It kind of looks like the article just changed the heading later since in the article the docker images they refer to etc. are all still called 'tensorrtserver' which was the project's name before they changed naming i…

Eh, I suspect the author is mistaken here, but who knows, maybe that was updated after the fact? Hard to say who is right.

Re: Triton: Open-Source GPU Programming for Neural Networks

#102

Post by 37ef_ced3 »

johndough wrote: post #95
37ef_ced3 wrote: post #36

Earlier quoted context omitted.

Use a domain-specific compiler to generate custom, stand-alone, massively multi-threaded AVX-512 inference C code: https://NN-512.com The generated code is easily twice as fast as TensorFlow's AVX-512 kernels (Intel's oneAPI).

Some user feedback: I tried to get the ResNet50 example to work, but I gave up after 2 hours. There really should be an end-to-end example like "./resnet50_example.py monkey.jpg". A few points where I struggled: - What are the names in the ResNet50Params struct? I looked at a few popular ResNet50 implementations, but I could not find any correspondence and since the names have been sorted, the order of the members mi…

Here is a guy who worked it out:

https://github.com/jonatron/test_nn512

That's a full ResNet50 conversion from the corresponding Caffe protobuf. He also wrote a small script to convert TensorFlow networks.

GCC warns about perfectly valid, correct C. It even warns about switch case fallthough. All of the generated code is fully correct. Turn the warnings off.

Please don't copy and paste from the website! Download the NN-512 executable and run it.

Email me if you still have difficulties.

Re: Triton: Open-Source GPU Programming for Neural Networks

#103

Post by dragontamer »

enos_feedler wrote: post #92
nl wrote: post #78

Earlier quoted context omitted.

This analogy is insufficient to capture what is going on. In web dev, the client and server are doing fundamentally different things (eg UX vs data storage) and while sharing business logic is interesting it doesn't solve a huge problem. CPU vs GPU is vaugly analogous to different web browsers on the client in the old days when they were very different. They have different performance characteristics, so shims worked…

I don’t think this is accurate. CPU and GPU are not trying to execute the same code with different performance characteristics. GPU and CPU are fundamentally doings different things. In fact the analogy for shared business logic works perfectly

The other poster is right. The CPU and GPU are just two execution resources available to the programmer, no different than spinning up a new pthread_create (well... a bit more complicated but you get the gist).

Even in video games, where GPUs do graphics and CPUs do physics, the lines have been blurred with TressFX (hair physics on GPU instead of CPU) or PhysX (collision physics done on GPU instead of CPU).

GPUs are general purpose computers now. You can program them to do whatever you want. So when the programmer is writing code, it pays to experiment a little: maybe run the code on the GPU... see if the GPU really is slower or faster. You might be surprised.

Ditto on the reverse: maybe some bit of GPU code would be better run on the CPU.

--------

This guess-and-check, program-and-profile methodology is the same as any other high-performance code people write. You throw away assumptions and just test the heck out of all your ideas, because you're surprised far more often than not in this field.

As such, having the ability to quickly move code from GPU-side to CPU-side (and vice versa) for these tests is extremely beneficial. You don't really know which device will run your code faster until you actually test it. (Though your instincts get better the more experience you gain)

Re: Triton: Open-Source GPU Programming for Neural Networks

#104

Post by johndough »

37ef_ced3 wrote: post #102
johndough wrote: post #95

Earlier quoted context omitted.

Some user feedback: I tried to get the ResNet50 example to work, but I gave up after 2 hours. There really should be an end-to-end example like "./resnet50_example.py monkey.jpg". A few points where I struggled: - What are the names in the ResNet50Params struct? I looked at a few popular ResNet50 implementations, but I could not find any correspondence and since the names have been sorted, the order of the members mi…

Here is a guy who worked it out: https://github.com/jonatron/test_nn512 That's a full ResNet50 conversion from the corresponding Caffe protobuf. He also wrote a small script to convert TensorFlow networks. GCC warns about perfectly valid, correct C. It even warns about switch case fallthough. All of the generated code is fully correct. Turn the warnings off. Please don't copy and paste from the website! Download the…

Thanks, that code does indeed work! And it is significantly faster than PyTorch (15 vs 60 milliseconds). Maybe I will use it in the future if AVX512 should become more widespread.

Re: Triton: Open-Source GPU Programming for Neural Networks

#105

Post by dragontamer »

N1H1L wrote: post #88
bradneuberg wrote: post #84

Earlier quoted context omitted.

Can you comment more on what’s going on at Argonne related to Aurora?

Aurora was supposed to be up and running in early 2018. It seems like that it won't be functional even in 2021. This is by the way, such an years long delay has never happened when it comes to US Leadership Computing Facilities. Intel, last I heard has written off close to 300 million dollars on Aurora.

Do you know if there's any post-mortems about what went wrong there?

I mean, I have the gist. Intel bet big on Xeon Phi, but that didn't seem to offer enough performance. Intel then switched over to this GPU-strategy (see Intel Xe), but that required them to rework virtually everything from scratch.

No one said anything in my previous paragraph. But its blatantly obvious: the Xeon Phi was being advertised very strongly by Intel up to the point that Aurora's design was reworked. Suddenly, Xeon Phi was cut, and Intel started talking about Xe (including OneAPI, and all sorts of stuff the HPC market cares about). I'm confident enough at reading between the lines that I'm willing to assume this in the absence of hard facts :-)

I'm kind of curious on the play by play, if at all possible. What test showed up that made Argonne National Laboratory decide that the Xeon Phi model wasn't going to work? Was it possible to come to this conclusion any sooner? I realize this sort of stuff is rarely published, but I guess that's what makes me interested in those juicy details.

Re: Triton: Open-Source GPU Programming for Neural Networks

#106

Post by 37ef_ced3 »

johndough wrote: post #104
37ef_ced3 wrote: post #102

Earlier quoted context omitted.

Here is a guy who worked it out: https://github.com/jonatron/test_nn512 That's a full ResNet50 conversion from the corresponding Caffe protobuf. He also wrote a small script to convert TensorFlow networks. GCC warns about perfectly valid, correct C. It even warns about switch case fallthough. All of the generated code is fully correct. Turn the warnings off. Please don't copy and paste from the website! Download the…

Thanks, that code does indeed work! And it is significantly faster than PyTorch (15 vs 60 milliseconds). Maybe I will use it in the future if AVX512 should become more widespread.

Awesome. Send me email any time and I will assist you.

Re: Triton: Open-Source GPU Programming for Neural Networks

#107

Post by gnufx »

dragontamer wrote: post #57
gnufx wrote: post #54

Earlier quoted context omitted.

In LLVM or in libomp? I don't know what omp simd is likely to get you over autovectorization. I know of cases where it was thought necessary (-fopenmp-simd, without -fopenmp) but wasn't with recent GCC.

Autovectorization has issues with function calls. "#pragma omp declare simd" applies over a function call, which then allows that function to be used inside of a "#pragma omp for simd" loop. A few keywords here and there really help the autovectorizer achieve closer to CUDA-like environments (like... actually having your SIMD code extend "through" a function call, so you can start splitting up the work a bit better).…

I took the example program from the OpenMP standard and built it with GCC 11 -Ofast. -fopt-info said the relevant loop was vectorized. Adding -fopenmp gave more vectorization messages from elsewhere, but I don't have time to figure out the difference from the tree dump (not being good with assembler). Doubtless the directives can help, but you do need to get them right, and I trust GCC more than me!

Re: Triton: Open-Source GPU Programming for Neural Networks

#108

Post by nl »

dragontamer wrote: post #103
enos_feedler wrote: post #92

Earlier quoted context omitted.

I don’t think this is accurate. CPU and GPU are not trying to execute the same code with different performance characteristics. GPU and CPU are fundamentally doings different things. In fact the analogy for shared business logic works perfectly

The other poster is right. The CPU and GPU are just two execution resources available to the programmer, no different than spinning up a new pthread_create (well... a bit more complicated but you get the gist). Even in video games, where GPUs do graphics and CPUs do physics, the lines have been blurred with TressFX (hair physics on GPU instead of CPU) or PhysX (collision physics done on GPU instead of CPU). GPUs are…

To expand on this:

GPUs are (basically) hardware devices for doing vast numbers of matrix manipulations very very fast in parallel, with limitations on the size of the matrix.

CPUs can do matrix manipulations, with less parallelism.

Things like TressFX uses the tridiagonal matrix algorithm[1] to solve a linear system representing the hair constraints system. This could be implemented on a CPU as well, doing exactly the same thing with the same algorithm. It would run slower though!

[1] https://en.wikipedia.org/wiki/Tridiagonal_matrix_algorithm

[2] Slide 37, https://www.slideshare.net/DevCentralAMD/gs4147-billbilodeau

Re: Triton: Open-Source GPU Programming for Neural Networks

#109

Post by nl »

enos_feedler wrote: post #92
nl wrote: post #78

Earlier quoted context omitted.

This analogy is insufficient to capture what is going on. In web dev, the client and server are doing fundamentally different things (eg UX vs data storage) and while sharing business logic is interesting it doesn't solve a huge problem. CPU vs GPU is vaugly analogous to different web browsers on the client in the old days when they were very different. They have different performance characteristics, so shims worked…

I don’t think this is accurate. CPU and GPU are not trying to execute the same code with different performance characteristics. GPU and CPU are fundamentally doings different things. In fact the analogy for shared business logic works perfectly

No, they are trying to execute exactly the same code. To quote the parent:

You can see that the fundamentals of bitboard manipulation (using the 64-bit number to represent the 64-squares of a chessboard) remain the same whether or not you're on a CPU or GPU.

In this case, you want the code to be shared between the two sides. Why write the code twice? Both CPUs and GPUs are very good at 64-bit integer manipulation.

The ceremony around co-ordinating threads and memory access is different, but the code that is being run is exactly the same.

Re: Triton: Open-Source GPU Programming for Neural Networks

#110

Post by pjmlp »

dragontamer wrote: post #26
yumraj wrote: post #7

Earlier quoted context omitted.

Sure, but if this abstraction layer becomes popular, then it becomes much easier to support other GPUs without requiring client libraries to change, which is a much harder problem.

A big reason why CUDA is popular with compilers is that the PTX assembly-ish language is well documented and reasonable. Compilers generate PTX, then the rest of the CUDA infrastructure turns PTX into Turing machine code, or Ampere machine code, or Pascal machine code. In theory, SPIR-V should do the same job, but its just not as usable right now. In the meantime, getting it to work on PTX is easier, and then there's…

PTX and the immense tooling around CUDA.

Khronos mindset of it must be C like and parterns will take care of the ecosystem is what doomed OpenCL.

All of their API design endevours are "design by committe" at its best.

No wonder that SYSCL is now backend agnostic.

Post reply on HN