Does Triton support automatic differentiation? I don't see that feature in a quick poke through the docs. If it does compile to LLVM, I suppose it can use Enzyme https://enzyme.mit.edu/
This sits BELOW the automatic differentiation layer.
Triton: Open-Source GPU Programming for Neural Networks
91–100 of 116 posts
Re: Triton: Open-Source GPU Programming for Neural Networks
#92Earlier quoted context omitted.
Sounds similar to business logic shared between and client and server web app. Thanks I just wonder if this alone is enough to make a unified programming environment the preferred way. On the web we had Meteor.js which tried this approach to unify client and server with javascript. The shared code was of similar types. Meteor never became the preferred way to write web apps. Im wondering if the same is true for GPU +…
This analogy is insufficient to capture what is going on. In web dev, the client and server are doing fundamentally different things (eg UX vs data storage) and while sharing business logic is interesting it doesn't solve a huge problem. CPU vs GPU is vaugly analogous to different web browsers on the client in the old days when they were very different. They have different performance characteristics, so shims worked…
I don’t think this is accurate. CPU and GPU are not trying to execute the same code with different performance characteristics. GPU and CPU are fundamentally doings different things. In fact the analogy for shared business logic works perfectly
Re: Triton: Open-Source GPU Programming for Neural Networks
#93A toy illustrative example, summing two arrays:
CUDA
c[i] = a[i] + b[i]
i += 1
Triton
c[i:i+16] = a[i:i+16] + b[i:i+16]
i += 16
A very exciting result of the incredible performance that Triton achieves, is the ability to fuse NN operations such as Matrix Multiply + LeakyReLU + Batch Norm. Previously, you needed to rely on cuBLAS for fast hand-written Matrix Multiply kernels, and then your LeakyReLU would need to read that result out of memory, and then your Batch Norm would read the LeakyReLU out of memory again.
The ability to write very fast kernels, and especially being able to fuse them together, to avoid unnecessary memory round-trips is a big deal!
Re: Triton: Open-Source GPU Programming for Neural Networks
#94Earlier quoted context omitted.
That's http://www.eecs.harvard.edu/~htk/publication/2019-mapl-tille... for those of us outside the paywall
Huh, I thought the ACM DL was open "now" during the pandemic. Apologies!
That was unfortunately short-lived. ACM announced on March 30, 2020 that they would open their DL for 90 days due to the pandemic [1]. I don't believe there was an extension, so it expired on June 30, 2020.
[1] https://www.acm.org/articles/bulletins/2020/march/dl-access-...
Re: Triton: Open-Source GPU Programming for Neural Networks
#95I have found writing CUDA code is much simpler than writing correct multi-threaded AVX2/AVX-512 code.
Use a domain-specific compiler to generate custom, stand-alone, massively multi-threaded AVX-512 inference C code: https://NN-512.com The generated code is easily twice as fast as TensorFlow's AVX-512 kernels (Intel's oneAPI).
Some user feedback: I tried to get the ResNet50 example to work, but I gave up after 2 hours. There really should be an end-to-end example like "./resnet50_example.py monkey.jpg".
A few points where I struggled:
- What are the names in the ResNet50Params struct? I looked at a few popular ResNet50 implementations, but I could not find any correspondence and since the names have been sorted, the order of the members might as well be random. I thought about matching the parameters by shape instead, but the chance of getting that correct is almost zero. Why no natural sort? Or even better, a naming scheme that sorts in the same order as execution order, for example with zero-prefixed numbers like "layer05"
- 6 MB C files are somewhat ridiculous. There should at least be a download link somewhere. Copy & pasting from the website would take forever because of scrolling. Also it is impossible to verify that that code monstrosity is not doing anything evil.
- Go seems like an odd choice when almost every machine learning project these days is developed in Python.
- GCC throws a few hundred warnings:
ResNet50.c: In function ‘ResNet50ThreeArrangeDats7Callee1’:
ResNet50.c:57009:15: warning: unused variable ‘rel27’ [-Wunused-variable]
57009 | ptrdiff_t rel27 = j62-0;
| ^~~~~
ResNet50.c: In function ‘ResNet50NetCreate’:
ResNet50.c:77852:20: warning: taking address of packed member of ‘struct ResNet50Params’ may result in an unaligned pointer value [-Waddress-of-packed-member]
77852 | params1->bn1Means,
| ~~~~~~~^~~~~~~~~~
- EDIT: I just realized I could probably just write a parser for that graph file format. This would be much easier if it was something standard like JSON instead.
- EDIT2: I think some shapes in the graph file definition are incorrect (possibly reversed per block?). For example, tensor "one1" should have "ToChannels=64" instead of "ToChannels=256".
Re: Triton: Open-Source GPU Programming for Neural Networks
#96Unfortunate name clash with NVIDIAs Triton Inference Server: https://developer.nvidia.com/nvidia-triton-inference-server
Maybe name it Trinitron. That name is not used so much anymore :)
Re: Triton: Open-Source GPU Programming for Neural Networks
#97Folks might find the author’s research paper [1] while at Harvard more informative. This is a great high-level description, but if you want more detail, I recommend the paper. [1] https://dl.acm.org/doi/abs/10.1145/3315508.3329973
Note, that there is a new (substantially accelerated) "autoscheduler" implementation in TVM (one of competitors in the article linked), details can be found in https://arxiv.org/abs/2006.06762
Re: Triton: Open-Source GPU Programming for Neural Networks
#98A toy illustrative example, summing two arrays: CUDA c[i] = a[i] + b[i] i += 1 Triton c[i:i+16] = a[i:i+16] + b[i:i+16] i += 16 The 16 in this example is the "block size", and could be anything. But this notion of expressing computation over blocks of dense data seems to be the big difference from other approaches. A very exciting result of the incredible performance that Triton achieves, is the ability to fuse NN op…
> Previously, you needed to rely on cuBLAS for fast hand-written Matrix Multiply kernels, and then your LeakyReLU would need to read that result out of memory,
You could do that, but you can also just tell cuBLAS to fuse ReLU, by just passing the "CUBLASLT_EPILOGUE_RELU" option (among others), see the manual: https://docs.nvidia.com/cuda/cublas/index.html#cublasLtEpilo...
This has been possible for years. It's the kind of 1 line change that makes a big difference.
Re: Triton: Open-Source GPU Programming for Neural Networks
#99Earlier quoted context omitted.
The author commented on reddit about that ( https://www.reddit.com/r/MachineLearning/comments/otdpkx/n_i... ) > PS: The name Triton was coined in mid-2019 when I released my PhD paper on the subject ( http://www.eecs.harvard.edu/~htk/publication/2019-mapl-tille... ). I chose not to rename the project when the Triton inference server came out a year later since it's the only thing that ties my helpful PhD advisors to…
The author is unfortunately wrong. NVIDIA's Triton was referenced in marketing material as far back as 2018. https://developer.nvidia.com/blog/nvidia-serves-deep-learnin...
Are we sure that marketing article was not changed after the fact? The Nvidia Triton Release notes show 'Starting in release 20.03, TensorRT Inference Server is now called Triton Inference Server.' [0]
It kind of looks like the article just changed the heading later since in the article the docker images they refer to etc. are all still called 'tensorrtserver' which was the project's name before they changed naming in Release 20.03.
The OpenAI-Triton Author says on reddit: 'You can also look at the github history and you will see that there is no mention of the "Triton inference server" up until version 2.0, which wasn't out in 2019 (I ran `git reset --hard v1.9.0 ; grep -ir "triton" .`)' [1]
- [0]: https://docs.nvidia.com/deeplearning/triton-inference-server...
- [1]: https://old.reddit.com/r/MachineLearning/comments/otdpkx/n_i...
Re: Triton: Open-Source GPU Programming for Neural Networks
#100Too bad it's CUDA Sooner or later this will become a problem because you are depending on the benevolence of a single manufacturer.
Seems like CUDA only is only at the moment.
From their github:
>Under development: AMD GPUs, CPUs