Compiling models to megakernels
blog.luminal.com
Compiling models to megakernels
1–10 of 20 posts
Re: Compiling models to megakernels
#2There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.
Re: Compiling models to megakernels
#3There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.
Dead code elimination is already a technique in AI when someone takes an MoE model and removes an unused "E" from it.
Re: Compiling models to megakernels
#4There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.
AI actually has some optimizations unique to the field. You can in fact optimize a model to make it work; not a lot of other disciplines put as much emphasis on this as AI
Re: Compiling models to megakernels
#5There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.
AI actually has some optimizations unique to the field. You can in fact optimize a model to make it work; not a lot of other disciplines put as much emphasis on this as AI
Can you list these optimizations?
Re: Compiling models to megakernels
#6Re: Compiling models to megakernels
#7There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.
Which categories do algorithmic optimizations fall under? For example:
Strassen algorithm for matrix multiplication https://en.wikipedia.org/wiki/Strassen_algorithm
FFT convolution https://dsp.stackexchange.com/a/63211
Winograd convolution https://www.cv-foundation.org/openaccess/content_cvpr_2016/p...
And of course optimization algorithms themselves.
Re: Compiling models to megakernels
#8There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.
Which categories do algorithmic optimizations fall under? For example: Strassen algorithm for matrix multiplication https://en.wikipedia.org/wiki/Strassen_algorithm FFT convolution https://dsp.stackexchange.com/a/63211 Winograd convolution https://www.cv-foundation.org/openaccess/content_cvpr_2016/p... And of course optimization algorithms themselves.
Partial evaluation on the symbolic structure of the problem.
Re: Compiling models to megakernels
#9Re: Compiling models to megakernels
#10There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.
Your list is so short it doesn't even include the basics such as reordering operations.
It also feels incredibly snarky to say "they knew about caching" and that they will get to partial evaluation and dead code elimination, when those seem to be particularly useless (beyond what the CUDA compiler itself does) when it comes to writing GPU kernels or doing machine learning in general.
You can't do any partial evaluation of a neural network because the activation functions are interrupting the multiplication of tensors. If you remove the activation function, then you end up with two linear layers that are equivalent to one linear layer, defeating the point of the idea. You could have trained a network with a single layer instead and achieved the same accuracy with a corresponding shorter training/inference time.
Dead code elimination is even more useless since most kernels are special purpose to begin with and you can't remove tensors without altering the architecture. Instead of adding useless tensors only to remove them, you could have simply used a better architecture.