11–20 of 20 posts
Earlier quoted context omitted.
RLHF is one that comes to mind
Well, this is an entirely other category of optimizations - not program performance but model performance.
Earlier quoted context omitted.
Well, this is an entirely other category of optimizations - not program performance but model performance.
Yes, in "runtime optimization" the model is just a computation graph so we can use a lot of well known tricks from compilation like dead code elimination and co..
What other optimizations are there that can be used than what explicitly falls into the 4 categories that the top commenter here listed out?
There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.
Which categories do algorithmic optimizations fall under? For example: Strassen algorithm for matrix multiplication https://en.wikipedia.org/wiki/Strassen_algorithm FFT convolution https://dsp.stackexchange.com/a/63211 Winograd convolution https://www.cv-foundation.org/openaccess/content_cvpr_2016/p... And of course optimization algorithms themselves.
There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.
Your list is so short it doesn't even include the basics such as reordering operations. It also feels incredibly snarky to say "they knew about caching" and that they will get to partial evaluation and dead code elimination, when those seem to be particularly useless (beyond what the CUDA compiler itself does) when it comes to writing GPU kernels or doing machine learning in general. You can't do any partial evaluati…
I'm not enough of an expert to see if there's any actualy merit to this idea, and if you can skip evaluating huge parts of the network and keeping track of such evaluations, is actually worth it, but it intuitively makes sense to me that making an omelette has nothing to do with the Battle of Hastings, so when making a query about the former, the neurons encoding the latter might not affect the output.
Afaik, there's already research into finding which network weight encode which concepts.
MOE is a somewhat cruder version of this technique.
Earlier quoted context omitted.
Yes, in "runtime optimization" the model is just a computation graph so we can use a lot of well known tricks from compilation like dead code elimination and co..
We are getting closer! What other optimizations are there that can be used than what explicitly falls into the 4 categories that the top commenter here listed out?
There is more to it, mentioned 4 categories are not the only ones, they are not even broad categories.
If somebody likes broad categories here is good one: "1s and 0s" and you can compute anything you want, there you go – single category for everything. Is it meaningful? Not really.
There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.
In this case are you're strickly fusing pre defined kernels or are you also optimizing them? Is this complimentary to your earlier work on search-based compilers?
Earlier quoted context omitted.
We are getting closer! What other optimizations are there that can be used than what explicitly falls into the 4 categories that the top commenter here listed out?
For inference assorted categories may include vectorization, register allocation, scheduling, lock elision, better algos, changing complexity, better data structures, profile guided specialization, layout/alignment changes, compression, quantization/mixed precision, fused kernels (goes beyond inlining), low rank adapters, sparsity, speculative decoding, parallel/multi token decoding, better sampling, prefill/decode s…
So if I'm understanding correctly, you decompose kernels into their per_sm_workload, then you figure out per_sm_data_dependency and then you can schedule sm_workloads from the next kernel to start running as soon as the data dependency is satisfied, not needing to wait for the other sms from the previous kernel to finish. In this case are you're strickly fusing pre defined kernels or are you also optimizing them? Is…
In other words, very complimentary to our search-based approach.
There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.