81–90 of 95 posts
Earlier quoted context omitted.
It is a computationally clever application of the chain rule to minimize the amount of computation needed to compute gradients for all parameters in the network.
> to minimize the amount of computation IMO backprop is the most trivial implementation of differentiation in neural networks. Do you know an easier way to compute gradients with larger overhead? If so, please share it.
Earlier quoted context omitted.
It is a computationally clever application of the chain rule to minimize the amount of computation needed to compute gradients for all parameters in the network.
> to minimize the amount of computation IMO backprop is the most trivial implementation of differentiation in neural networks. Do you know an easier way to compute gradients with larger overhead? If so, please share it.
Earlier quoted context omitted.
These kind of Genetic Algorithms are still being researched in academia. I attended a seminar a couple of years ago on the subject. It’s still a total dead end imho.
I used to (early 00s) be super big into GA and GP until a professor of mine at the time described the whole class of algorithms as "Marginally better than brute forcing". That really resonated with my experience, and was just too spot-on to ignore.
I wonder how this would apply to compiler/interpreter optimizations. Is it possible that it can "disect" some parts of the execution, perhaps at assembly level, and come up with optimizations specific to the compiled code without changing the output (I mean expected program output, not emitted binary), that modern compilers have not deterministically come up with?
I expect the answer is "no". I wouldn't expect a tool like this to "discover" assembly without being trained on the compiled output. The model has no notion of how or where the code runs. After decades of compiler research and super compilers chugging away, we're sort of at a point where discovering novel optimizations with results that are more than a smidge of improvement is almost impossibly unlikely. Compilers to…
I agree when it comes to peephole optimizations, but there's still a lot of juice left in exploiting language guarantees (immutability, non-aliasing, data-parallelism), however most compiler developer energy is spent propping up C/C++ and consequently optimizations are developed with those languages in mind.
It's funny, this kind of subtree mutation was looked at pretty deeply by Koza and Adamı in the 90s under the rubric of Genetic Algorithms, but with a slightly different optimization function One ref in the paper to 2000 for GAs for fast generation of program trees, but that's missing the main show Hope they're reading this and dig into those guys work
You can also say backpropagation is the chain rule from centuries ago.
I went digging in wikipedia.. the Backpropagation article was created in 2005 and yet the mention of association/derivation from the chain rule wasn't mentioned until 2014, through a borrow from the German article
https://en.wikipedia.org/w/index.php?title=Backpropagation&o...
It's funny, this kind of subtree mutation was looked at pretty deeply by Koza and Adamı in the 90s under the rubric of Genetic Algorithms, but with a slightly different optimization function One ref in the paper to 2000 for GAs for fast generation of program trees, but that's missing the main show Hope they're reading this and dig into those guys work
Whoops!
It's funny, this kind of subtree mutation was looked at pretty deeply by Koza and Adamı in the 90s under the rubric of Genetic Algorithms, but with a slightly different optimization function One ref in the paper to 2000 for GAs for fast generation of program trees, but that's missing the main show Hope they're reading this and dig into those guys work
Some more recent alternatives to Koza's GP use some very different search mechanisms. FFX & PGE are both very fast. https://seminars.math.binghamton.edu/ComboSem/worm-chiu.pge_... https://arxiv.org/pdf/2209.09675 I authored PGE and have thought that RL, and more recently diffusion techniques, might help these algos. All of the algos need better ways to guide the search, or help it get unstuck from local optima, which…
I wonder how this would apply to compiler/interpreter optimizations. Is it possible that it can "disect" some parts of the execution, perhaps at assembly level, and come up with optimizations specific to the compiled code without changing the output (I mean expected program output, not emitted binary), that modern compilers have not deterministically come up with?
There are people applying synthesis techniques to superoptimization. So something like this would possibly apply.
Could anyone clarify how they integrate beam search with the reverse diffusion- do they sample m > k nodes from a reverse diffusion step and expand only the top k nodes?