[flagged]
[dead]
AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
261–270 of 298 posts
Re: AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
#262Earlier quoted context omitted.
And the invention of AGI will have the same impact as the electricity, i mean, they're both inventions, right ? I can't wait for all these billions of new full time jobs coming to replace the current ones. Farmers couldn't be replaced by low skill labor because they are low skill. Farmer didn't stop existing, but we went from 80% of the population farming is to 1-10%. If farming automation had happened in 1800 when 8…
Great points. Besides if one look around even today beyond people who somehow still succeeding at software hustle, there are already tons of people fallen off IT gravy train. A lot of manual testers who would make decent living are now eliminated by automation. Software document writers jobs are kind of gone. Developers are supposed to create document themselves on Confluent wiki etc. A lot of prized SAP consultants…
I feel like this begs for a careful analysis, we could probably drop a dozen more reasons as to why demand in devs and perhaps even in # of lines of code might plateau or decline in the near future, without even mentioning AI at any point.
Re: AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
#263Earlier quoted context omitted.
Remember that GPUs have cache hierarchies and matching block sizes to optimally hit those caches is a big win that you often don't get by default, just because the number of important kernels times important GPUs times effort to properly tune one is greater than what people are willing to do for others for free in open source. Not to mention kernel fusion and API boundaries that socially force suboptimal choices for…
Hey, do you have any suggestions for resources to learn more about this kind of custom optimisation? Sounds interesting, but not sure where to start?
https://ppc.cs.aalto.fi/ covers some of this (overlapping with the topics the person you responded to mentioned, but not covering all, and including some others)
Re: AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
#264Earlier quoted context omitted.
Let's assume that the 46 multiplication algorithm was known, prior to AlphaEvolve re-discovering it. AlphaEvolve still has made an improvement to a performance critical area that has had likely had thousands of engineer-hours put into it. None of those engineers apparently knew about the improved algorithm, or were able to implement the algorithm. This is empirical evidence of an LLM outperforming its (domain expert)…
Isn't this like comparing a human historian to Wikipedia though? Of course the knowledge in Wikipedia will in most cases beat the human. However, that's not the kind of thing we're looking for here.
I don't think that's quite the comparison we're looking for though, because this wasn't just rote data retrieval. It was the recognition of patterns that humans could have noticed given enough time, but had not. It's more like a system that can make inferences as insightful as a thousand skilled human historians typing away at typewriters, armed with the collective knowledge of Wikipedia, in a short period of time.
Re: AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
#265Earlier quoted context omitted.
Great points. Besides if one look around even today beyond people who somehow still succeeding at software hustle, there are already tons of people fallen off IT gravy train. A lot of manual testers who would make decent living are now eliminated by automation. Software document writers jobs are kind of gone. Developers are supposed to create document themselves on Confluent wiki etc. A lot of prized SAP consultants…
I feel like this begs for a careful analysis, we could probably drop a dozen more reasons as to why demand in devs and perhaps even in # of lines of code might plateau or decline in the near future, without even mentioning AI at any point.
I totally agree that there are whole host of reasons for this. One of them I think is plainly a ton of work which was novel and custom in past is now subsumed in base infrastructure of today's world. Despite big tech and other vendor keep pushing of new things lot of people and companies are finding base level acceptable for many tasks which would need custom work in past. So no growing demand custom software for everything.
We can already see for example as per Ericsson report[1] after exponential demand for mobile data and speed it is now plateauing and next decade will not see huge growth. As users are finding things they do and apps they use with phones are now fast/good enough.
1. https://www.ericsson.com/en/reports-and-papers/mobility-repo...
Re: AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
#266> AlphaEvolve achieved up to a 32.5% speedup for the FlashAttention kernel implementation in Transformer-based AI models > In roughly 75% of cases, it rediscovered state-of-the-art solutions, to the best of our knowledge. > And in 20% of cases, AlphaEvolve improved the previously best known solutions These sound like incredible results. I'd be curious what kind of improvements were made / what the improvements were.…
From the paper it was a speedup on the XLA GPU kernel they wrote using Jax, which is probably not SOTA. I don't think Jax even has a official flash attention implementation.
Not sure what “official” means but would direct you to the GCP MaxText [0] framework which is not what this GDM paper is referring to but rather this repo contains various attention implementations in MaxText/layers/attentions.py
Re: AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
#267Earlier quoted context omitted.
Great points. Besides if one look around even today beyond people who somehow still succeeding at software hustle, there are already tons of people fallen off IT gravy train. A lot of manual testers who would make decent living are now eliminated by automation. Software document writers jobs are kind of gone. Developers are supposed to create document themselves on Confluent wiki etc. A lot of prized SAP consultants…
There are two ways to think about your work in IT, one is as a person who really understands how to use a particular technology, and the other is as someone who can figure out how to use technology in general to achieve some particular end. Anybody who picks the first path might have a very well paid career for the few years that that technology is relevant, but will eventually crash out. You need to always be lookin…
I am talking about aggregate distribution not individual motivation to do better. Of course some can do better, learn new things, update LinkedIn profile regularly, attend local networking events, take paid membership to enhance visibility in job search and so on.
The point is more and more people who survived with average skills or one time learning in past are not able to do so now. And it is not a complain about anything or anyone but observation that few people are seeing marked improvement in their lives with changing technology and many more are seeing degradation due to same changes.
Re: AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
#268Earlier quoted context omitted.
Isn't this like comparing a human historian to Wikipedia though? Of course the knowledge in Wikipedia will in most cases beat the human. However, that's not the kind of thing we're looking for here.
I don't think that's quite the comparison we're looking for though, because this wasn't just rote data retrieval. It was the recognition of patterns that humans could have noticed given enough time, but had not. It's more like a system that can make inferences as insightful as a thousand skilled human historians typing away at typewriters, armed with the collective knowledge of Wikipedia, in a short period of time.
[deleted]
Re: AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
#269From the paper, "Notably, for multiplying two 4 × 4 matrices, applying the algorithm of Strassen recursively results in an algorithm with 49 multiplications, which works over any field...AlphaEvolve is the first method to find an algorithm to multiply two 4 × 4 complex-valued matrices using 48 multiplications." If you do naive matrix multiplication, you get a sense that you're doing similar work multiple times, but i…
By googling "4x4 matrices multiplication 48" I ended up on this discussion on math.stackexchange https://math.stackexchange.com/questions/578342/number-of-el... , where in 2019 someone stated "It is possible to multiply two 4×4 matrix A,B with only 48 multiplications.", with a link to a PhD thesis. This might mean that the result was already known (I still have to check the outline of the algorithm).
One of the authors here. We are aware of the Winograd scheme, but note that it only works over commutative rings, which means that it's not applicable recursively to larger matrices (and doesn't correspond to a rank 48 factorization of the matrix multiplication tensor). The MathOverflow answer had a mistake corrected in the comments by Benoit Jacob.
More details: the Winograd scheme computes (x1+ y2 )(x2+ y1 ) + (x3+y4)(x4+y3)-Ai-Bj, and relies on y2y1 (that comes from expanding the first brackets) cancelling with y1y2 in Bj=y1y2 + y3y4. This is fine when working with numbers, but if you want to apply the algorithm recursively to large matrices, on the highest level of recursion you're going to work with 4x4 block matrices (where each block is a big matrix itself), and for matrices Y2Y1 != Y1Y2 (for general matrices).
Here is a website that tracks fastest (recursively applicable) matrix multiplication algorithms for different matrix sizes, and it stands at 49: https://fmm.univ-lille.fr/4x4x4.html
UPD: s/fields/rings/ and fixed equation rendering
Re: AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
#270Why do I get the feeling they are doing the "IBM Watson" thing where different efforts are being put underneath the same brand name? Not saying it is that egregious, but it's a slippery slope from "well, it didn't do all these different things out of the box, unsupervised".
One thing I've really internalized since IBM Watson is that the first reports of any breakthrough will always be the most skeevy. This is because to be amplified it can be either true or exaggerated, and exaggeration is easier. That is to say, if you model the process as a slowly increasing "merit term" plus a random "error term", the first samples that cross a threshold will always have unusually high errors. For th…
IBM Watson is old, before deep learning.
Deep learning systems have exceeded the hype. In 2016 we saw potential with models like AlphaGo Zero but no one could foresee the capability of LLMs (a type of deep learning model).