Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

github.com

Post reply on HN
Print view View original

31–40 of 88 posts

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#31

Post by thald »

pteetor wrote: post #3

In case you are unfamiliar with Karpathy's Loop[1], it is a genetic algorithm[2] where the genetic "mutations" are clever-but-random ideas generated by an LLM agent, aimed at improving a system. (1) Let the LLM randomly perturbate the system. (2) Measure the system's performance. (3a) If the perturbation improved performance, keep the change. (3b) Otherwise, don't. (4) Repeat [1] https://github.com/karpathy/autoresea…

I was working some time ago on LLM assisted optimizations and algorithm discovery and this does not look like a novel idea.

AlphaEvolve from google is evolutionary algorithm which uses LLMs for Idea generation following very similar loop:

- https://deepmind.google/blog/alphaevolve-a-gemini-powered-co...

- Open source implementation of the algorithm: https://github.com/algorithmicsuperintelligence/openevolve

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#33

Post by yk09123 »

pteetor wrote: post #3

In case you are unfamiliar with Karpathy's Loop[1], it is a genetic algorithm[2] where the genetic "mutations" are clever-but-random ideas generated by an LLM agent, aimed at improving a system. (1) Let the LLM randomly perturbate the system. (2) Measure the system's performance. (3a) If the perturbation improved performance, keep the change. (3b) Otherwise, don't. (4) Repeat [1] https://github.com/karpathy/autoresea…

This is like idiocracy for Software Devs at this point

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#35

Post by fy20 »

outside1234 wrote: post #6

Has anyone actually written a verifier for a business / project?

I used it (well, a skill based on the same idea) to optimise a prompt that does data extraction from UGC.

However there isn't really a "correct" answer that's easy to define in code (I could manually label a training set, but wanted to avoid that) so I had the LLM just analyse the results itself and decide if they are better or not. It wrote deterministic rules for a few things, but overall it just reviewed the results of each round and decided if the are better or not.

Reviewing the before and after results, I would say yes, it's a big improvement in quality. It also optimised the prompt size to reduce input tokens by 25% and switched to a smaller/cheaper model.

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#36

Post by wiseowise »

beepdyboop wrote: post #15
2001zhaozhao wrote: post #13

Earlier quoted context omitted.

Wtf, this has a name now? I thought of this exact idea literally months ago but never had the time to do any experiments on it. At the time I dismissed it as potentially being incredibly expensive for the improvement you do get, and runs into typical pitfalls of evolutionary algorithms (in the same way evolution doesn't let an organism grow a wheel, your LLM evolution algorithm will never come up with something that…

This is not a new idea at all, many many have had it, no one really can claim it

Don’t worry, Twitter bros already coined it.

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#37

Post by bodegajed »

pteetor wrote: post #3

In case you are unfamiliar with Karpathy's Loop[1], it is a genetic algorithm[2] where the genetic "mutations" are clever-but-random ideas generated by an LLM agent, aimed at improving a system. (1) Let the LLM randomly perturbate the system. (2) Measure the system's performance. (3a) If the perturbation improved performance, keep the change. (3b) Otherwise, don't. (4) Repeat [1] https://github.com/karpathy/autoresea…

thanks, I thought as a researcher Kaparthy would include and cite relevant papers. I quickly became disappointed. I already knew openevolve and the ACE Framework paper. This is the first time I learned about Genetic Algorithm and I now have some clear roadmap for studying.

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#38

Post by KronisLV »

yk09123 wrote: post #33
pteetor wrote: post #3

In case you are unfamiliar with Karpathy's Loop[1], it is a genetic algorithm[2] where the genetic "mutations" are clever-but-random ideas generated by an LLM agent, aimed at improving a system. (1) Let the LLM randomly perturbate the system. (2) Measure the system's performance. (3a) If the perturbation improved performance, keep the change. (3b) Otherwise, don't. (4) Repeat [1] https://github.com/karpathy/autoresea…

This is like idiocracy for Software Devs at this point

Is it? Evolution also seems to be a result of semi-random crap over the span of millenia and nobody is critiquing it like that.

Why should throwing ideas at the wall in regards to optimizing code be any different: as long as you can measure and verify it, are okay with added complexity, and are capable of making the code itself not be crap by the end of it?

If an approach is found that improves how well something works, you can even treat the AI slop as a draft and iterate upon it yourself further.

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#39

Post by [deleted] »

thin_carapace wrote: post #5

> "If you can write the rules down, an agent will satisfy them faster than your team will." a fantastic opportunity to become the next next big thing and write a verifier verifier. at the hypothesized inflexion point where AI instantly performs exactly as commanded, what happens to heavily regulated industries like medical? do we get huge leaps and bounds everywhere EXCEPT where it matters, or is regulation going to…

[deleted]
Post reply on HN