Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

github.com

Post reply on HN
Print view View original

81–88 of 88 posts

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#81

Post by ks2048 »

jstanley wrote: post #70
pteetor wrote: post #3

In case you are unfamiliar with Karpathy's Loop[1], it is a genetic algorithm[2] where the genetic "mutations" are clever-but-random ideas generated by an LLM agent, aimed at improving a system. (1) Let the LLM randomly perturbate the system. (2) Measure the system's performance. (3a) If the perturbation improved performance, keep the change. (3b) Otherwise, don't. (4) Repeat [1] https://github.com/karpathy/autoresea…

That's not a genetic algorithm, that's stochastic gradient descent. To be a genetic algorithm it would need to have mutation (which you have here) and crossover (which you don't).

I agree it's not a genetic algorithm, but, it's also not stochastic gradient descent. There is no gradient. The "step direction" (code modification) is chosen by an LLM, which is "smart enough" to guess something that might be an improvement.

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#82

Post by 8note »

embedding-shape wrote: post #61
robviren wrote: post #60

I love genetic algorithms and find using LLMs as part of them super compelling. I always find the fitness functions to be the most difficult part. The algorithm naturally tries to exploit any little gap you leave it in cheating. Best part is not needing back propagation in solving a problem. However that is also the worst part in all the solutions just being one level above a random walk. The LLM augmentation really…

> I love genetic algorithms Is this a "genetic algorithm" though? Besides "select the best performing run", it doesn't seem to have anything to do with crossovers, mutations, etc at all, just "select best", which makes it seem less of a genetic algorithm to me I guess. Might just be me being confused about what counts as an "genetic algorithm" vs not though, I won't claim to be an expert in the field exactly.

i think there's a lack of verbiage for "optimization that doesnt involve gradient descent"

i think there might be a conflation of "problems a genetic algorithm could be used for" vs "using the genetic algorithm to solve the problem"

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#83

Post by Legend2440 »

pteetor wrote: post #3

In case you are unfamiliar with Karpathy's Loop[1], it is a genetic algorithm[2] where the genetic "mutations" are clever-but-random ideas generated by an LLM agent, aimed at improving a system. (1) Let the LLM randomly perturbate the system. (2) Measure the system's performance. (3a) If the perturbation improved performance, keep the change. (3b) Otherwise, don't. (4) Repeat [1] https://github.com/karpathy/autoresea…

You're missing a step. The perturbations are not fully random. The LLM also looks at the result and tries to do credit assignment to determine what changes to try in the next round.

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#84

Post by bishopp92 »

pteetor wrote: post #3

In case you are unfamiliar with Karpathy's Loop[1], it is a genetic algorithm[2] where the genetic "mutations" are clever-but-random ideas generated by an LLM agent, aimed at improving a system. (1) Let the LLM randomly perturbate the system. (2) Measure the system's performance. (3a) If the perturbation improved performance, keep the change. (3b) Otherwise, don't. (4) Repeat [1] https://github.com/karpathy/autoresea…

It is rather a variation of hill climbing [1]. As others pointed out evolutionary algorithms employ a richer set of search operators.

[1] https://en.wikipedia.org/wiki/Hill_climbing

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#85

Post by bsder »

vessenes wrote: post #52
bsder wrote: post #17

> The frontier is the verifier. Um, yes? The big value that AMD had in the x86 market over competitors was their verification model. This has been known for decades. > 3-seed nextpnr P&R on a Gowin GW2A-LV18 (Tang Nano 20K) — median Fmax × CoreMark iter/cycle = fitness Every single "improvement" is basically about routing around how absolutely abysmally bad the Gowin FPGAs are. Kudos to that, I guess? Gowin FPGAs hav…

Amazing comment. As a non-hardware guy, I read, “well, duh, for a 20yr practitioner dealing with the intricacies of specific FPGA series, all this makes tons of sense”.

It only makes sense to me because I tried to implement a RISC-V on these Gowin FPGAs and banged into the limitations and can distill them down. A junior engineer looks at this post-AI, shrugs, and says "I'm done."

The AI doesn't flag "Hey, my adder sucks. Move to a better FPGA architecture." A junior engineer pre-AI would have to bang on this a while, get frustrated at the critical paths, and eventually ask for help. At which point we would both look at this, identify that the adder was doing a 32-bit ripple carry, both have a "WTF?!" moment, and switch FPGA families.

In addition, the AI also doesn't flag how close to the margin you are. To my eye, almost all the Fmax gains look like PnR (place and route) noise. The DIV/REM obviously isn't and the replay predictor looks real. To top it off, the branch predictor wins look anomalously low to my eye.

This is what a bunch of us are yelling about with AI. AI gets you a thing. AI gets you no insight into that thing. And because the juniors will use the AI, they will never learn the insight.

Side note: The granularity of the CM/MHz numbers look a bit suspicious. Why are there identical entries?

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#86

Post by bsder »

fesens wrote: post #54
bsder wrote: post #17

> The frontier is the verifier. Um, yes? The big value that AMD had in the x86 market over competitors was their verification model. This has been known for decades. > 3-seed nextpnr P&R on a Gowin GW2A-LV18 (Tang Nano 20K) — median Fmax × CoreMark iter/cycle = fitness Every single "improvement" is basically about routing around how absolutely abysmally bad the Gowin FPGAs are. Kudos to that, I guess? Gowin FPGAs hav…

"I would LOVE somebody to bounce AI off of reversing the architecture and bitstreams for the stupid-ass closed-source FPGAs." The only reason I'm using Gowin is because it has a slightly more mature opensource tooling. Maybe we can apply this loop to nextpnr also

Please apply this loop to nextpnr for any of the commodity Xilinx, Altera, or Lattice parts. For example, everything about Lattice has been stuck for almost a decade at this point.

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#87

Post by mikewarot »

I'm interesting in replicating the experiment, but I can't seem to find the actual hardware specifications. I assume this is something I could pick up from Digikey, Mouser, or Amazon.

I have variations on Karpathy's loop that I'd like to try out with real world hardware.

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#88

Post by cadamsdotcom »

fesens wrote: post #72
sho_hn wrote: post #2

Salient on the value of the verifier. Matches my experience in the last two quarters. Nice detail on the encountered failures. Very similar experiences with my own loops against testsuites. Great post. A snapshot in time.

Tks! Did you apply it to hardware design or to another field?

It applies to every field, to the extent that it’s possible to provide a safe space for iteration, define success, & validate improvement.

Iteratively improvable problems are going to get attacked with this approach and iterated right to the limit of what we can define and hill-climb towards.

That’s a very very good thing - it’ll get us to the frontier where the next leaps only come from ingenuity.

Post reply on HN