Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

github.com

Post reply on HN
Print view View original

61–70 of 88 posts

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#61

Post by embedding-shape »

robviren wrote: post #60

I love genetic algorithms and find using LLMs as part of them super compelling. I always find the fitness functions to be the most difficult part. The algorithm naturally tries to exploit any little gap you leave it in cheating. Best part is not needing back propagation in solving a problem. However that is also the worst part in all the solutions just being one level above a random walk. The LLM augmentation really…

> I love genetic algorithms

Is this a "genetic algorithm" though? Besides "select the best performing run", it doesn't seem to have anything to do with crossovers, mutations, etc at all, just "select best", which makes it seem less of a genetic algorithm to me I guess. Might just be me being confused about what counts as an "genetic algorithm" vs not though, I won't claim to be an expert in the field exactly.

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#62

Post by tverbeure »

bsder wrote: post #17

> The frontier is the verifier. Um, yes? The big value that AMD had in the x86 market over competitors was their verification model. This has been known for decades. > 3-seed nextpnr P&R on a Gowin GW2A-LV18 (Tang Nano 20K) — median Fmax × CoreMark iter/cycle = fitness Every single "improvement" is basically about routing around how absolutely abysmally bad the Gowin FPGAs are. Kudos to that, I guess? Gowin FPGAs hav…

Assuming that your claims about GoWin FPGA flaws are correct, isn’t the point of this experiment that it was able to exploit these flaws without manual guidance?

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#63

Post by fesens »

riedel wrote: post #44
amelius wrote: post #43

Earlier quoted context omitted.

I mean, this is such low hanging fruit, you have to be careful not to step on it.

Just because it is a nice meme I want to throw in Schmidhuber's work on (do not treat this comment as serious except you are Schmidhuber himself): * Gödel Machine (2006-2007) [1] * Optimal Ordered Problem Solver (2002) [2] * Meta-Learning and Artificial Curiosity (1990s onward) [3] [1] https://arxiv.org/html/2505.22954v3 [2] https://arxiv.org/abs/cs/0207097 [3] https://evolution.ml/pdf/schmidhuber.pdf Edit: markdown…

Nice references! tks

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#64

Post by fesens »

2ndorderthought wrote: post #49
KronisLV wrote: post #38

Earlier quoted context omitted.

Is it? Evolution also seems to be a result of semi-random crap over the span of millenia and nobody is critiquing it like that. Why should throwing ideas at the wall in regards to optimizing code be any different: as long as you can measure and verify it, are okay with added complexity, and are capable of making the code itself not be crap by the end of it? If an approach is found that improves how well something wor…

It's basically saying to randomly slop something and see if it gets better. Evolution has physical principles and guard rails backing it. Here there are no principals whatsoever, just slopping the slopper to see if it's somehow less sloppy then writing a gist with a slop machine. I wouldn't call it karpathys loop I'd call it slop descent. Or descent into slop. Or something like that

Is slop verifiable? If so we can throw it in the loop... The point is that this loop can be pointed at any verifiable work. Yeah you are seeing it raw, the verifier is the principle you talked about. Yes it was fully AI generated, It will be refined

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#65

Post by fesens »

faangguyindia wrote: post #20
pteetor wrote: post #3

In case you are unfamiliar with Karpathy's Loop[1], it is a genetic algorithm[2] where the genetic "mutations" are clever-but-random ideas generated by an LLM agent, aimed at improving a system. (1) Let the LLM randomly perturbate the system. (2) Measure the system's performance. (3a) If the perturbation improved performance, keep the change. (3b) Otherwise, don't. (4) Repeat [1] https://github.com/karpathy/autoresea…

i actually do it differently > (1) Let the LLM randomly perturbate the system. instead of this i ask LLM to what's least likely to improve performance and then measure it. sometimes big gains come from places you thought are least likely.

For sure! The hypothesis generation gotta be improved. Your take on the "least likely" is interesting. In the beginning of the repo I was having problems with "hypothesis convergence", your idea may be a nice way to introduce the much needed variability

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#66

Post by fesens »

utopiah wrote: post #23
beepdyboop wrote: post #15

Earlier quoted context omitted.

This is not a new idea at all, many many have had it, no one really can claim it

Stigler's law of eponymy https://en.wikipedia.org/wiki/Stigler%27s_law_of_eponymy

I genuinely laughed reading the first words. Yeah, its hard to be novel

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#67

Post by fesens »

vessenes wrote: post #51
fc417fc802 wrote: post #4

Extremely interesting but I don't understand why it was written by an LLM. Either the frontier models are far better than I realized or else writing this document required a lot of manual work regardless at which point why not keep it in your own voice? > The agent did not know that would also halve the LUT count. It found out by doing it and watching the synthesizer. So I guess this is an example of an LLM anthropom…

Yeah I find this current LLM voice very tiring to read; I get enough of it day-to-day wrangling claude and others. I don’t think ‘writing’ this took very much work though, it was probably a “read the research logs, and write a blog post with charts showing our amazing results and hammering on the idea that verifiers matter” as a prompt. The rest you could go have a coffee for. That said, the core idea of this — verif…

Yeah, you are totally right. Its a work in progress, and the post was written by an LLM - Im trying to improve on it (dash pun intended).

Regarding the benchmark overfitting, absolutely, it's pretty much overfitted. This CPU will only be as good as it benchmark. If I have the time I will try to get some applications and optimize for those.

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#69

Post by coredog64 »

embedding-shape wrote: post #48
iterateoften wrote: post #45

Earlier quoted context omitted.

Lol, I respect karpathy a lot, but this is such an obvious in your face idea that it is laughable to put someone’s name on it. What’s next “karpathy investing” where ai in a loop builds a portfolio?

I'd go a step further and say that sort of loop is probably the first thing most people who play around with agent harnesses try, pretty much the first "Hmm, what should I do now?" thing that pops into people's head.

It's less the idea and more the simplicity of it. It's a distillation of something that works and lets newer practitioners get their feet wet before moving on to more complex implementation.

Re: Show HN: Auto-Architecture: Karpathy's Loop, pointed at a CPU

#70

Post by jstanley »

pteetor wrote: post #3

In case you are unfamiliar with Karpathy's Loop[1], it is a genetic algorithm[2] where the genetic "mutations" are clever-but-random ideas generated by an LLM agent, aimed at improving a system. (1) Let the LLM randomly perturbate the system. (2) Measure the system's performance. (3a) If the perturbation improved performance, keep the change. (3b) Otherwise, don't. (4) Repeat [1] https://github.com/karpathy/autoresea…

That's not a genetic algorithm, that's stochastic gradient descent.

To be a genetic algorithm it would need to have mutation (which you have here) and crossover (which you don't).

Post reply on HN