Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

blog.skypilot.co

Post reply on HN
Print view View original

91–100 of 126 posts

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#91

Post by augment_me »

Isn't this apples to pears comparison? It's really just saying that having a bigger credit card gets you shit faster, but it's actually worse in terms of GPU utilization and efficiency.

1) The total amount of time is not the same if you just count GPU-hours. If you have 16 GPUs, it makes sense to run them for 4.5 hours to get to 72h for an even comparison, not 8.

2) If we stop at 4.5 hours(and are generous including the big drop), the loss is about 0.978, which is the same as about 44 hours with the sequential solution, making the sequential solution about twice as efficient.

So the real conclusion here is that we are able to run things in parallel at an efficiency loss but at a time win as long as we have access to more hardware. I feel like the blog oversells itself.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#92

Post by nurettin »

nfg wrote: post #47
emp17344 wrote: post #45

Earlier quoted context omitted.

Why should we care that he’s famous?

Fame doesn’t enter it - the point is Karpathy has about as strong a claim as anyone to having “actually used LLMs for non trivial tasks”.

That is not the case at all, considering that he himself started using and tweeting about llms for coding fairly recently. He's probably less experienced in that area than most people who started using claude cli last year.

He is a researcher who understands neural networks and their architectures exceptionally well. That is all.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#93

Post by suddenlybananas »

GorbachevyChase wrote: post #82
rogerrogerr wrote: post #7

Earlier quoted context omitted.

Why do we think this emerged “on its own”? Surely this technique has been discussed in research papers that are in the training set.

You probably express very few truly original ideas. Let’s not set the bar quite so high unless we are all just a sad simulacrum of “pure” thought.

But humans are capable of very many original ideas. Look around you, humans were able to remake the entire world because of these original thoughts.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#94

Post by OJFord »

nfg wrote: post #27
saberience wrote: post #25

Earlier quoted context omitted.

Have you actually used LLMs for non trivial tasks? They are still incredibly bad when it comes to actually hard engineering work and they still lie all the time, it's just gotten harder to notice, especially if you're just letting it run all night and generate reams of crap. Most people are optimizing for terrible benchmarks and then don't really understand what the model did anyone and just assume it did something g…

Do you realise who you’re replying to?

To be fair Karpathy isn't known for using LLMs—not that I would assume or question whether he's used them 'for non-trivial tasks', but it's not like making the same comment in reply to Steve Yegge or someone. (However trivial we may think Gastown/Wasteland is in the other sense!)

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#95

Post by dag100 »

misiti3780 wrote: post #63
password54321 wrote: post #34

Earlier quoted context omitted.

The number of refurbished mac minis that are available in my country has suddenly dramatically increased ever since the Clawdbot tweet. People never learn.

increased or decreased?

"increased" implies that people bought brand-new Mac Minis to run ClawdBot on, got bored of it, and then sold them back to be refurbished and resold.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#97

Post by claud_ia »

herf wrote: post #44

This "early velocity only" approach seems like a problem - how do you know with 5-minute training runs that you aren't affecting the overall asymptote? e.g., what if the AI picks a quantizer that happens to be faster in the first five minutes, but has a big noise floor where it can't make more progress?

[dead]

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#100

Post by touristtam »

kraddypatties wrote: post #20
karpathy wrote: post #15

Earlier quoted context omitted.

Wrong and short-sighted take given that the LLM explores serially learning along the way, and can tool use and change code arbitrarily. It seems to currently default to something resembling hyperparameter tuning in absence of more specific instructions. I briefly considered calling the project “autotune” at first but I think “autoresearch” will prove to be the significantly more appropriate name.

I can believe that in the long run. Does the agent have access to arxiv (a brief skim of the README didn't have an answer)? If not, it could be that the current approach of relying on the model's weights only is resulting in the perceived local optimum of hyperparameter tuning. Anecdotally, we built a little MCP for arxiv to help with our internal research, noticed a significant boost in the diversity of methods (arc…

care to share?
Post reply on HN