Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

blog.skypilot.co

Post reply on HN
Print view View original

31–40 of 126 posts

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#31

Post by pbkhrv »

> How parallelism changed the agent’s research strategy > With a single GPU, the agent is stuck doing greedy hill-climbing: try one thing, check the result, pick a direction, try the next thing. With 16 GPUs, the strategy shifts. ...skip... 12 experiments in a single 5-minute wave. This makes it much harder to get stuck in local optima and much easier to find interaction effects between parameters.

The agent can theoretically come up with a protocol to run those same 12 experiments one-by-one and only then decide which branch to explore next - which I think would lead to the same outcome?

But in this case, it just happened to have stumbled on this particular outcome only because it didn't get a chance to execute a greedy strategy after the first 1 or 2 results.

Worse experiment design + parallelism = better experiment design + serialized execution ?

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#32

Post by rogerrogerr »

fdghrtbrt wrote: post #26
rogerrogerr wrote: post #7

Earlier quoted context omitted.

Why do we think this emerged “on its own”? Surely this technique has been discussed in research papers that are in the training set.

Why surely? Have you never seen an LLM try something new?

Is your assertion that no one has ever written "we tried some stuff on the small inexpensive platform first, then moved to the bigger more expensive platform with the more promising options" in a research paper or literally anywhere else?

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#34

Post by password54321 »

saberience wrote: post #24

Wait, "Karpathy's Autoresearch", you mean a loop that prompts the agent to improve a thing given a benchmark? People have been doing this for a year or more, Ralph loops etc. I hate the weird strange Twitter world of hero-worship for folks that seems to arise just out of large followings. Joe no-followers does this six months ago, nobody cares. Karpathy writes a really basic loop and it's now a kind of AI miracle pro…

The number of refurbished mac minis that are available in my country has suddenly dramatically increased ever since the Clawdbot tweet. People never learn.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#35

Post by fdghrtbrt »

rogerrogerr wrote: post #32
fdghrtbrt wrote: post #26

Earlier quoted context omitted.

Why surely? Have you never seen an LLM try something new?

Is your assertion that no one has ever written "we tried some stuff on the small inexpensive platform first, then moved to the bigger more expensive platform with the more promising options" in a research paper or literally anywhere else?

No, that's not my assertion. In fact I asserted nothing at all.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#37

Post by rogerrogerr »

fdghrtbrt wrote: post #35
rogerrogerr wrote: post #32

Earlier quoted context omitted.

Is your assertion that no one has ever written "we tried some stuff on the small inexpensive platform first, then moved to the bigger more expensive platform with the more promising options" in a research paper or literally anywhere else?

No, that's not my assertion. In fact I asserted nothing at all.

You're speaking in riddles; your communication would be more effective if you didn't do that.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#39

Post by fdghrtbrt »

rogerrogerr wrote: post #37
fdghrtbrt wrote: post #35

Earlier quoted context omitted.

No, that's not my assertion. In fact I asserted nothing at all.

You're speaking in riddles; your communication would be more effective if you didn't do that.

You said "surely", and I asked:

> Why surely? Have you never seen an LLM try something new?

I'm afraid I can't make it any simpler than this.

And I still don't know the answer to how you're so sure. To me there's several explanations, and it seems to you there's only one.

I'm pretty happy with my communication style.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#40

Post by caconym_ »

fdghrtbrt wrote: post #26
rogerrogerr wrote: post #7

Earlier quoted context omitted.

Why do we think this emerged “on its own”? Surely this technique has been discussed in research papers that are in the training set.

Why surely? Have you never seen an LLM try something new?

I honestly don't think I have.

In this case, using a cheap(er) signal or heuristic as an initial filter before spending more resources on cases that pass the filter is a pattern that shows up all over the place, and LLMs are good at picking up on patterns like that and generalizing them. AFAICT.

Post reply on HN