Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

blog.skypilot.co

Post reply on HN
Print view View original

61–70 of 126 posts

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#61

Post by wenc »

kraddypatties wrote: post #2

I feel like most of this recent Autoresearch trend boils down to reinventing hyper-parameter tuning. Is the SOTA still Bayesian optimization when given a small cluster? It was ~3 years ago when I was doing this kind of work, haven't kept up since then. Also, shoutout SkyPilot! It's been a huge help for going multi-cloud with our training and inference jobs (getting GPUs is still a nightmare...)!

I wonder if it's more like "qualitative gradient descent" on a very non-linear non-convex surface.

You can try this yourself in a simple fashion -- let's say you have piece of code that you want to speed up. Point your agent to a code profiler (your oracle -- typically your Python profiler) and tell it speed up the code. I've tried it. It works.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#62

Post by ericd »

CamperBob2 wrote: post #59
ericd wrote: post #56

Earlier quoted context omitted.

Shades of https://news.ycombinator.com/item?id=35079

That whole thread is just amazing, if you back up a couple of levels from ground zero. Great perspectives from a lot of thoughtful posters. E.g., you can see a post from a user named dhouston, who mentioned that he was thinking about starting an online file sync/backup service of some sort.

Haha awesome. I guess they were going through YC right then, I still remember their launch video from around then and thinking it was one of the best ads I’d ever seen.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#63

Post by misiti3780 »

password54321 wrote: post #34
saberience wrote: post #24

Wait, "Karpathy's Autoresearch", you mean a loop that prompts the agent to improve a thing given a benchmark? People have been doing this for a year or more, Ralph loops etc. I hate the weird strange Twitter world of hero-worship for folks that seems to arise just out of large followings. Joe no-followers does this six months ago, nobody cares. Karpathy writes a really basic loop and it's now a kind of AI miracle pro…

The number of refurbished mac minis that are available in my country has suddenly dramatically increased ever since the Clawdbot tweet. People never learn.

increased or decreased?

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#64

Post by whiplash451 »

rfw300 wrote: post #60
karpathy wrote: post #41

Earlier quoted context omitted.

The most recent round of autoresearch (round 2) which decreased "time to GPT-2" from 1.8 hours to 1.65 hours had some examples. I adjusted the program.md to "look at modded nanogpt project and draw inspirations from there for things to try" and it came back with a bunch of tuning, but also tried and implemented new architecture changes, some of which actually helped including the smear gate and the backout skip conne…

Do you have a sense of whether these validation loss improvements are leading to generalized performance uplifts? From afar I can't tell whether these are broadly useful new ideas or just industrialized overfitting on a particular (model, dataset, hardware) tuple.

Why set the bar higher on generalization for autoresearch vs the research humans generally do?

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#65

Post by rfw300 »

pbkhrv wrote: post #31

> How parallelism changed the agent’s research strategy > With a single GPU, the agent is stuck doing greedy hill-climbing: try one thing, check the result, pick a direction, try the next thing. With 16 GPUs, the strategy shifts. ...skip... 12 experiments in a single 5-minute wave. This makes it much harder to get stuck in local optima and much easier to find interaction effects between parameters. The agent can theo…

Yeah, assuming there's no active monitoring during the training runs, you can trivially give the agent an abstraction which turns "1 GPU" into "16 GPUs" that just so happens to take 16x the wall-clock time to run.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#67

Post by rfw300 »

rfw300 wrote: post #65
pbkhrv wrote: post #31

> How parallelism changed the agent’s research strategy > With a single GPU, the agent is stuck doing greedy hill-climbing: try one thing, check the result, pick a direction, try the next thing. With 16 GPUs, the strategy shifts. ...skip... 12 experiments in a single 5-minute wave. This makes it much harder to get stuck in local optima and much easier to find interaction effects between parameters. The agent can theo…

Yeah, assuming there's no active monitoring during the training runs, you can trivially give the agent an abstraction which turns "1 GPU" into "16 GPUs" that just so happens to take 16x the wall-clock time to run.

In fact, looking at the blog post, the agent orchestrating 16 GPUs is half as efficient as the agent using 1 GPU in GPU-time. Since it uses 16 GPUs to reach the same result as 1 GPU in 1/8 of the time.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#68

Post by gwern »

pbkhrv wrote: post #31

> How parallelism changed the agent’s research strategy > With a single GPU, the agent is stuck doing greedy hill-climbing: try one thing, check the result, pick a direction, try the next thing. With 16 GPUs, the strategy shifts. ...skip... 12 experiments in a single 5-minute wave. This makes it much harder to get stuck in local optima and much easier to find interaction effects between parameters. The agent can theo…

> The agent can theoretically come up with a protocol to run those same 12 experiments one-by-one and only then decide which branch to explore next - which I think would lead to the same outcome?

At least in theory, adaptiveness should save samples and in this case, compute. (As noted, you can always turn the parallel into serial and so the serial approach, which gets information 'from the future', should be able to meet or beat any parallel approach on sample-efficiency.)

So if the batch only matches the adaptive search, that suggests that the LLM is not reasoning well in the adaptive setting and is poorly exploiting the additional information. Maybe some sort of more explicit counterfactual reasoning/planning over a tree of possible outcomes?

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#69

Post by gwern »

herf wrote: post #44

This "early velocity only" approach seems like a problem - how do you know with 5-minute training runs that you aren't affecting the overall asymptote? e.g., what if the AI picks a quantizer that happens to be faster in the first five minutes, but has a big noise floor where it can't make more progress?

Yes, it's greedy so may hit local optima. You can fit learning curves and try to extrapolate out to avoid that problem, to let you run long enough to be reasonably sure of a dead end, and periodically revive past candidates to run longer. See past hyperparameter approaches like freeze-thaw https://arxiv.org/abs/1406.3896 .

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#70

Post by youngprogrammer »

rfw300 wrote: post #60
karpathy wrote: post #41

Earlier quoted context omitted.

The most recent round of autoresearch (round 2) which decreased "time to GPT-2" from 1.8 hours to 1.65 hours had some examples. I adjusted the program.md to "look at modded nanogpt project and draw inspirations from there for things to try" and it came back with a bunch of tuning, but also tried and implemented new architecture changes, some of which actually helped including the smear gate and the backout skip conne…

Do you have a sense of whether these validation loss improvements are leading to generalized performance uplifts? From afar I can't tell whether these are broadly useful new ideas or just industrialized overfitting on a particular (model, dataset, hardware) tuple.

industrialized overfitting is basically what ML researchers do
Post reply on HN