Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

blog.skypilot.co

Post reply on HN
Print view View original

11–20 of 126 posts

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#11

Post by hhh »

zhwu wrote: post #3

The most surprising part: the agent had access to both H100s and H200s. Without being told, it noticed H200s scored better and started screening ideas on H100s, then promoting winners to H200s for validation. That strategy emerged entirely on its own.

Why?… The experiment.yaml shows that it is calling h100/200 explicitly, it’s pretty common for humans to say “number bigger more gooder” for anything… Lie and reverse the values and see what happens. I would put money on a rabbit hole of complaining about it being misconfigured.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#13

Post by kraddypatties »

ipsum2 wrote: post #10
kraddypatties wrote: post #2

I feel like most of this recent Autoresearch trend boils down to reinventing hyper-parameter tuning. Is the SOTA still Bayesian optimization when given a small cluster? It was ~3 years ago when I was doing this kind of work, haven't kept up since then. Also, shoutout SkyPilot! It's been a huge help for going multi-cloud with our training and inference jobs (getting GPUs is still a nightmare...)!

Hyperparam tuning that has better intuition and can incorporate architecture changes automatically. It won't invent something completely new though.

Hm, that's fair. It does feel like there's low hanging fruit in combining "old school" methods for conducting a hyperparameter sweep efficiently _with_ the higher level architecture edit ability of Autoresearch.

Probably would cut the number of runs down by a significant number (as far as I can tell it's doing a grid search once it decides to mess with a knob or section of the architecture).

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#14

Post by chaos_emergent »

covi wrote: post #4

This feels like the chimpanzee with a power drill. An agent is honestly just brute-force search, but guided.

Human-driven research is also brute-force but with a more efficient search strategy. One can think of a parameter that represents research-search-space-navigation efficiency. RL-trained agents will inevitably optimize for that parameter. I agree with your statement insomuch as the value of that efficiency parameter is lower for agents than humans today.

It's really hard to imagine that they __won't__ exceed the human value for that efficiency parameter rather soon given that 1. there are plenty of scalar value functions that can represent research efficiency, of which a subset will result in robust training, and 2. that AI labs have a massive incentive to increase their research efficiency overall, along with billions of dollars and really good human researchers working on the problem.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#15

Post by karpathy »

kraddypatties wrote: post #2

I feel like most of this recent Autoresearch trend boils down to reinventing hyper-parameter tuning. Is the SOTA still Bayesian optimization when given a small cluster? It was ~3 years ago when I was doing this kind of work, haven't kept up since then. Also, shoutout SkyPilot! It's been a huge help for going multi-cloud with our training and inference jobs (getting GPUs is still a nightmare...)!

Wrong and short-sighted take given that the LLM explores serially learning along the way, and can tool use and change code arbitrarily. It seems to currently default to something resembling hyperparameter tuning in absence of more specific instructions. I briefly considered calling the project “autotune” at first but I think “autoresearch” will prove to be the significantly more appropriate name.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#16

Post by ed »

hhh wrote: post #11
zhwu wrote: post #3

The most surprising part: the agent had access to both H100s and H200s. Without being told, it noticed H200s scored better and started screening ideas on H100s, then promoting winners to H200s for validation. That strategy emerged entirely on its own.

Why?… The experiment.yaml shows that it is calling h100/200 explicitly, it’s pretty common for humans to say “number bigger more gooder” for anything… Lie and reverse the values and see what happens. I would put money on a rabbit hole of complaining about it being misconfigured.

Models are familiar with H100’s. They even predate ChatGPT.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#17

Post by fabmilo »

I am fascinated by this example of using AI to improve AI. I won a small prize using this technique on helion kernels at a pytorch hackathon in SF.

The next step are: - give the agent the whole deep learning literature research and do tree search over the various ideas that have been proposed in the past. - have some distributed notepad that any of these agents can read and improve upon.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#18

Post by groby_b »

covi wrote: post #4

This feels like the chimpanzee with a power drill. An agent is honestly just brute-force search, but guided.

Is there anything in the research space that doesn't fit "brute-force search, but guided"?

All of science is "gather inputs, make hypothesis, test, analyse" on repeat.

There's plenty to critique in the particular guidance approach, but the overall method is the same.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#19

Post by corndoge »

karpathy wrote: post #15
kraddypatties wrote: post #2

I feel like most of this recent Autoresearch trend boils down to reinventing hyper-parameter tuning. Is the SOTA still Bayesian optimization when given a small cluster? It was ~3 years ago when I was doing this kind of work, haven't kept up since then. Also, shoutout SkyPilot! It's been a huge help for going multi-cloud with our training and inference jobs (getting GPUs is still a nightmare...)!

Wrong and short-sighted take given that the LLM explores serially learning along the way, and can tool use and change code arbitrarily. It seems to currently default to something resembling hyperparameter tuning in absence of more specific instructions. I briefly considered calling the project “autotune” at first but I think “autoresearch” will prove to be the significantly more appropriate name.

Would you say it's fair to describe autoresearch as a form of neural architecture search? I am curious what you think the core differences are between them.

Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster

#20

Post by kraddypatties »

karpathy wrote: post #15
kraddypatties wrote: post #2

I feel like most of this recent Autoresearch trend boils down to reinventing hyper-parameter tuning. Is the SOTA still Bayesian optimization when given a small cluster? It was ~3 years ago when I was doing this kind of work, haven't kept up since then. Also, shoutout SkyPilot! It's been a huge help for going multi-cloud with our training and inference jobs (getting GPUs is still a nightmare...)!

Wrong and short-sighted take given that the LLM explores serially learning along the way, and can tool use and change code arbitrarily. It seems to currently default to something resembling hyperparameter tuning in absence of more specific instructions. I briefly considered calling the project “autotune” at first but I think “autoresearch” will prove to be the significantly more appropriate name.

I can believe that in the long run.

Does the agent have access to arxiv (a brief skim of the README didn't have an answer)? If not, it could be that the current approach of relying on the model's weights only is resulting in the perceived local optimum of hyperparameter tuning.

Anecdotally, we built a little MCP for arxiv to help with our internal research, noticed a significant boost in the diversity of methods (architecture or otherwise) Claude and friends were able to reference.

Post reply on HN