Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
1–10 of 126 posts
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#2I feel like most of this recent Autoresearch trend boils down to reinventing hyper-parameter tuning. Is the SOTA still Bayesian optimization when given a small cluster? It was ~3 years ago when I was doing this kind of work, haven't kept up since then.
Also, shoutout SkyPilot! It's been a huge help for going multi-cloud with our training and inference jobs (getting GPUs is still a nightmare...)!
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#3The most surprising part: the agent had access to both H100s and H200s. Without being told, it noticed H200s scored better and started screening ideas on H100s, then promoting winners to H200s for validation. That strategy emerged entirely on its own.
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#4This feels like the chimpanzee with a power drill. An agent is honestly just brute-force search, but guided.
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#5The most surprising part: the agent had access to both H100s and H200s. Without being told, it noticed H200s scored better and started screening ideas on H100s, then promoting winners to H200s for validation. That strategy emerged entirely on its own.
Yeah I thought that was a particularly neat part
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#6I feel like most of this recent Autoresearch trend boils down to reinventing hyper-parameter tuning. Is the SOTA still Bayesian optimization when given a small cluster? It was ~3 years ago when I was doing this kind of work, haven't kept up since then. Also, shoutout SkyPilot! It's been a huge help for going multi-cloud with our training and inference jobs (getting GPUs is still a nightmare...)!
[deleted]
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#7The most surprising part: the agent had access to both H100s and H200s. Without being told, it noticed H200s scored better and started screening ideas on H100s, then promoting winners to H200s for validation. That strategy emerged entirely on its own.
Why do we think this emerged “on its own”? Surely this technique has been discussed in research papers that are in the training set.
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#8[dead]
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#9A cluster is 2 nodes? That's technically true, but not very exciting.
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#10I feel like most of this recent Autoresearch trend boils down to reinventing hyper-parameter tuning. Is the SOTA still Bayesian optimization when given a small cluster? It was ~3 years ago when I was doing this kind of work, haven't kept up since then. Also, shoutout SkyPilot! It's been a huge help for going multi-cloud with our training and inference jobs (getting GPUs is still a nightmare...)!
Hyperparam tuning that has better intuition and can incorporate architecture changes automatically. It won't invent something completely new though.