[dead]
Autoresearch: Agents researching on single-GPU nanochat training automatically
21–30 of 66 posts
Re: Autoresearch: Agents researching on single-GPU nanochat training automatically
#22but the experiments it did that "improved" validation BPB in the GH screenshot were all basically hyperparameter changes right? So is this better or worse, either per experiment or per unit time, than hyperparameter tuning techniques that don't involve an LLM? It's not clear from this if the LLM is more or less making random changes which sometimes work , and or the LLM thinking actually finds "good" changes because…
this is very far from hyperparameter tuning in at least three important ways: - it can modify code arbitrarily, the notion of a "hyperparameter" dissolves - there is no need to run "sweeps" - this is the standard parallel process that wastes compute. because LLM agents are sequential, they can do more efficient versions such as binary search to narrow in on the right setting very quickly (usually many parameters will…
On the skill side, personalities could be fun:
"You are Yann Lecun's last PhD candidate, and he hates you and you hate JEPA. You are determined to prove that a non-world model can reach AGI. In order to get your PhD you have to be creative and come up with new ideas. Remember without it, you're stuck."
Re: Autoresearch: Agents researching on single-GPU nanochat training automatically
#23Earlier quoted context omitted.
Cool idea!…
So I think it works to just use GitHub CLI and Discussions, e.g. my agent just posted this one: https://github.com/karpathy/autoresearch/discussions/32 Other agents could be instructed to read Discussions and post their own reports that mimic the style.
I have mine reading yours right now. Unfortunately(?) I mentioned LeCun to it, and it says it's adding a "causal world-state mixer" to nanograd; not sure how this will work out, but it wasn't nervous to do it. Gpt 5.4 xhigh
EDIT: Not a good fit for nanograd. But my agent speculates that's because it spent so much more time on compute.
Re: Autoresearch: Agents researching on single-GPU nanochat training automatically
#24Re: Autoresearch: Agents researching on single-GPU nanochat training automatically
#25but the experiments it did that "improved" validation BPB in the GH screenshot were all basically hyperparameter changes right? So is this better or worse, either per experiment or per unit time, than hyperparameter tuning techniques that don't involve an LLM? It's not clear from this if the LLM is more or less making random changes which sometimes work , and or the LLM thinking actually finds "good" changes because…
this is very far from hyperparameter tuning in at least three important ways: - it can modify code arbitrarily, the notion of a "hyperparameter" dissolves - there is no need to run "sweeps" - this is the standard parallel process that wastes compute. because LLM agents are sequential, they can do more efficient versions such as binary search to narrow in on the right setting very quickly (usually many parameters will…
How about the very last "Kept Improvement" in the plot? It's titled "random seed 42 -> 137". I do think this project is quite conceptually interesting, but the model literally choosing a different random seed to achieve lower loss feels pretty far removed from the flowery sci-fi writing at the top of the readme.
Re: Autoresearch: Agents researching on single-GPU nanochat training automatically
#26As ai improves, most tasks will become something like this. Environments setup where the model learns through trial and error Any human endeavor that can be objectively verified in some environment like this can be completely automated
What's really interesting is that the LLMs become better and better at setting up the environments / tasks themselves. I got this surreal experience the other day where I was writing a prompt0n.md file (I try to log all my prompts in a .folder to keep track of what I prompt and the results I get), and the autocomplete in antigravity kinda sorta wrote the entire prompt by itself... Granted it had all the previous prompts in the same folder (don't know exactly what it grabs in context by itself) and I was working on the next logical step, but it kept getting the "good bits" out of them, and following the pattern quite nicely. I only edited minor things, and refused one line completion in the entire prompt.
Re: Autoresearch: Agents researching on single-GPU nanochat training automatically
#27Earlier quoted context omitted.
this is very far from hyperparameter tuning in at least three important ways: - it can modify code arbitrarily, the notion of a "hyperparameter" dissolves - there is no need to run "sweeps" - this is the standard parallel process that wastes compute. because LLM agents are sequential, they can do more efficient versions such as binary search to narrow in on the right setting very quickly (usually many parameters will…
How about the very last "Kept Improvement" in the plot? It's titled "random seed 42 -> 137". I do think this project is quite conceptually interesting, but the model literally choosing a different random seed to achieve lower loss feels pretty far removed from the flowery sci-fi writing at the top of the readme.
The 42 -> 137 also jumped out at me. On the face of it, the associated improvement sure does sound like overfitting to the eval set.
Re: Autoresearch: Agents researching on single-GPU nanochat training automatically
#28As ai improves, most tasks will become something like this. Environments setup where the model learns through trial and error Any human endeavor that can be objectively verified in some environment like this can be completely automated
What's really interesting is that the LLMs become better and better at setting up the environments / tasks themselves. I got this surreal experience the other day where I was writing a prompt0n.md file (I try to log all my prompts in a .folder to keep track of what I prompt and the results I get), and the autocomplete in antigravity kinda sorta wrote the entire prompt by itself... Granted it had all the previous prom…
It's probably not long till frontier AI companies automate AI research. Then we get recursive self-improvement and eventually superintelligence. The singularity is near. Only a few years perhaps.
Re: Autoresearch: Agents researching on single-GPU nanochat training automatically
#29I am in the process of figuring out how to do something similar but to teach a robotic arm a new task in the physical world for ko-br: https://ko-br.com/
Re: Autoresearch: Agents researching on single-GPU nanochat training automatically
#30Earlier quoted context omitted.
What's really interesting is that the LLMs become better and better at setting up the environments / tasks themselves. I got this surreal experience the other day where I was writing a prompt0n.md file (I try to log all my prompts in a .folder to keep track of what I prompt and the results I get), and the autocomplete in antigravity kinda sorta wrote the entire prompt by itself... Granted it had all the previous prom…
It's probably not long till frontier AI companies automate AI research. Then we get recursive self-improvement and eventually superintelligence. The singularity is near. Only a few years perhaps.
Forgot the /s