I'm trying to find the 'wow' factor in this. Finding the optimal combination of parameters, given a validation criteria should be a boring repetitive task for a machine or a human. Is it about determining how to utilize the given hardware?
Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
101–110 of 126 posts
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#102[dead]
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#103Earlier quoted context omitted.
Have you actually used LLMs for non trivial tasks? They are still incredibly bad when it comes to actually hard engineering work and they still lie all the time, it's just gotten harder to notice, especially if you're just letting it run all night and generate reams of crap. Most people are optimizing for terrible benchmarks and then don't really understand what the model did anyone and just assume it did something g…
Do you realise who you’re replying to?
[dead]
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#104Earlier quoted context omitted.
Out of curiosity, what sort of things have you seen it do that better fit 'autoresearch' than 'autotune' thus far? Optimizations it made that wouldn't be been surfaced by an autotune system, I suppose.
The most recent round of autoresearch (round 2) which decreased "time to GPT-2" from 1.8 hours to 1.65 hours had some examples. I adjusted the program.md to "look at modded nanogpt project and draw inspirations from there for things to try" and it came back with a bunch of tuning, but also tried and implemented new architecture changes, some of which actually helped including the smear gate and the backout skip conne…
Did you consider providing the LLM with a framework for automatic hyperparamter tuning? This would free up its capacity to focus on the more important architectural decisions.
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#105This feels like the chimpanzee with a power drill. An agent is honestly just brute-force search, but guided.
Your "brute-force search, but guided" feels like oxymoron. How does it differ from "guided search"?
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#106Earlier quoted context omitted.
I think we need to separate theory from practice. In theory, it can edit the training loop and come up with novel techniques. That is interesting. In practice, the vast majority of the changes that auto research actually made would have been found much faster with BO if properly parameterized. You do not need an LLM to find a better batch size or learning rate.
I’d always hoped something like this could take advantage of FPGAs directly
FPGAs won't rebuild fast enough for it to matter vs software simulation I'd wager. Even FPGA-in-CPU has been a dream for decades and there you have more time for some workloads, still never was commercially viable for general computing.
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#107Earlier quoted context omitted.
Why do we think this emerged “on its own”? Surely this technique has been discussed in research papers that are in the training set.
You probably express very few truly original ideas. Let’s not set the bar quite so high unless we are all just a sad simulacrum of “pure” thought.
Original ideas are easy if you allow for bad ideas.
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#108Earlier quoted context omitted.
Why do we think this emerged “on its own”? Surely this technique has been discussed in research papers that are in the training set.
You probably express very few truly original ideas. Let’s not set the bar quite so high unless we are all just a sad simulacrum of “pure” thought.
Then "on its own" has no meaning, i.e. everything an LLM does is "on its own".
Re: Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
#109Earlier quoted context omitted.
Have you actually used LLMs for non trivial tasks? They are still incredibly bad when it comes to actually hard engineering work and they still lie all the time, it's just gotten harder to notice, especially if you're just letting it run all night and generate reams of crap. Most people are optimizing for terrible benchmarks and then don't really understand what the model did anyone and just assume it did something g…
Do you realise who you’re replying to?
[deleted]