This is really cool - i'm going to try it on my old disseration.
Autoresearch on an old research idea
91–100 of 110 posts
Re: Autoresearch on an old research idea
#92Does autoresearch work for projects that are not llm based? Eg in karpathy's example he is optimizing the nanogpt. What if I wanted to improve a Unet for image segmentation?
Yes, that's the real strenght of it. The structure is dead simple so you just have to switch the goal metric.
I used it on a data science project to find the best rules for achieving a defined outcome. At first, for fun, then I actually used some of its insights (and it caught a sampling issue I overlooked, oops)
Re: Autoresearch on an old research idea
#93> “ The agent acted like a hyperparameter optimization algorithm with some basic reasoning baked in.” Good lens. The crux of the auto research repo is basically one file - program.md which is a system prompt that can be summarized as “do this in a loop: improve train.py, run the training, run evals, record result. Favor simplicity”. The other files are an arbitrary ML model that is being trained.
This is something I could almost never be bothered to do before, but I can now very lazily set up large parameter sweeps and visualization scripts to really probe things. There's a danger of "analysis paralysis" but I've still found it quite useful. Although I'm not sure it saves me time as much as sanity.
Re: Autoresearch on an old research idea
#94> Like with any LLM project, the first 90% of the work was super smooth and barely needed my intervention. The last 10% was a slog.
The author doesn't really describe which part was a slog, I thought autoresearch was supposed to be pretty much set and forget.
Re: Autoresearch on an old research idea
#95There are better techniques for hyper-parameter optimisation, right? I fear I have missed something important, why has Autoresearch blown up so much? The bottleneck in AI/ML/DL is always data (volume & quality) or compute. Does/can Autoresearch help improve large-scale datasets? Is it more compute efficien than humans?
> The bottleneck in AI/ML/DL is always data (volume & quality) or compute. Not true at all. The whole point of ML is to find better mappings from X to Y, even for the same X. Many benchmarks can’t be solved by just throwing more compute at the problem. They need to learn better functions which traditionally requires humans. And sometimes an algorithm lets you tap into more data. For example transformers had better pa…
Fair push back, but I do think the LSTM vs Transformers point kinda supports my position in the limit, not refutes. Once the compute bottleneck is removed, LSTMs scale favourably.
https://arxiv.org/pdf/2510.02228 (I believe there's similar work done on vanilla LSTMs, but I'd have to go digging)
So the bottleneck was compute. Which is compatible with 'data or compute'. But to accept your point, at the time the algorothmic advances were useful/did unlock/remove the bottleneck.
A wider point is that eventually (once compute and data are scaled enough) the algorithms are all learning the same representations: https://arxiv.org/pdf/2405.07987
And of course the canon: https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dat... http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Scaling compute & data > algorithmic cleverness
Re: Autoresearch on an old research idea
#96Earlier quoted context omitted.
Can we modify this approach to get LLMs that are good at specific programming languages or frameworks? That seems to be where local LLMs could really shine.
Would love to have a small local model that only knows about rails and mvc web development Alternatively, a modular model with multiple “experts” that I could mix and match for my specific stack I don’t need the model to know all of the Internet plus 20 different human languages. I just want it to be really good with the stack of the project
LLMs shine through emergent behaviour. Finding an LLM that does Rails doesn't know poetry is like finding a Rails human developer who doesn't have a hobby e.g. basketball. So what if they play basketball? They can code too!
Re: Autoresearch on an old research idea
#97I often use LLMs to explore prior art and maybe find some alternative ways of thinking of problems. About 90% of what it tells me is useless or inapplicable to my domain due to a technicality it could not have known, but the other 10% is nice and has helped me learn some great new things. I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are…
maybe you can preselect good ideas, build up guidelines describing most common pitfalls, extrapolate from ideas you already vetted etc and run on autopilot on a safe-ish subset
Re: Autoresearch on an old research idea
#98Earlier quoted context omitted.
It would seem wise to modify the autoresearch instructions to first estimate the computational costs rigorously and then sort and compare the proposals for human review, and for each actually executed attempt to feed back the computational costs with LoRa adapter? i.e. perhaps minimal changes to autoresearch can take control for cost-effective research to occur.
Yes but at that point you may as well use a proper hyperparameter tuning framework like optuna if all the LLM agent is supposed to do is do hyperparameter tuning.
Does optuna think abstractly (i.e. use LLM to interpret the code and come up with insights), or just perform hyperparameter tuning experiments on user-indicated parameters?
Re: Autoresearch on an old research idea
#99Autoresearch is nothing new, big players are already in the game with more sophisticated solutions: - https://arxiv.org/abs/2602.02660 (MARS) - https://arxiv.org/abs/2601.14525 (Execution-grounded automated AI research) - https://arxiv.org/abs/2601.10402 (ML-Master 2.0) The mostly used benchmark for automated AI engineering/ research is: https://github.com/openai/mle-bench
The thing is, autoresearch feels more accessible that the listed solutions. I can use it trivially on virtually any problem that has verifiable rewards and a feedback loop.
That’s because it is literally just a feedback loop?
Re: Autoresearch on an old research idea
#100This feels less like automated research and more like structured trial and error with a decent feedback loop. Still useful, but I think the real bottleneck is how good your eval metric is. If that’s weak, the whole loop just optimizes for the wrong thing faster.
I mean, isn’t that “the scientific method”?
Only if the model is actually a human or equivalent, otherwise we don’t know what it is.