Autoresearch on an old research idea
ykumar.me
Autoresearch on an old research idea
1–10 of 110 posts
Re: Autoresearch on an old research idea
#2So... It did work. It found bugs (that he didn't know about) and it did optimization (that he hadn't done).
Re: Autoresearch on an old research idea
#3[flagged]
Re: Autoresearch on an old research idea
#4I often use LLMs to explore prior art and maybe find some alternative ways of thinking of problems. About 90% of what it tells me is useless or inapplicable to my domain due to a technicality it could not have known, but the other 10% is nice and has helped me learn some great new things.
I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are very poorly maintained / niche libraries that have quite a lot of content written about them but what I can only imagine is very limited use in real production environments.
On the other hand, we have domain expert “consultants” in our leadership’s ears making equally absurd recommendations that we constantly have to disprove. Maybe an agent can occupy those consultants and let us do our work in peace.
Re: Autoresearch on an old research idea
#5Try this if the main link is not responsive - https://archive.is/6xLiU
Re: Autoresearch on an old research idea
#6Awesome breakdown! It really feels like a hyper-hyper parameter search + bug fixer.
I started looking at Kaggle again and autoresearch seems to converge to many of the solution vibes there.
Wild ensembles, squeezing a bit of loss out. More engineering than research IMO
Re: Autoresearch on an old research idea
#7Does autoresearch work for projects that are not llm based? Eg in karpathy's example he is optimizing the nanogpt. What if I wanted to improve a Unet for image segmentation?
Re: Autoresearch on an old research idea
#8I often use LLMs to explore prior art and maybe find some alternative ways of thinking of problems. About 90% of what it tells me is useless or inapplicable to my domain due to a technicality it could not have known, but the other 10% is nice and has helped me learn some great new things. I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are…
> agent try everything that the LLM chatbot had recommended ($$$)
A lot depends on whether it is expensive to you. I use Claude Code for the smallest of whims and rarely run out of tokens on my Max plan.
Re: Autoresearch on an old research idea
#9> “ The agent acted like a hyperparameter optimization algorithm with some basic reasoning baked in.”
Good lens.
The crux of the auto research repo is basically one file - program.md which is a system prompt that can be summarized as “do this in a loop: improve train.py, run the training, run evals, record result. Favor simplicity”. The other files are an arbitrary ML model that is being trained.
Re: Autoresearch on an old research idea
#10Does autoresearch work for projects that are not llm based? Eg in karpathy's example he is optimizing the nanogpt. What if I wanted to improve a Unet for image segmentation?
The gist of these things is you point them at an eval metric and say 'make it go better.' so, you can point it at anything you can measure. The example in the blog post here is bonding boxes on wood cut images.