Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autoresearch on an old research idea

ykumar.me

Post reply on HN
Print view View original

21–30 of 110 posts

Re: Autoresearch on an old research idea

#21

Post by nextos »

jpcompartir wrote: post #16

There are better techniques for hyper-parameter optimisation, right? I fear I have missed something important, why has Autoresearch blown up so much? The bottleneck in AI/ML/DL is always data (volume & quality) or compute. Does/can Autoresearch help improve large-scale datasets? Is it more compute efficien than humans?

AFAIK, it's a bit more than hyper-parameter tuning as it can also make non-parametric (structural) changes.

Non-parametric optimization is not a new idea. I guess the hype is partly because people hope it will be less brute force now.

Re: Autoresearch on an old research idea

#22

Post by DoctorOetker »

dvt wrote: post #14

Ok, so looking at the commit log[1], I was mostly interested in seeing what the "moonshot ideas" implementations looked like, but basically everything is just hyperparameter tuning. Which is nice, but likely not worth the $$$ spent on the tokens. Am I missing something here? [1] https://github.com/ykumards/eCLIP/commits/main/autoresearch

It would seem wise to modify the autoresearch instructions to first estimate the computational costs rigorously and then sort and compare the proposals for human review, and for each actually executed attempt to feed back the computational costs with LoRa adapter?

i.e. perhaps minimal changes to autoresearch can take control for cost-effective research to occur.

Re: Autoresearch on an old research idea

#23

Post by andy12_ »

datsci_est_2015 wrote: post #4

I often use LLMs to explore prior art and maybe find some alternative ways of thinking of problems. About 90% of what it tells me is useless or inapplicable to my domain due to a technicality it could not have known, but the other 10% is nice and has helped me learn some great new things. I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are…

I think the main value lies in allowing the agent to try many things while you aren't working (when you are sleeping or doing other activities), so even if many tests are not useful, with many trials it can find something nice without any effort on your part.

This is, of course, only applicable if doing a single test is relatively fast. In my work a single test can take half a day, so I'd rather not let an agent spend a whole night doing a bogus test.

Re: Autoresearch on an old research idea

#24

Post by Denzel »

simonw wrote: post #11
BrokenCogs wrote: post #7

Does autoresearch work for projects that are not llm based? Eg in karpathy's example he is optimizing the nanogpt. What if I wanted to improve a Unet for image segmentation?

Tobi from Shopify used a variant of autoresearch to optimize the Liquid template engine, and found a 53% speedup after ~120 experiments: https://github.com/Shopify/liquid/pull/2056 I wrote up some more notes on that here: https://simonwillison.net/2026/Mar/13/liquid/

How much did this cost? Has there ever been an engineering focus on performance for liquid?

It’s certainly cool, but the optimizations are so basic that I’d expect a performance engineer to find these within a day or two with some flame graphs and profiling.

Re: Autoresearch on an old research idea

#25

Post by _pdp_ »

Take some working code. Ask an LLM to fix bugs. Measure performance and test coverage. Feed the results back into the LLM. Repeat.

This has been the standard approach for more complex LLM deployments for a while now in our shop.

Using different models across iterations is also something I've found useful in my own experiments. It's like getting a fresh pair of eyes.

Re: Autoresearch on an old research idea

#26

Post by coppsilgold »

nextos wrote: post #21
jpcompartir wrote: post #16

There are better techniques for hyper-parameter optimisation, right? I fear I have missed something important, why has Autoresearch blown up so much? The bottleneck in AI/ML/DL is always data (volume & quality) or compute. Does/can Autoresearch help improve large-scale datasets? Is it more compute efficien than humans?

AFAIK, it's a bit more than hyper-parameter tuning as it can also make non-parametric (structural) changes. Non-parametric optimization is not a new idea. I guess the hype is partly because people hope it will be less brute force now.

Perhaps LLM-guided Superoptimization: https://en.wikipedia.org/wiki/Superoptimization>

I recall reading about a stochastic one years ago: https://github.com/StanfordPL/stoke>

Re: Autoresearch on an old research idea

#27

Post by gwerbin »

nextos wrote: post #21
jpcompartir wrote: post #16

There are better techniques for hyper-parameter optimisation, right? I fear I have missed something important, why has Autoresearch blown up so much? The bottleneck in AI/ML/DL is always data (volume & quality) or compute. Does/can Autoresearch help improve large-scale datasets? Is it more compute efficien than humans?

AFAIK, it's a bit more than hyper-parameter tuning as it can also make non-parametric (structural) changes. Non-parametric optimization is not a new idea. I guess the hype is partly because people hope it will be less brute force now.

It's an LLM-powered evolutionary algorithm.

Re: Autoresearch on an old research idea

#28

Post by simonw »

Denzel wrote: post #24
simonw wrote: post #11

Earlier quoted context omitted.

Tobi from Shopify used a variant of autoresearch to optimize the Liquid template engine, and found a 53% speedup after ~120 experiments: https://github.com/Shopify/liquid/pull/2056 I wrote up some more notes on that here: https://simonwillison.net/2026/Mar/13/liquid/

How much did this cost? Has there ever been an engineering focus on performance for liquid? It’s certainly cool, but the optimizations are so basic that I’d expect a performance engineer to find these within a day or two with some flame graphs and profiling.

He used Pi as the harness but didn't say which underlying model. My stab-in-the-air guess would be no more than a few hundred dollars in token spend (for 120 experiments run over a few days assuming Claude Opus 4.6 used without the benefits of the Claude Max plan.)

So cheaper than a performance engineer for a day or two... but the Shopify CEO's own time is likely a whole lot more expensive than a regular engineer!

Re: Autoresearch on an old research idea

#29

Post by cyanydeez »

_pdp_ wrote: post #25

Take some working code. Ask an LLM to fix bugs. Measure performance and test coverage. Feed the results back into the LLM. Repeat. This has been the standard approach for more complex LLM deployments for a while now in our shop. Using different models across iterations is also something I've found useful in my own experiments. It's like getting a fresh pair of eyes.

Can we modify this approach to get LLMs that are good at specific programming languages or frameworks? That seems to be where local LLMs could really shine.

Re: Autoresearch on an old research idea

#30

Post by lucasay »

This feels less like automated research and more like structured trial and error with a decent feedback loop. Still useful, but I think the real bottleneck is how good your eval metric is. If that’s weak, the whole loop just optimizes for the wrong thing faster.
Post reply on HN