Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autoresearch on an old research idea

ykumar.me

Post reply on HN
Print view View original

31–40 of 110 posts

Re: Autoresearch on an old research idea

#31

Post by ainch »

gwerbin wrote: post #27
nextos wrote: post #21

Earlier quoted context omitted.

AFAIK, it's a bit more than hyper-parameter tuning as it can also make non-parametric (structural) changes. Non-parametric optimization is not a new idea. I guess the hype is partly because people hope it will be less brute force now.

It's an LLM-powered evolutionary algorithm.

I'd like see a system like this take more inspiration from the ES literature, similar to AlphaEvolve. Let's see an archive of solutions, novelty scoring and some crossover rather than purely mutating the same file in a linear fashion.

Re: Autoresearch on an old research idea

#32

Post by ainch »

sdenton4 wrote: post #12
lamroger wrote: post #6

Awesome breakdown! It really feels like a hyper-hyper parameter search + bug fixer. I started looking at Kaggle again and autoresearch seems to converge to many of the solution vibes there. Wild ensembles, squeezing a bit of loss out. More engineering than research IMO

For raw hyperparameter search, though, I would expect a proper Bayesian framework to be much better. Eg, vizier.

I think it depends whether you can leverage some knowledge. It's possible for a person/LLM to look at a loss curve and say "oh that's undertraining, let's bump the lr" - whereas a Bayesian method doesn't necessarily have deeper understanding, so it'll waste a lot of time exploring the search space on poor options.

If you're resource unconstrained then BO should ofc do very well though.

Re: Autoresearch on an old research idea

#33

Post by kridsdale1 »

lucasay wrote: post #30

This feels less like automated research and more like structured trial and error with a decent feedback loop. Still useful, but I think the real bottleneck is how good your eval metric is. If that’s weak, the whole loop just optimizes for the wrong thing faster.

I mean, isn’t that “the scientific method”?

Re: Autoresearch on an old research idea

#34

Post by barrenko »

cyanydeez wrote: post #29
_pdp_ wrote: post #25

Take some working code. Ask an LLM to fix bugs. Measure performance and test coverage. Feed the results back into the LLM. Repeat. This has been the standard approach for more complex LLM deployments for a while now in our shop. Using different models across iterations is also something I've found useful in my own experiments. It's like getting a fresh pair of eyes.

Can we modify this approach to get LLMs that are good at specific programming languages or frameworks? That seems to be where local LLMs could really shine.

It's just RL-everything.

Re: Autoresearch on an old research idea

#36

Post by mandevil »

dvt wrote: post #14

Ok, so looking at the commit log[1], I was mostly interested in seeing what the "moonshot ideas" implementations looked like, but basically everything is just hyperparameter tuning. Which is nice, but likely not worth the $$$ spent on the tokens. Am I missing something here? [1] https://github.com/ykumards/eCLIP/commits/main/autoresearch

Optuna or skopt are open source and won't take any GPU time at all to do it.

Re: Autoresearch on an old research idea

#37

Post by 1970-01-01 »

> The original paper used several medical X-ray datasets which I don’t have access to anymore, so I needed a new dataset with spatial annotations to test the expert attention mechanism. I picked the Ukiyo-eVG dataset: ~11K Japanese woodblock prints

That's such a weird switch. There's lots of free medical imaging online. Example: https://www.cancerimagingarchive.net/

Re: Autoresearch on an old research idea

#40

Post by sdenton4 »

ainch wrote: post #32
sdenton4 wrote: post #12

Earlier quoted context omitted.

For raw hyperparameter search, though, I would expect a proper Bayesian framework to be much better. Eg, vizier.

I think it depends whether you can leverage some knowledge. It's possible for a person/LLM to look at a loss curve and say "oh that's undertraining, let's bump the lr" - whereas a Bayesian method doesn't necessarily have deeper understanding, so it'll waste a lot of time exploring the search space on poor options. If you're resource unconstrained then BO should ofc do very well though.

Yah, I'm a bit skeptical - ime humans tend to under explore due to incorrect assumptions. Often this is due to forming a narrative to explain some result, and then over attaching to it. Also, agents aren't actually good at reasoning yet.

Good Bayesian exploration is much, much better than grid search, and does indeed learn to avoid low value regions of the parameter space. If we're talking about five minute experiments (as in the blog post), Bayesian optimization should chew through the task no problem.

Post reply on HN