Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autoresearch on an old research idea

ykumar.me

Post reply on HN
Print view View original

51–60 of 110 posts

Re: Autoresearch on an old research idea

#51

Post by Eufrat »

M4v3R wrote: post #45
Eufrat wrote: post #15

Earlier quoted context omitted.

I find LLMs useful in regurgitating one-liners that I can’t be bothered to remember or things where even being flat out wrong is okay and you just do it yourself. For all the folks spending a lot of time and energy in setting up MCP servers, AGENTS.md, etc. I think this represents more that the LLM cannot do what it is being sold as by AI boosters and needs extreme amounts of guidance to reach a desired goal, if it e…

> I find LLMs useful in regurgitating one-liners This was the case for me a year ago. Now Claude or Codex are routinely delivering finished & tested complete features in my projects. I move much, much faster than before and I don’t have an elaborate setup - just a single CLAUDE.md file with some basic information about the project and that’s it.

People keep saying this and I agree Claude has gotten a lot better even in my own experience, but I think the value is questionable.

What’s the point of adding features that are inscrutable? I have gotten Claude to make a feature and it mostly works and if it doesn’t work quite right I spend a massive amount of time trying to understand what is going on. For things that don’t matter too much, like prototyping, I think it’s great to just be able to get a working demo out faster, but it’s kind of terrifying when people start doing this for production stuff. Especially if their domain knowledge is limited. I can personally attest to seeing multiple insane things that are clearly vibe coded by people who don’t understand things. In one case, I saw API keys exposed because they were treating database users as regular user accounts for website login auth.

> I move much, much faster than before

This is a bad metric as has been attested multiple times in unrelated situations. Moving faster is not necessarily productivity nor is it value.

Re: Autoresearch on an old research idea

#52

Post by electroglyph »

foobarian wrote: post #20
Eufrat wrote: post #15

Earlier quoted context omitted.

I find LLMs useful in regurgitating one-liners that I can’t be bothered to remember or things where even being flat out wrong is okay and you just do it yourself. For all the folks spending a lot of time and energy in setting up MCP servers, AGENTS.md, etc. I think this represents more that the LLM cannot do what it is being sold as by AI boosters and needs extreme amounts of guidance to reach a desired goal, if it e…

> I find LLMs useful in regurgitating one-liners that I can’t be bothered to remember I found LLMs make a fabulous frontend for git :-D

ah, you've found the danger zone!

Re: Autoresearch on an old research idea

#54

Post by bonoboTP »

jpcompartir wrote: post #16

There are better techniques for hyper-parameter optimisation, right? I fear I have missed something important, why has Autoresearch blown up so much? The bottleneck in AI/ML/DL is always data (volume & quality) or compute. Does/can Autoresearch help improve large-scale datasets? Is it more compute efficien than humans?

There is a field of AutoML, with its own specialized academic literature and libraries that tried to achieve this type of thing but didn't work very well in practice.

Years ago there were big hopes about bayesian hyperparameter optimization, predicting performance with Gaussian processes etc, hyperopt library, but it was often starting wasteful experiments because it really didn't have any idea what the parameters did. People mostly just do grid search and random search with a configuration that you set up by intuition and experience. Meanwhile LLMs can see what each hyperparameter does, it can see what techniques and settings have worked in the literature, it can do something approximating common sense regarding what has a big enough effect. It's surprisingly difficult to precisely define when a training curve has really flattened for example.

So in theory there are many non-LLM approaches but they are not great. Maybe this is also not so great yet. But maybe it will be.

Re: Autoresearch on an old research idea

#55

Post by genxy »

andy12_ wrote: post #23
datsci_est_2015 wrote: post #4

I often use LLMs to explore prior art and maybe find some alternative ways of thinking of problems. About 90% of what it tells me is useless or inapplicable to my domain due to a technicality it could not have known, but the other 10% is nice and has helped me learn some great new things. I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are…

I think the main value lies in allowing the agent to try many things while you aren't working (when you are sleeping or doing other activities), so even if many tests are not useful, with many trials it can find something nice without any effort on your part. This is, of course, only applicable if doing a single test is relatively fast. In my work a single test can take half a day, so I'd rather not let an agent spen…

> single test can take half a day

Why is that?

I don't doubt you, but when Shigeo Shingo created SMED (Single Minute Exchange of Die), die changes were an hours long process.

Re: Autoresearch on an old research idea

#56

Post by UncleOxidant »

gwerbin wrote: post #27
nextos wrote: post #21

Earlier quoted context omitted.

AFAIK, it's a bit more than hyper-parameter tuning as it can also make non-parametric (structural) changes. Non-parametric optimization is not a new idea. I guess the hype is partly because people hope it will be less brute force now.

It's an LLM-powered evolutionary algorithm.

That was my impression. Including evolutionary programming which normally would happen at the AST level, with the LLM it can happen at the source level.

Re: Autoresearch on an old research idea

#58

Post by pikachu0625 »

It's better to outsource optimization phases. Our idea should be for constraint, assumptions etc. for breakthrough. Boyd often argues that once you can express a problem in a standard mathematical form, the implementation becomes a commodity that software can handle automatically.

Re: Autoresearch on an old research idea

#60

Post by lucasay »

kridsdale1 wrote: post #33
lucasay wrote: post #30

This feels less like automated research and more like structured trial and error with a decent feedback loop. Still useful, but I think the real bottleneck is how good your eval metric is. If that’s weak, the whole loop just optimizes for the wrong thing faster.

I mean, isn’t that “the scientific method”?

Partially—but science also questions the hypothesis and the metric. This mostly assumes both are correct and just optimizes within that box.
Post reply on HN