Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autoresearch on an old research idea

ykumar.me

Post reply on HN
Print view View original

101–110 of 110 posts

Re: Autoresearch on an old research idea

#101

Post by stingraycharles »

DoctorOetker wrote: post #98
stingraycharles wrote: post #86

Earlier quoted context omitted.

Yes but at that point you may as well use a proper hyperparameter tuning framework like optuna if all the LLM agent is supposed to do is do hyperparameter tuning.

Does optuna think abstractly (i.e. use LLM to interpret the code and come up with insights), or just perform hyperparameter tuning experiments on user-indicated parameters?

The latter, but it uses fairly optimized approaches to ensure it selects the best candidates.

If you look at the commits, you can see that all it does is just set different values for different parameters of continuous values: the type of thing that I trust statistics a lot more than reasoning. Optuna can make very informed decisions when making lots of different changes at once, slowly converging towards optimal parameters, where the LLM seems to be throwing stuff at a wall and see what sticks.

What would work best if the LLM would try to approach things on a higher level, ie use Optuna, but reason about better approaches for algorithms and/or data or whatever. But what it ends up doing is tuning parameters manually, only one / a few at a time, extremely inefficient and unlikely to be optimal.

Re: Autoresearch on an old research idea

#102

Post by buzarchitect »

Eufrat wrote: post #15
datsci_est_2015 wrote: post #4

I often use LLMs to explore prior art and maybe find some alternative ways of thinking of problems. About 90% of what it tells me is useless or inapplicable to my domain due to a technicality it could not have known, but the other 10% is nice and has helped me learn some great new things. I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are…

I find LLMs useful in regurgitating one-liners that I can’t be bothered to remember or things where even being flat out wrong is okay and you just do it yourself. For all the folks spending a lot of time and energy in setting up MCP servers, AGENTS.md, etc. I think this represents more that the LLM cannot do what it is being sold as by AI boosters and needs extreme amounts of guidance to reach a desired goal, if it e…

This matches my experience. I've been building structured pipelines around LLMs, and the biggest lesson is that the raw model is maybe 30% of the value. The other 70% is the methodology you wrap around it; what data you feed in before the conversation starts, what you do when the model gives a weak answer, and whether you track open questions and circle back to them.

The irony is that "extreme amounts of guidance" is exactly what makes a human domain expert valuable, too. A senior consultant isn't smarter than a junior one; they have a better methodology for directing attention to what matters. The actual problem with the "just throw an agent at it" approach isn't cost. It's that without structure, you can't tell the 10% of useful output from the 90% of noise

Re: Autoresearch on an old research idea

#103

Post by noobermin »

datsci_est_2015 wrote: post #4

I often use LLMs to explore prior art and maybe find some alternative ways of thinking of problems. About 90% of what it tells me is useless or inapplicable to my domain due to a technicality it could not have known, but the other 10% is nice and has helped me learn some great new things. I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are…

This is so funny. The consultants are having their ai agents tell your boss the same thing about you, but you're different, you're bright. I bet chat told you that too.

Re: Autoresearch on an old research idea

#104

Post by janalsncm »

jpcompartir wrote: post #95
janalsncm wrote: post #63

Earlier quoted context omitted.

> The bottleneck in AI/ML/DL is always data (volume & quality) or compute. Not true at all. The whole point of ML is to find better mappings from X to Y, even for the same X. Many benchmarks can’t be solved by just throwing more compute at the problem. They need to learn better functions which traditionally requires humans. And sometimes an algorithm lets you tap into more data. For example transformers had better pa…

Fair push back, but I do think the LSTM vs Transformers point kinda supports my position in the limit, not refutes. Once the compute bottleneck is removed, LSTMs scale favourably. https://arxiv.org/pdf/2510.02228 (I believe there's similar work done on vanilla LSTMs, but I'd have to go digging) So the bottleneck was compute. Which is compatible with 'data or compute'. But to accept your point, at the time the algorot…

Algorithms do matter because compute is not unlimited in practice. Otherwise we might as well use bogo sort because the result is eventually the same. Yes the platonic ideal of a sorted list looks the same but that doesn’t tell you anything about how to get there or whether you can in this lifetime.

I bring up transformers because scaling compute and data was unlocked by a better algorithm. It matters a lot because scaling compute isn’t always an option.

Re: Autoresearch on an old research idea

#105

Post by nico »

mememememememo wrote: post #96
nico wrote: post #46

Earlier quoted context omitted.

Would love to have a small local model that only knows about rails and mvc web development Alternatively, a modular model with multiple “experts” that I could mix and match for my specific stack I don’t need the model to know all of the Internet plus 20 different human languages. I just want it to be really good with the stack of the project

LLMs shine through emergent behaviour. Finding an LLM that does Rails doesn't know poetry is like finding a Rails human developer who doesn't have a hobby e.g. basketball. So what if they play basketball? They can code too!

Then it might need a new type of architecture to work. I’m not attached to LLMs. If a new model comes out that can do only the things I want it do it, then great

Re: Autoresearch on an old research idea

#106

Post by mememememememo »

nico wrote: post #105
mememememememo wrote: post #96

Earlier quoted context omitted.

LLMs shine through emergent behaviour. Finding an LLM that does Rails doesn't know poetry is like finding a Rails human developer who doesn't have a hobby e.g. basketball. So what if they play basketball? They can code too!

Then it might need a new type of architecture to work. I’m not attached to LLMs. If a new model comes out that can do only the things I want it do it, then great

Sure. I'd use it too. I am just not sure they have that trade off to make. Yet (maybe with more research)

Re: Autoresearch on an old research idea

#107

Post by DoctorOetker »

stingraycharles wrote: post #101
DoctorOetker wrote: post #98

Earlier quoted context omitted.

Does optuna think abstractly (i.e. use LLM to interpret the code and come up with insights), or just perform hyperparameter tuning experiments on user-indicated parameters?

The latter, but it uses fairly optimized approaches to ensure it selects the best candidates. If you look at the commits, you can see that all it does is just set different values for different parameters of continuous values: the type of thing that I trust statistics a lot more than reasoning. Optuna can make very informed decisions when making lots of different changes at once, slowly converging towards optimal par…

but you said

> Yes but at that point you may as well use a proper hyperparameter tuning framework like optuna if all the LLM agent is supposed to do is do hyperparameter tuning.

while the "novelty" of autoresearch is that it may symbolically reason about the computation, analyze the codebase, etc. i.e. a wider search space (harder) but symbolic reasoning.

Re: Autoresearch on an old research idea

#108

Post by M4v3R »

Eufrat wrote: post #51
M4v3R wrote: post #45

Earlier quoted context omitted.

> I find LLMs useful in regurgitating one-liners This was the case for me a year ago. Now Claude or Codex are routinely delivering finished & tested complete features in my projects. I move much, much faster than before and I don’t have an elaborate setup - just a single CLAUDE.md file with some basic information about the project and that’s it.

People keep saying this and I agree Claude has gotten a lot better even in my own experience, but I think the value is questionable. What’s the point of adding features that are inscrutable? I have gotten Claude to make a feature and it mostly works and if it doesn’t work quite right I spend a massive amount of time trying to understand what is going on. For things that don’t matter too much, like prototyping, I thin…

> What’s the point of adding features that are inscrutable?

You are assuming that the additional speed comes at a cost of codebase comprehension. For me it's not the case - I never push generated code I don't fully understand. It does take time, sure, but it still takes me much less time to write a spec, execute with AI and then review than write the thing myself.

Re: Autoresearch on an old research idea

#109

Post by nblintao »

datsci_est_2015 wrote: post #47
andy12_ wrote: post #23

Earlier quoted context omitted.

I think the main value lies in allowing the agent to try many things while you aren't working (when you are sleeping or doing other activities), so even if many tests are not useful, with many trials it can find something nice without any effort on your part. This is, of course, only applicable if doing a single test is relatively fast. In my work a single test can take half a day, so I'd rather not let an agent spen…

Experiments for us cost on the order of tens of dollars, so doing 100 of them every night quickly becomes the price of an entire new employee. And that’s not even including the cost of letting agents run all night. Definitely not in the budget for non-VC-backed companies who aren’t in the AI bubble.

The "price of an entire new employee" framing is spot on. I kept running into the same thing: individual experiments are cheap, but they add up fast, and nobody wants to approve that budget for speculative ideas.

I've been thinking of this as a gap between VC/Kickstarter and just doing it yourself. Most early ML experiments are too small for formal funding but too expensive to casually self-fund. So I built ML Patron where anyone can chip in a few bucks to sponsor an experiment they're curious about. I honestly don't have a good answer yet for how this turns into returns for sponsors in a traditional business sense. For now it's just open research patronage, like "I'd pay to know the answer to this". Platform runs it on cloud GPUs with public MLflow tracking.

Still very early: https://news.ycombinator.com/item?id=47563959.

Post reply on HN