Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autoresearch on an old research idea

ykumar.me

Post reply on HN
Print view View original

81–90 of 110 posts

Re: Autoresearch on an old research idea

#81

Post by make3 »

ykumards wrote: post #70
1970-01-01 wrote: post #37

> The original paper used several medical X-ray datasets which I don’t have access to anymore, so I needed a new dataset with spatial annotations to test the expert attention mechanism. I picked the Ukiyo-eVG dataset: ~11K Japanese woodblock prints That's such a weird switch. There's lots of free medical imaging online. Example: https://www.cancerimagingarchive.net/

That’s true! It felt a bit flippant to give medical data to an agent. Also, I wanted to see if the model would work in other domains!

but doesn't it break the assumption that it should ideally be able to reproduce your original results

Re: Autoresearch on an old research idea

#83

Post by ykumards »

make3 wrote: post #81
ykumards wrote: post #70

Earlier quoted context omitted.

That’s true! It felt a bit flippant to give medical data to an agent. Also, I wanted to see if the model would work in other domains!

but doesn't it break the assumption that it should ideally be able to reproduce your original results

IMO it would be hard to reproduce the results using autoresearch setup.

To get CLIP to work properly we typically need large batch sizes. So the experiments in the original paper were quite heavy, and ran parallel across 8 GPUs.

Re: Autoresearch on an old research idea

#84

Post by ricksunny »

With all the posts lately about Karpathy's autoresearch, it remains unclear to me whether this name is intended to convey that this LLM-codebase should be useful for research across all domains - like molecular biology, aircraft control, sociological, ww2 history, etc. or is it intended only to discover new LLM capabilities.

Re: Autoresearch on an old research idea

#85

Post by Xx_crazy420_xX »

Autoresearch is nothing new, big players are already in the game with more sophisticated solutions:

  - https://arxiv.org/abs/2602.02660 (MARS)
  - https://arxiv.org/abs/2601.14525 (Execution-grounded automated AI research)
  - https://arxiv.org/abs/2601.10402 (ML-Master 2.0)
The mostly used benchmark for automated AI engineering/ research is: https://github.com/openai/mle-bench

Re: Autoresearch on an old research idea

#86

Post by stingraycharles »

DoctorOetker wrote: post #22
dvt wrote: post #14

Ok, so looking at the commit log[1], I was mostly interested in seeing what the "moonshot ideas" implementations looked like, but basically everything is just hyperparameter tuning. Which is nice, but likely not worth the $$$ spent on the tokens. Am I missing something here? [1] https://github.com/ykumards/eCLIP/commits/main/autoresearch

It would seem wise to modify the autoresearch instructions to first estimate the computational costs rigorously and then sort and compare the proposals for human review, and for each actually executed attempt to feed back the computational costs with LoRa adapter? i.e. perhaps minimal changes to autoresearch can take control for cost-effective research to occur.

Yes but at that point you may as well use a proper hyperparameter tuning framework like optuna if all the LLM agent is supposed to do is do hyperparameter tuning.

Re: Autoresearch on an old research idea

#87

Post by bluequbit »

Xx_crazy420_xX wrote: post #85

Autoresearch is nothing new, big players are already in the game with more sophisticated solutions: - https://arxiv.org/abs/2602.02660 (MARS) - https://arxiv.org/abs/2601.14525 (Execution-grounded automated AI research) - https://arxiv.org/abs/2601.10402 (ML-Master 2.0) The mostly used benchmark for automated AI engineering/ research is: https://github.com/openai/mle-bench

The thing is, autoresearch feels more accessible that the listed solutions. I can use it trivially on virtually any problem that has verifiable rewards and a feedback loop.

Re: Autoresearch on an old research idea

#89

Post by gf000 »

datsci_est_2015 wrote: post #47
andy12_ wrote: post #23

Earlier quoted context omitted.

I think the main value lies in allowing the agent to try many things while you aren't working (when you are sleeping or doing other activities), so even if many tests are not useful, with many trials it can find something nice without any effort on your part. This is, of course, only applicable if doing a single test is relatively fast. In my work a single test can take half a day, so I'd rather not let an agent spen…

Experiments for us cost on the order of tens of dollars, so doing 100 of them every night quickly becomes the price of an entire new employee. And that’s not even including the cost of letting agents run all night. Definitely not in the budget for non-VC-backed companies who aren’t in the AI bubble.

The costs keep decreasing and self-hosted models may be able to do some of the tasks as well.

So this may be only temporarily unavailable for many.

Re: Autoresearch on an old research idea

#90

Post by baxtr »

bluequbit wrote: post #87
Xx_crazy420_xX wrote: post #85

Autoresearch is nothing new, big players are already in the game with more sophisticated solutions: - https://arxiv.org/abs/2602.02660 (MARS) - https://arxiv.org/abs/2601.14525 (Execution-grounded automated AI research) - https://arxiv.org/abs/2601.10402 (ML-Master 2.0) The mostly used benchmark for automated AI engineering/ research is: https://github.com/openai/mle-bench

The thing is, autoresearch feels more accessible that the listed solutions. I can use it trivially on virtually any problem that has verifiable rewards and a feedback loop.

People underestimate UX and accessibility. The iPhone was nothing new.
Post reply on HN