Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The AI Scientist: Towards Automated Open-Ended Scientific Discovery

sakana.ai

Post reply on HN
Print view View original

111–120 of 144 posts

Re: The AI Scientist: Towards Automated Open-Ended Scientific Discovery

#111

Post by adroniser »

agf wrote: post #5

> For example, in one run, it edited the code to perform a system call to run itself. This led to the script endlessly calling itself. In another case, its experiments took too long to complete, hitting our timeout limit. Instead of making its code run faster, it simply tried to modify its own code to extend the timeout period. They go on to say that the solution is sandboxing, but still, this feels like burying the…

Does it really? If you want an LLM to edit code you need to feed it every single line of code in a prompt. Is it really that surprising that having just learnt it has been timed out, and then seeing code that has an explicit timeout in it, it edits it?? This is just a claim about the underlying foundational LLM since the whole science thing is just a wrapper.

I think this bit of it is just a gimmick put in for hype purposes.

Re: The AI Scientist: Towards Automated Open-Ended Scientific Discovery

#112

Post by adroniser »

plaidfuji wrote: post #106

When “executing the experiment” amounts to modifying ~50 lines of PyTorch code tweaking model architecture, I’d bloody well expect that you can automate it. That’s not “automating scientific discovery”, that’s “procedurally optimizing model architecture” (and one iteration of exploration at that!). In any other field of science the actual work and data generated by the AI Scientist would be a sub-section of the Suppo…

I think the whole paper is a satire lol.

Re: The AI Scientist: Towards Automated Open-Ended Scientific Discovery

#113

Post by adroniser »

vouaobrasil wrote: post #47

As someone who truly loves science, the idea of automating the creative parts strikes me at the core as a horrible mistake. Yes, even before AI, we've already tried some automations -- actually some of those I even believe is a bad thing, such as the internet. Most people would disagree no doubt, but I feel like automating science, especially with regard to the more "creative parts" makes it more like an industry, ri…

I completely agree this shit is so depressing. When I saw the AlphaProof paper I basically spent 3 days in mourning basically, because their approach was so simple.

Re: The AI Scientist: Towards Automated Open-Ended Scientific Discovery

#114

Post by Rebuff5007 »

jltsiren wrote: post #105
Rebuff5007 wrote: post #93

Earlier quoted context omitted.

To produce scientific work, one needs to follow the scientific method. This involves stating a hypothesis, designing an experiment that would test this hypothesis, conducting the experiment with controls, and analyzing the data w.r.t to the hypothesis being tested. Access to past works is only useful in informing what is a good hypothesis worth testing. And data is only useful when generated by an experiment that is…

The kind of cargo cult science you describe is the main reason for the replication crisis. The more people believe that they will reach true knowledge by following the sacred rituals to the letter, the more likely they will do things in the established ways without thinking and repeat the same mistakes over and over again. In actual science, the key step is stopping to question yourself all the time. Does the thing y…

Excuse me, what?! How is this cargo cult science? I literally just described the scientific method. Here it is described on Wikipedia:

> The process in the scientific method involves making conjectures (hypothetical explanations), deriving predictions from the hypotheses as logical consequences, and then carrying out experiments or empirical observations based on those predictions.

The other questions you added are not bad questions to ask if you are practical and resource-constrained, but this is besides the point.

https://en.wikipedia.org/wiki/Scientific_method

Re: The AI Scientist: Towards Automated Open-Ended Scientific Discovery

#115

Post by pemrick79 »

Stfu and listen to the music and enjoy it the way you want. Who gives a shit what some tech nerd or stuffy audiophile has to say? Yup only them. You guys laugh at each other think the other is ridiculous. Well we all laugh at both of you. Man what a waste of the last 15 min of my life. Back to enjoying my music anywhere and everywhere I go. Should try it

Re: The AI Scientist: Towards Automated Open-Ended Scientific Discovery

#116

Post by unraveller »

JBorrow wrote: post #61

As someone 'in academia', I worry that tools like this fundamentally discard significant fractions of both the scientific process and why the process is structured that way. The reason that we do research is not simply so that we can produce papers and hence amass knowledge in an abstract sense. A huge part of the academic world is training and building up hands-on institutional knowledge within the population so tha…

The original purpose of science was to get closer to god. Things change.

Arts degrees were also once 7-8 years before the french fought for radical simplification to get them down. There is no law of nature that says PhDs cannot be simplified down more too.

The stupification argument comes kneejerk with every new medium, you don't have to believe everything you watch on TV or form a crutch to a screen oracle. Either way, we need not concern ourselves with the average scientist who fails forgettably by being stupid when the potential upsides are great scientists who succeed with it.

One point of an AI paper farm could just be to increase that discovery hit space a little while you sleep but no one claims it can or will replace every scientist, hopefully only the pessimistic ones.

Re: The AI Scientist: Towards Automated Open-Ended Scientific Discovery

#117

Post by tossandthrow »

swayvil wrote: post #70
JBorrow wrote: post #61

As someone 'in academia', I worry that tools like this fundamentally discard significant fractions of both the scientific process and why the process is structured that way. The reason that we do research is not simply so that we can produce papers and hence amass knowledge in an abstract sense. A huge part of the academic world is training and building up hands-on institutional knowledge within the population so tha…

We already substitute "good authority" (be it consensus or a talking head) for "empirical grounding" all the time. Faith in AI scientific overlords seems a trivial step from there.

Why shouldn't we?

Re: The AI Scientist: Towards Automated Open-Ended Scientific Discovery

#118

Post by jltsiren »

Rebuff5007 wrote: post #114
jltsiren wrote: post #105

Earlier quoted context omitted.

The kind of cargo cult science you describe is the main reason for the replication crisis. The more people believe that they will reach true knowledge by following the sacred rituals to the letter, the more likely they will do things in the established ways without thinking and repeat the same mistakes over and over again. In actual science, the key step is stopping to question yourself all the time. Does the thing y…

Excuse me, what?! How is this cargo cult science? I literally just described the scientific method. Here it is described on Wikipedia: > The process in the scientific method involves making conjectures (hypothetical explanations), deriving predictions from the hypotheses as logical consequences, and then carrying out experiments or empirical observations based on those predictions. The other questions you added are n…

It's cargo cult science precisely because you described the scientific method. Cargo cult science is about focusing on the form without understanding its purpose. Or in Feynman's words:

> So I call these things Cargo Cult Science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.

If you are using the scientific method because it's the scientific method, you are doing cargo cult science. The scientific method is a useful tool in a scientist's toolkit, but it's not appropriate to every situation. And even when it's appropriate, it's not a central part of doing science. You can easily fool yourself using the scientific method, if you don't keep questioning yourself and your work.

Re: The AI Scientist: Towards Automated Open-Ended Scientific Discovery

#119

Post by swayvil »

tossandthrow wrote: post #117
swayvil wrote: post #70

Earlier quoted context omitted.

We already substitute "good authority" (be it consensus or a talking head) for "empirical grounding" all the time. Faith in AI scientific overlords seems a trivial step from there.

Why shouldn't we?

We shouldn't because it isn't science. It's junk epistemology, relatively speaking.

We should because it's a cheap way of managing knowledge in our society.

So there's a tradeoff there.

Re: The AI Scientist: Towards Automated Open-Ended Scientific Discovery

#120

Post by acureau »

agubelu wrote: post #37

I worked with some people who were actively working on this last year, focusing on CS research. The biggest issue was validation. We could get a system to spit out possible research directions automatically, but who decides if they're reasonable and/or promising? A human, of course. Moreover, we gave different humans the same set of hypotheses to validate and they came back with wildly different annotations.

This was exactly what I was thinking. This may be a useful tool for human researchers to drive if it turns out it can generate anything valuable. I can't understand the papers it wrote, much less determine if they make any contributions. I don't think the self-evaluation is going to be too fruitful. Thanks for sharing
Post reply on HN