Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

terminal-bench-science.ai

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

#31

Post by overdrive110 »

saithound wrote: post #17
johnnyApplePRNG wrote: post #9

Not surprised to see Claude significantly higher in scientific intelligence than Sol. You can tell that Claude really does grasp a wide array of highly specific scientific and mathematical nuances... where's codex is just basically for coding and that's it. That's the feel I get from the both of them anyways and I've used both on the 20x plan for the past week at length. Don't get me wrong, codex is great at finding…

> You can tell that Claude really does grasp a wide array of highly specific scientific and mathematical nuances... where's codex is just basically for coding and that's it. If you have time, can you elaborate or give some examples of mathematical nuances? I am evaluating Sol and Fable on a fairly large dataset of subtly flawed informal mathematical arguments (task is to identify and name propositions with substantia…

Is this benchmark public? Anecdotally I have had decent results asking Sol to nitpick my proofs (mostly probability theory but nothing super dense). I have never tried Claude seriously, so I am very curious about what the failures look like with Fable.

Re: Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

#33

Post by jampekka »

jubilanti wrote: post #26

No. Please no. I don't want science vibecoded. Software can rely on layers of testing and verification and most code is applying decades-old patterns to a customer's donain and gluing libraries together until they click. That simply don't work when you're on the frontier of something entirely new. There was already a glut of slop science before 2024. We don't need even more science-shaped slop clogging the peer revie…

I generally trust vibe code a lot more than researcher code.

Re: Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

#34

Post by walrus01 »

j_maffe wrote: post #29

The tasks are the thing to really look at here: https://github.com/harbor-framework/terminal-bench-science/t...

I am hoping someone with more free time than myself can contribute some things in the RF engineering domain in the 'engineering-sciences' section. There's some problems out there that will definitely stump even a smart LLM.

Re: Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

#35

Post by frozenseven »

jubilanti wrote: post #26

No. Please no. I don't want science vibecoded. Software can rely on layers of testing and verification and most code is applying decades-old patterns to a customer's donain and gluing libraries together until they click. That simply don't work when you're on the frontier of something entirely new. There was already a glut of slop science before 2024. We don't need even more science-shaped slop clogging the peer revie…

"Vibecoded" science is well on its way to being better than your "real Science™". Withholding the future is not an option.

Re: Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

#36

Post by anmolkabra »

overdrive110 wrote: post #31
saithound wrote: post #17

Earlier quoted context omitted.

> You can tell that Claude really does grasp a wide array of highly specific scientific and mathematical nuances... where's codex is just basically for coding and that's it. If you have time, can you elaborate or give some examples of mathematical nuances? I am evaluating Sol and Fable on a fairly large dataset of subtly flawed informal mathematical arguments (task is to identify and name propositions with substantia…

Is this benchmark public? Anecdotally I have had decent results asking Sol to nitpick my proofs (mostly probability theory but nothing super dense). I have never tried Claude seriously, so I am very curious about what the failures look like with Fable.

Yes! All tasks are on github: https://github.com/harbor-framework/terminal-bench-science. They were contributed through PRs so the discussion and reviewing (before tasks were accepted) is also fully public.

Re: Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

#37

Post by anmolkabra »

CJefferson wrote: post #21

I worry this doesn’t check correctness. I’ve been finding Claude is lately awful at folllowing instructions, I’ll ask it to implement the algorithm from a paper and it will do something simpler and slower and when challenged do it’s stupid apology thing. It can’t be trusted with anything I’d put in a paper, it lies too much. 4.6 couldn’t do as complex tasks, but it would do what was asked.

Benchmark does check correctness. Team is working to write a paper that will likely contain failure mode analysis; checking for instruction following could be a good idea.

Agents are given instructions in markdown format, allowed to read data and libraries sandboxed in a Docker container, and evaluated on deterministic pytests on the outcome. Two things the team aimed to enforce to add tasks we could trust: - Scientific workflows are often simulations that are correct upto numerical tolerances (the scientist decides what's reasonable), so task verifiers' evaluate results of agent-written code within the tolerances. - There could be multiple solution codes to a scientific workflow, and the team tried to ensure the verifier tests accommodate those. Not overfit to the oracle reference code, written by the scientist.

Instruction following is implicitly assumed, if the model gives up and doesn't complete the task it counts as a failure because the verifier tests fail.

Re: Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

#38

Post by jvwww »

WithinReason wrote: post #23
johnnyApplePRNG wrote: post #9

Not surprised to see Claude significantly higher in scientific intelligence than Sol. You can tell that Claude really does grasp a wide array of highly specific scientific and mathematical nuances... where's codex is just basically for coding and that's it. That's the feel I get from the both of them anyways and I've used both on the 20x plan for the past week at length. Don't get me wrong, codex is great at finding…

So why is Sol the one solving Erdős problems?

Yeah exactly - found the parent comment quite funny.

For instance, one of the other top comments berates Claude for terrible instruction following with regards to scientific papers, whereas this one is full of praise.

Everyone is just making up their thoughts on these models based on vibes.

Re: Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

#39

Post by mullenba »

frozenseven wrote: post #35
jubilanti wrote: post #26

No. Please no. I don't want science vibecoded. Software can rely on layers of testing and verification and most code is applying decades-old patterns to a customer's donain and gluing libraries together until they click. That simply don't work when you're on the frontier of something entirely new. There was already a glut of slop science before 2024. We don't need even more science-shaped slop clogging the peer revie…

"Vibecoded" science is well on its way to being better than your "real Science™". Withholding the future is not an option.

I'm sure there will be some golden nuggets in the petabytes of slop that are about to be unleashed.

The signal to noise ratio will be so bad that all of science will suffer.

Re: Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

#40

Post by frozenseven »

mullenba wrote: post #39
frozenseven wrote: post #35

Earlier quoted context omitted.

"Vibecoded" science is well on its way to being better than your "real Science™". Withholding the future is not an option.

I'm sure there will be some golden nuggets in the petabytes of slop that are about to be unleashed. The signal to noise ratio will be so bad that all of science will suffer.

Every doomer claim of this sort has not only failed to come true, but the exact opposite has happened. For instance, see the tsunami of new incredible results in mathematics.
Post reply on HN