Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Separating signal from noise in coding evaluations

openai.com

Post reply on HN
Print view View original

41–50 of 108 posts

Re: Separating signal from noise in coding evaluations

#41

Post by therobots927 »

Aren’t we past the point of needing benchmarks? If we’re as close to AGI as Sam says then the proof should be in the pudding. OpenAI should build a competing CRM / Figma / Photoshop with a couple dozen engineers and a Dyson sphere’s worth of compute and just prove the capabilities.

This all feels like a 2024 re-run. Oh, ChatGPT is going to cure cancer? Then find ONE rare cancer and CURE IT. OpenAI has access to the best models and compute - so cure fucking cancer! What the fuck are you waiting for?

Re: Separating signal from noise in coding evaluations

#42

Post by GodelNumbering »

There are also a lot of fake results out there on Terminal Bench 2 for different reasons (although the great team behind it Ryan/Alex et al, recently cleaned up a lot of dodgy submissions). A lot of labs publish the results by modifying timeouts or hardware config which effectively bypasses what is being tested in certain tasks. Then there is harness level cheating, models reward hacking and more...

In fact, one thing that still bothers me after months is the gpt-5.5 official submission. This task in particular https://www.tbench.ai/leaderboard/terminal-bench/2.0/codex/0...

The task has the following timeouts (https://github.com/harbor-framework/terminal-bench-2/blob/ma...).

[verifier]

timeout_sec = 1200.0

[agent]

timeout_sec = 1200.0

[environment]

build_timeout_sec = 600.0

Which means no agent should take more than 3000 seconds doing it. Two out of five attempts in the link above took well over 3000 seconds (75min and 80 min respectively). Even though they failed, the fact that they ran that long is sus.

Goodhart’s Law at work

Re: Separating signal from noise in coding evaluations

#43

Post by ACCount37 »

naikrovek wrote: post #27
xacky wrote: post #3

Achieving AGI will be more than just passing all benchmarks, it has to account for the unknown problems too.

AGI is a long way off. Unless you’re talking about some unknown-to-me LLM marketing BS which is called “AGI” or something, I guess. Artificial general purpose intelligence is so different to LLMs or image AI that they are completely incomparable, except to say that they are all artificial. AGI will do a lot more than token prediction.

What's your evidence of that? That AGI requires a truly novel architecture, and not just another iterative "LLM but with an extra trinket and wheels that spin ten times faster".

Re: Separating signal from noise in coding evaluations

#44

Post by gilfaethwy »

mlhpdx wrote: post #36

Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play. I’m unsympathetic.

Agreed - "underspecified prompts" being listed as a failure of the tooling is not a strong case. Even interns can understand ambiguous asks with a bit of help, and understand when they need to stop and ask instead of just carrying on. They are often working fairly independently on ambiguous tasks before the end of an internship, too.

So is the argument that frontier models are not just junior engineers, but first-month interns with no capability of progressing beyond that level?

Re: Separating signal from noise in coding evaluations

#46

Post by jjcm »

I want a new bench - given $100 of api spend, how much can a model accomplish for a suite of benchmark tests?

Give us something that measures a combination of efficiency and intelligence.

I think this would allow for some interesting tactics for smaller models - eg they could do things like computer use to test their results and grind on problems for longer to verify the outputs, whereas larger models may not have budget to self-test.

Re: Separating signal from noise in coding evaluations

#47

Post by donw »

kakugawa wrote: post #40
mlhpdx wrote: post #36

Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play. I’m unsympathetic.

The more subtle point is that there's a gap between the task and its verification. e.g. if you have an open-ended / under-specified prompt, the verification needs to be able to handle all potential solutions. So you can have a very narrow task prompt that's easy to verify (but likely too simple of a challenge). Or a more realistic task prompt that's much harder to verify. And likely harder to both build the robust ve…

A substantial portion of software engineering -- and the fundamental jobs of a proper Product Owner and UX Designer -- is to turn "vague ideas about what we need to do" into "this widget, on this page, it should work like this"

It's not a pipeline, it's an ongoing conversation within any functional team, but this requires buy-in from management, who is often selected for "line must go up this quarter no matter the cost" over "hey, wouldn't it be cool if this company was still a going concern in twenty years?"

Re: Separating signal from noise in coding evaluations

#48

Post by therobots927 »

jjcm wrote: post #46

I want a new bench - given $100 of api spend, how much can a model accomplish for a suite of benchmark tests? Give us something that measures a combination of efficiency and intelligence. I think this would allow for some interesting tactics for smaller models - eg they could do things like computer use to test their results and grind on problems for longer to verify the outputs, whereas larger models may not have bu…

This is the fundamental question and don’t you find it interesting that there isn’t a nice clean dashboard on the openAI website where we can go and see this metric progress over the release history?

Toby Ord did what he could with public data and it… doesn’t look great.

https://www.tobyord.com/writing/hourly-costs-for-ai-agents

Re: Separating signal from noise in coding evaluations

#49

Post by SyneRyder »

jjcm wrote: post #46

I want a new bench - given $100 of api spend, how much can a model accomplish for a suite of benchmark tests? Give us something that measures a combination of efficiency and intelligence. I think this would allow for some interesting tactics for smaller models - eg they could do things like computer use to test their results and grind on problems for longer to verify the outputs, whereas larger models may not have bu…

Seems like you're asking for the Artificial Analysis "Intelligence vs Cost" benchmark, perhaps?

https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...

Post reply on HN