Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-bench Verified no longer measures frontier coding capabilities

openai.com

Post reply on HN
Print view View original

71–80 of 209 posts

Re: SWE-bench Verified no longer measures frontier coding capabilities

#71

Post by Escapado »

Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

I agree with the sentiment but I wonder if a sufficiently large amount of sufficiently sophisticated benchmarks existed then I would be surprised if a model would only memorize those benchmarks while showing terrible real world performance. We are not there yet but maybe one day we will be.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#72

Post by adamandsteve »

Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

"The community" is astroturfed as hell though. Anthropic pays influencers to promote Claude Code and likely bots a ton as well, so it's hard to come to any kind of consensus online. Even if everyone was acting in good faith, some people will have a much better experience than others because of the domain they're working in (e.g. AI being much better at frontend and commonly used libraries).

The only real way to evaluate a model is to test it yourself but that's exhausting for each new model and not comprehensive anyway.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#74

Post by mnky9800n »

kqr wrote: post #40
mnky9800n wrote: post #29

Earlier quoted context omitted.

This is why I made Zork bench. Zork, the text adventure game, is in the training data for LLMs. It’s also deterministic. Therefore it should be easy for an LLM to play and complete. Yet they don’t. Understanding why is the goal of Zork bench. https://github.com/mnky9800n/zork-bench

I have worked on similar problems. See e.g. [1]. The LLMs I have tested have terrible world models and intuitions for how actions change the environment. They're also not great at discerning and pursuing the right goals. They're like an infinitely patient five-year old with amazing vocabulary. [1]: https://entropicthoughts.com/updated-llm-benchmark (more descriptions available in earlier evaluations referenced from t…

we should talk. i sent you an email.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#75

Post by cbg0 »

Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

> let the community decide

Which community are we talking about? The professionals with 10+ years experience using LLMs, the vibe coders that have no experience writing code and everyone in between? If you read some of the online communities the experiences with the models all over the place, some compare GPT 5.5 to the second coming of JC while others think it's stupider than 5.4.

I personally don't have time to build a set of private benchmarks to compare the models that are coming out so I'm mostly relying on private and semi-private benchmarks to get a feel for how models are improving before I subscribe to a service and start using it myself. At least it's something a bit more reliable than the vibes of random people and bots on reddit.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#77

Post by CamperBob2 »

mnky9800n wrote: post #29
Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

This is why I made Zork bench. Zork, the text adventure game, is in the training data for LLMs. It’s also deterministic. Therefore it should be easy for an LLM to play and complete. Yet they don’t. Understanding why is the goal of Zork bench. https://github.com/mnky9800n/zork-bench

Actually the Zorks weren't deterministic, especially Zork II. The Wizard could F you over pretty badly if he appeared at an inopportune time.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#80

Post by [deleted] »

LukaD wrote: post #7
w4yai wrote: post #2

I don't understand these websites which force translation to my native language. I mean, it's fine as it's useful for many people, but where is the button for disabling it ? Or why is it enabled by default ? "codage de pointe" sounds so weird and cringe in French.

Does your browser request French via an Accept-Language header perhaps? What really infuriates me is when sites don’t respect that header and give you a translation based on IP location.

[deleted]
Post reply on HN