Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Top model scores may be skewed by Git history leaks in SWE-bench

github.com

Post reply on HN
Print view View original

141–150 of 167 posts

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#141

Post by fp64 »

sigmoid10 wrote: post #131
zelphirkalt wrote: post #129

Can anyone tell me what is the difficulty in simply not having .git at all during a benchmark run? Why not simply remove anything that is not the code the benchmark runs on? Or just simple oversight?

Coding agents are so powerful because they are not just looking at static code. Looking through git histories is a valid method for humans to solve certain kinds of bugs, so it makes sense that models should also be able to do that too. And realistically, a lot of modern production code will have git information, so it's not like this wouldn't be a common real world application.

Well, there's legacy code and/or horrible git history that also needs fixing at some point. Also I have witnessed how the history can send you down a wrong path. I don't agree that this is a good argument.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#142

Post by hhh »

jsheard wrote: post #8
stefan_ wrote: post #5

Earlier quoted context omitted.

So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all. I don't get it, who is so opposed to doing the bare minimum of manual work and check what these models are doing? At least back in the day grad students doing an easy meta-paper understood it meant doing some repetitive manual work. Now we got benchmarks by hype vendors who think they can use the thing they are benchmarking to .. mark the b…

> So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all. Seems on-brand for an LLM-related thing to claim that it has verified something without actually checking.

Verified has a completely different meaning for this, it's that the questions have verified valid solutions.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#143

Post by diggan »

Workaccount2 wrote: post #47
slacktivism123 wrote: post #12

Fascinating case showing how LLM promoters will happily take "verified" benchmarks at their word. It's easy to publish "$NEWMODEL received an X% bump in SWE-Bench Verified!!!!". Proper research means interrogating the traces, like these researchers did (the Gist shows Claude 4 Sonnet): https://gist.github.com/jacobkahn/bd77c69d34040a9e9b10d56baa... Commentary: https://x.com/bwasti/status/1963288443452051582 , https:/…

The best benchmark is the community vibe in the weeks following a release. Claude benchmarks poorly but vibes well. Gemini benchmarks well and vibes well. Grok benchmarks well but vibes poorly. (yes I know you are gushing with anecdotes, the vibes are simply the approximate color of gray born from the countless black and white remarks.)

> The best benchmark is the community vibe in the weeks following a release.

True, just be careful what community you use as a vibe-check. Most of the mainstream/big ones around AI and LLMs basically have influence campaigns run against them, are made of giant hive-minds that all think alike and you need to carefully asses if anything you're reading is true or not, and votes tend to make it even worse.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#144

Post by DrScientist »

robcohen wrote: post #100
coldtea wrote: post #87

Earlier quoted context omitted.

Different, but probably not as orthogonal as one might think. E.g. cooperating ethics had been necessary for the further development of human populations intelligence (and culture, technology, material wealth, nutrition etc that lead to further increases in intelligence). So lack of ethics might be a sign of intelligence, but it's also a parasitic intelligence that benefits the individual, and beyond certain level an…

Aren't there only two rules that all groups follow in the animal kingdom? - don't lie too often - don't kill members of the in group Seems like these would be required for any group to survive, which makes sense why they are universal. All other rules/ethics seem to be dependent on resource scarcity.

Groups don't follow rules as such, group behaviours emerge from the interaction of individual behaviours.

As to whether all groups display those rules - I suspect not - though it rather does depend on how you define a group - the definition of group probably has some sort of colloboration built in ( as oppose to a bunch of indviduals that happen to live in the same geographic area ).

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#145

Post by ActionHank »

sigmoid10 wrote: post #131
zelphirkalt wrote: post #129

Can anyone tell me what is the difficulty in simply not having .git at all during a benchmark run? Why not simply remove anything that is not the code the benchmark runs on? Or just simple oversight?

Coding agents are so powerful because they are not just looking at static code. Looking through git histories is a valid method for humans to solve certain kinds of bugs, so it makes sense that models should also be able to do that too. And realistically, a lot of modern production code will have git information, so it's not like this wouldn't be a common real world application.

That is a weak argument.

The point is to benchmark against a human solving a problem. Typically these problems are posed as a question or a blank project, without that history.

You are arguing for a an apples to oranges comparison because the LLM performs better. Rather than a realistic comparison.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#146

Post by sigmoid10 »

ActionHank wrote: post #145
sigmoid10 wrote: post #131

Earlier quoted context omitted.

Coding agents are so powerful because they are not just looking at static code. Looking through git histories is a valid method for humans to solve certain kinds of bugs, so it makes sense that models should also be able to do that too. And realistically, a lot of modern production code will have git information, so it's not like this wouldn't be a common real world application.

That is a weak argument. The point is to benchmark against a human solving a problem. Typically these problems are posed as a question or a blank project, without that history. You are arguing for a an apples to oranges comparison because the LLM performs better. Rather than a realistic comparison.

You apparently don't know what SWE-bench is [1]. First of all, it tries to evaluate skills that explicitly go beyond blank project questions with given solutions. Secondly, it does not contain "optimal" or sometimes even correct solutions. That's because it uses real world software development examples from actual PRs in popular repos. These very likely had humans use all the tools at their disposal as well (e.g. web search, git commands, code execution). Assuming an LLM could have solved these just by looking at a piece of code turns out to be very myopic.

[1] https://arxiv.org/html/2310.06770v3

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#147

Post by theblazehen »

diggan wrote: post #143
Workaccount2 wrote: post #47

Earlier quoted context omitted.

The best benchmark is the community vibe in the weeks following a release. Claude benchmarks poorly but vibes well. Gemini benchmarks well and vibes well. Grok benchmarks well but vibes poorly. (yes I know you are gushing with anecdotes, the vibes are simply the approximate color of gray born from the countless black and white remarks.)

> The best benchmark is the community vibe in the weeks following a release. True, just be careful what community you use as a vibe-check. Most of the mainstream/big ones around AI and LLMs basically have influence campaigns run against them, are made of giant hive-minds that all think alike and you need to carefully asses if anything you're reading is true or not, and votes tend to make it even worse.

I generally check LM Arena as well as which models have had the most weekly tokens on openrouter

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#148

Post by zelphirkalt »

sigmoid10 wrote: post #146
ActionHank wrote: post #145

Earlier quoted context omitted.

That is a weak argument. The point is to benchmark against a human solving a problem. Typically these problems are posed as a question or a blank project, without that history. You are arguing for a an apples to oranges comparison because the LLM performs better. Rather than a realistic comparison.

You apparently don't know what SWE-bench is [1]. First of all, it tries to evaluate skills that explicitly go beyond blank project questions with given solutions. Secondly, it does not contain "optimal" or sometimes even correct solutions. That's because it uses real world software development examples from actual PRs in popular repos. These very likely had humans use all the tools at their disposal as well (e.g. web…

This does not present a good understanding of how to test machine learning models. If you let the "labels" be visible at test time, your test is worthless. Your analogy of "real world software development" doesn't hold up. In real world software development, I don't have magical commits from the future, that I merely need to check out and look at to see the solution.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#149

Post by jbellis »

lostmsu wrote: post #121
jbellis wrote: post #114

Earlier quoted context omitted.

no, I'm pretty skeptical that it's better than qwen3 coder but if you have evidence that it could be, I'm down to test it

It has the same score on https://lmarena.ai/leaderboard/webdev , but AFAIK Air version is much smaller.

I've added results for GLM 4.5 and 4.5 Air.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#150

Post by sigmoid10 »

zelphirkalt wrote: post #148
sigmoid10 wrote: post #146

Earlier quoted context omitted.

You apparently don't know what SWE-bench is [1]. First of all, it tries to evaluate skills that explicitly go beyond blank project questions with given solutions. Secondly, it does not contain "optimal" or sometimes even correct solutions. That's because it uses real world software development examples from actual PRs in popular repos. These very likely had humans use all the tools at their disposal as well (e.g. web…

This does not present a good understanding of how to test machine learning models. If you let the "labels" be visible at test time, your test is worthless. Your analogy of "real world software development" doesn't hold up. In real world software development, I don't have magical commits from the future, that I merely need to check out and look at to see the solution.

This was never in question and an actual developer has already answered in another comment why this is not as big of an issue as the title might suggest. The question in this chain was about the value of additional data, which is perfectly clear if you're familiar with this particular benchmark.
Post reply on HN