Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Top model scores may be skewed by Git history leaks in SWE-bench

github.com

Post reply on HN
Print view View original

131–140 of 167 posts

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#131

Post by sigmoid10 »

zelphirkalt wrote: post #129

Can anyone tell me what is the difficulty in simply not having .git at all during a benchmark run? Why not simply remove anything that is not the code the benchmark runs on? Or just simple oversight?

Coding agents are so powerful because they are not just looking at static code. Looking through git histories is a valid method for humans to solve certain kinds of bugs, so it makes sense that models should also be able to do that too. And realistically, a lot of modern production code will have git information, so it's not like this wouldn't be a common real world application.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#132

Post by diggan »

sigmoid10 wrote: post #131
zelphirkalt wrote: post #129

Can anyone tell me what is the difficulty in simply not having .git at all during a benchmark run? Why not simply remove anything that is not the code the benchmark runs on? Or just simple oversight?

Coding agents are so powerful because they are not just looking at static code. Looking through git histories is a valid method for humans to solve certain kinds of bugs, so it makes sense that models should also be able to do that too. And realistically, a lot of modern production code will have git information, so it's not like this wouldn't be a common real world application.

I think this issue is specifically about the agents looking at "future repository state" (according to the linked issue at least), so while looking at the history might be a normal method for solving issues, running `git log --all` to take a peek at the future which already includes the fix isn't very typical (yet?).

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#135

Post by numbsafari »

cjsaltlake wrote: post #62
bflesch wrote: post #53

Earlier quoted context omitted.

> This is a natural part of running a benchmark, I'm sure tiny things like this will keep on getting discovered and we'll keep on fixing them. You're all extremely clever and I can't seem to understand how you missed thinking about such a simple edge case. It's like building a chroot and then allowing `cd ..` to break out of it. What other maybe extremely basic edge cases were missed? > This doesn't change the overal…

I'm also on the SWE-bench team. This was simply a classic bug. We had code before that we believed was sufficient to hide / remove future GitHub history and it turns out it was not. We've patched it.

Your classic bug is being used as justification to destroy the careers and lives of tens of thousands of people. Read the room.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#136

Post by haskellshill »

stronglikedan wrote: post #64
ares623 wrote: post #56

Earlier quoted context omitted.

They really did a "trust me bro" and "do your own research" huh

the strange thing to me is that people would have it any other way. if you don't trust someone, why would you trust them to do the research for you? bit of entitlement if you ask me

That's silly. If they show their work I won't have to trust them. Compare answering "The answer is 5, just compute it yourself." on a math test, vs. actually showing the calculation. The former clearly implies the person doesn't know what they're talking about.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#137

Post by kronks »

yorwba wrote: post #21
stefan_ wrote: post #5

Earlier quoted context omitted.

So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all. I don't get it, who is so opposed to doing the bare minimum of manual work and check what these models are doing? At least back in the day grad students doing an easy meta-paper understood it meant doing some repetitive manual work. Now we got benchmarks by hype vendors who think they can use the thing they are benchmarking to .. mark the b…

The "Verified" part of "SWE-Bench Verified" means that there was plain "SWE-Bench" before it, which had actually not been verified at all and included a lot of tasks that didn't really make sense for use as a benchmark: https://openai.com/index/introducing-swe-bench-verified/#ada... Data contamination stemming from the fact that it's based on already-solved problems in public repositories is a different issue that ca…

[dead]

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#138

Post by hamonrye34 »

plumb_bob_00 wrote: post #66
ofirpress wrote: post #48

Earlier quoted context omitted.

[flagged]

If you are going to represent your team in public, you owe them better than a response like this.

This is contingent on whether SWE N-class frontier models can do deep packet inspection.

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#139

Post by skeezyboy »

bryan0 wrote: post #78

hah the model should get extra credit for discovering this! > Now I understand the situation perfectly! The issue described in the problem statement is a real bug that was already identified and fixed in later versions of pytest. Since we're working with pytest 5.2.4, we need to apply the same fix. https://gist.github.com/jacobkahn/bd77c69d34040a9e9b10d56baa...

> hah the model should get extra credit for discovering this!

you whoever included it in the training data should get the credit

Re: Top model scores may be skewed by Git history leaks in SWE-bench

#140

Post by skeezyboy »

ripped_britches wrote: post #105
pessimizer wrote: post #103

Earlier quoted context omitted.

Because they are. But stochastic parrots are awesome.

I challenge you! Try giving this exact prompt to GPT-5-Thinking (medium or high reasoning if API). It is able to (without external code tools) solve a never before seen cypher that is not present in its training data. I think this pretty clearly demonstrates that the “stochastic parrot” is no longer an apt description of its capabilities in generalization: ———— You are given a character-by-character decode table `map…

{ "decoded_prefix": "nxcznchvhvvrddqinqtrrqdboctzzimxmhlyflcjfjapponydzwkxdtdehldmodizslzl", "last_10": "sxmb", "vowel_counts": { "a": 10, "e": 6, "i": 13, "o": 13, "u": 6 } }

took about 2 seconds, must have had it cached

Post reply on HN