Top model scores may be skewed by Git history leaks in SWE-bench
1–10 of 167 posts
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#2Not “may be”: just look how swe-bench scores drop to single digits once it in C#
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#3Very interested to see the updated results. This could really shake up the leaderboard.
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#4Very interested to see the updated results. This could really shake up the leaderboard.
I hope it does. These coding benchmarks have often seemed frustratingly out of touch with my experience.
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#5Not “may be”: just look how swe-bench scores drop to single digits once it in C# https://arxiv.org/html/2506.12286v3
So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all.
I don't get it, who is so opposed to doing the bare minimum of manual work and check what these models are doing? At least back in the day grad students doing an easy meta-paper understood it meant doing some repetitive manual work. Now we got benchmarks by hype vendors who think they can use the thing they are benchmarking to .. mark the bench.
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#6Not “may be”: just look how swe-bench scores drop to single digits once it in C# https://arxiv.org/html/2506.12286v3
I was going to argue "LLM's need code samples to-do well on languages and if we are honest C# is a language mostly held in private repo's" but Github's 2024 report[0] says its the 5th most used language (I'm to lazy to check if this report includes private repo's but I'll assume it doesn't).
So kinda neat to see this paper!
[0]https://github.blog/news-insights/octoverse/octoverse-2024/#...
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#7I speculate something similar (or even worse) is going on with Terminal-Bench [1].
Like, seriously, how come all these agents are beating Claude Code? In practice, they are shitty and not even close. Yes. I tried them.
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#8Not “may be”: just look how swe-bench scores drop to single digits once it in C# https://arxiv.org/html/2506.12286v3
So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all. I don't get it, who is so opposed to doing the bare minimum of manual work and check what these models are doing? At least back in the day grad students doing an easy meta-paper understood it meant doing some repetitive manual work. Now we got benchmarks by hype vendors who think they can use the thing they are benchmarking to .. mark the b…
> So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all.
Seems on-brand for an LLM-related thing to claim that it has verified something without actually checking.
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#9It's honestly ridiculous they left git history lying around during a benchmark, and this benchmark made to ICLR in Jan 2024 and no one has detected this issue until now. I don't really trust any benchmarking or tools or claims from this space when they can make such huge basic errors.
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#10Not “may be”: just look how swe-bench scores drop to single digits once it in C# https://arxiv.org/html/2506.12286v3
Personally I don't look at or respect LLM benchmarks at all. I've seen SOTA models fail in incredibly shocking ways even recently. Those moments immediately bring me out of the delusion that LLMs have thinking capacity or an understanding of code.