In the meawhile, Oracle stock went up 40% in one one day, based on what Wall Street thinks AI might be...in 4 years...Not a bubble at all...
Top model scores may be skewed by Git history leaks in SWE-bench
11–20 of 167 posts
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#12Fascinating case showing how LLM promoters will happily take "verified" benchmarks at their word.
It's easy to publish "$NEWMODEL received an X% bump in SWE-Bench Verified!!!!".
Proper research means interrogating the traces, like these researchers did (the Gist shows Claude 4 Sonnet): https://gist.github.com/jacobkahn/bd77c69d34040a9e9b10d56baa...
Commentary: https://x.com/bwasti/status/1963288443452051582, https://x.com/tmkadamcz/status/1963996138044096969
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#13Man I feel so dumb. Why haven't I been doing this in my job, if I could just see the commit that fixed my issue this would all be so easy.
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#14I'm not surprised. People really thought the models just kept getting better and better?
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#15Earlier quoted context omitted.
So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all. I don't get it, who is so opposed to doing the bare minimum of manual work and check what these models are doing? At least back in the day grad students doing an easy meta-paper understood it meant doing some repetitive manual work. Now we got benchmarks by hype vendors who think they can use the thing they are benchmarking to .. mark the b…
> So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all. Seems on-brand for an LLM-related thing to claim that it has verified something without actually checking.
that was my exact thought. how fitting
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#16Man I feel so dumb. Why haven't I been doing this in my job, if I could just see the commit that fixed my issue this would all be so easy.
Someone did comment that it's actually smart to check if something is fixed on the unstable branch, or I suppose in your coworkers' branches. A good task for an LLM.
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#17Not “may be”: just look how swe-bench scores drop to single digits once it in C# https://arxiv.org/html/2506.12286v3
I was going to argue "LLM's need code samples to-do well on languages and if we are honest C# is a language mostly held in private repo's" but Github's 2024 report[0] says its the 5th most used language (I'm to lazy to check if this report includes private repo's but I'll assume it doesn't). So kinda neat to see this paper! [0] https://github.blog/news-insights/octoverse/octoverse-2024/#...
5th most used language based on private repos that the group making the report has the exclusive direct access to seeing
I don't see that contradicting your assumption
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#18In the meawhile, Oracle stock went up 40% in one one day, based on what Wall Street thinks AI might be...in 4 years...Not a bubble at all...
The real bubble will come once interest rates start dropping.
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#19[I'm on the SWE-bench team] Multiple people have looked into this, for example right in that thread: https://github.com/SWE-bench/SWE-bench/issues/465#issuecomme...
This issue had affected a tiny fraction of existing agents in a tiny fraction of their runs. And we've now issued a fix.
This is a natural part of running a benchmark, I'm sure tiny things like this will keep on getting discovered and we'll keep on fixing them. This doesn't change the overall picture or trends at all.
Re: Top model scores may be skewed by Git history leaks in SWE-bench
#20Very interested to see the updated results. This could really shake up the leaderboard.
I hope it does. These coding benchmarks have often seemed frustratingly out of touch with my experience.
Because I would argue there is no benchmark to rule them all. It highly depends on individual use cases.