Some critical issues with the SWE-bench dataset
1–10 of 121 posts
Re: Some critical issues with the SWE-bench dataset
#2> 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments.
Is this what Hofstadter means by a strange-loop?
Re: Some critical issues with the SWE-bench dataset
#3> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%.
This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.
Re: Some critical issues with the SWE-bench dataset
#4> 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments. Is this what Hofstadter means by a strange-loop?
Turns out "AI deep research reasoning agent" was just "we can print the training set"
Re: Some critical issues with the SWE-bench dataset
#5> 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments. Is this what Hofstadter means by a strange-loop?
Turns out "AI deep research reasoning agent" was just "we can print the training set"
...by piping it through the world's most inefficient echo function.
Re: Some critical issues with the SWE-bench dataset
#6So what we need is something like a versioned crowdsourced coding LLM eval dataset.
Every quarter, you have a couple thousand volunteers provide 2 GitHub issues from the past 3 months, which are nontrivial to resolve, and where there exists strong test cases. Each volunteer then cross-checks 2 issues from other volunteers. The volunteers get 1 month free subscription to some AI service in return.
This dataset is then published as SWE-UberBench-2025-02 or something. People can then only evaluate their coding LLM on datasets published after their training period.
Re: Some critical issues with the SWE-bench dataset
#7I am shocked—shocked—when a vendor cheats in order to increase their benchmark scores.
I always tell my customers to ignore benchmarks and compare outcomes with their own workloads. Benchmarks are almost completely useless in the real world.
Re: Some critical issues with the SWE-bench dataset
#8You need benchmarks with the following three properties:
1) No known solutions, so there's no "ground truth" dataset to train on
2) Presumably hard to solve
3) But easy to verify a solution if one is provided.
This, of course, is easier done on the STEM side of things, but how do you automatically test creativity, or philosophical aptitude?
Re: Some critical issues with the SWE-bench dataset
#9Earlier quoted context omitted.
Turns out "AI deep research reasoning agent" was just "we can print the training set"
...by piping it through the world's most inefficient echo function.
You should immediately publish a paper on Arvix with your revolutionary IEF brand, an improvement on transformers and mamba architectures. Then, like Ilya, take $1B in funding the following week.
Re: Some critical issues with the SWE-bench dataset
#10> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.
Yep anecdotally that's basically spot-on. It's also one of the reasons that I still find copilot vastly more useful than highly autonomous AI tooling (cursor, roocode, avante, etc.)