Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some critical issues with the SWE-bench dataset

arxiv.org

Post reply on HN
Print view View original

41–50 of 121 posts

Re: Some critical issues with the SWE-bench dataset

#41

Post by aprilthird2021 »

jandrese wrote: post #30
bearjaws wrote: post #24

I would argue almost every popular benchmark quoted by the big LLM companies is tainted. OAI, xAI, Antropic, Google all score incredibly well, then you go to try and write code and its just okay . They claim it can do PHD level reasoning, but here I am not trusting it on basic computational thinking.

Yeah, that's true in many fields with these AI agents. They demo well, but when you put them to actual work they fall right on their face. Even worse, the harder the task you set for them the more they lie to you. It's like hiring a junior dev from one of those highly regimented societies where it's more important to save face than to get the job done.

Your last sentence feels kind of spot on. The lack of transparency around confidence in the answer makes it hard to use (and I know it would not be simple to add such a thing)

Re: Some critical issues with the SWE-bench dataset

#42

Post by delusional »

modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

> where the resolution rates of the models drop significantly, which are 0.73%, 0.55%, and 3.83%, respectively.

Matches my experience pretty well as too. It'll usually output something that a novice would assume is correct but an expert can clearly identify as "know it all teenager forum post" level stuff.

Re: Some critical issues with the SWE-bench dataset

#43

Post by dingnuts »

aprilthird2021 wrote: post #35
TheDong wrote: post #32

Earlier quoted context omitted.

Different people seem to get wildly different results here, and I'm not sure what percentage is down to the type of software being built vs the usage patterns. In my case, I would guess less than 10% of the code I get out of AIs is useful. What sort of code are you getting those results with? Is it yet-another-react-frontend-button? Is it ebpf programs? Is it a parser in rust? For the latter two, I've found AI to hav…

My gut tells me the AIs will be best for small web projects that are greenfield. The kind a 1-3 person team could maintain. And my gut tells me they are the worst for the kinds of long-established software conglomerates many professionals work at, which have tons of internal services, integrated acquisitions, etc. etc. Ultimately the AI is good at what the average developer online is good at, probably full-stack web…

but that kind of code is so easy to write, and code is already way more terse than natural language! it's literally more typing to explain to an LLM how to write some greenfield web CRUD than it is to just type out the code, and if there's a lot of boilerplate it's faster to generate the repetitive parts with keyboard macros!

where's the value everyone on this site and on LinkedIn (but NONE in my real or professional life) seems to get?

I feel like I'm being gaslit when people say Cursor writes 80% of their code, and honestly, it's the conclusion that makes the most sense to me -- the people making these posts must be well-invested in the startups that stand to profit if AI is actually as good as they say. You know, shills.

Re: Some critical issues with the SWE-bench dataset

#44

Post by KaoruAoiShiho »

theturtletalks wrote: post #26
modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

I personally use Aider's Polyglot Benchmark [0] which is a bit low-key and not gamed just yet. It matches my experience too where Claude Sonnet 3.5 is the best and still beats the new reasoning models like o3-mini, DeepSeek, etc. 0. https://aider.chat/docs/leaderboards/

Sonnet is literally lower on the aider benchmark you just linked. It's only the top with Deepseek as architect, otherwise it's lower than many others.

Re: Some critical issues with the SWE-bench dataset

#46

Post by flakiness »

jbellis wrote: post #20
ukFxqnLa2sBSBf6 wrote: post #13

There’s a few things I’m not understanding here. 1. Did the benchmark authors not review the issues and make sure the solution was not present in the issue? 2. Are the issues locked after they’re included in the dataset? You’d think they would be immutable for reproducibility. 3. For the agents writing patches, is test running part of their inner loop validation? If they write a patch that makes the test pass, then t…

Especially with swe-verified, I thought that was the whole point of that dataset

This was also my first thought, but reading [1] again, what they did was labeling like:

> Whether we consider the issue description to be underspecified and hence unfair to be testing on. > Whether the FAIL_TO_PASS unit tests filter out valid solution

and a bit more. This is pointed out in the linked paper too.

The moral of the story to me is that, don't believe the paid human annotator. You can (hopefully) still believe the PhD students doing these unpaid jobs as their research ;-)

[1] https://openai.com/index/introducing-swe-bench-verified/

Re: Some critical issues with the SWE-bench dataset

#47

Post by refulgentis »

KaoruAoiShiho wrote: post #44
theturtletalks wrote: post #26

Earlier quoted context omitted.

I personally use Aider's Polyglot Benchmark [0] which is a bit low-key and not gamed just yet. It matches my experience too where Claude Sonnet 3.5 is the best and still beats the new reasoning models like o3-mini, DeepSeek, etc. 0. https://aider.chat/docs/leaderboards/

Sonnet is literally lower on the aider benchmark you just linked. It's only the top with Deepseek as architect, otherwise it's lower than many others.

Let's steelman a bit: once you multiply out the edit accuracy versus completion accuracy, Sonnet, on its own, is within 5% of the very top one not using sonnet.

Re: Some critical issues with the SWE-bench dataset

#48

Post by serjester »

OsrsNeedsf2P wrote: post #17
modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

I write a lot of Python and personally I find Claude significantly worse than OpenAI’s reasoning models. I really feel like this varies a ton language to language.

Re: Some critical issues with the SWE-bench dataset

#49

Post by theturtletalks »

KaoruAoiShiho wrote: post #44
theturtletalks wrote: post #26

Earlier quoted context omitted.

I personally use Aider's Polyglot Benchmark [0] which is a bit low-key and not gamed just yet. It matches my experience too where Claude Sonnet 3.5 is the best and still beats the new reasoning models like o3-mini, DeepSeek, etc. 0. https://aider.chat/docs/leaderboards/

Sonnet is literally lower on the aider benchmark you just linked. It's only the top with Deepseek as architect, otherwise it's lower than many others.

Yes, but I use Cursor Composer Agent mode with Sonnet which is like Aider's architect mode where 1 LLM is instructing another one. Not to mention the new reasoning models can't use tool calling (except o3-mini which is not multi-modal).

Re: Some critical issues with the SWE-bench dataset

#50

Post by sebzim4500 »

fooker wrote: post #39
sebzim4500 wrote: post #28

Earlier quoted context omitted.

LLMs do not reliably reproduce their training data. This is quite easy to demonstrate, every LLM has been trained on all of wikipedia (at minimum) and yet there if you ask it a niche fact mentioned once on wikipedia it is highly likely to get it wrong.

Larger llms do pretty well with this. Smaller ones don't.

Large ones do better than small ones but still do worse than I would have expected before I tested them. E.g. `o1` doesn't know things which are repeated several times on wikipedia.
Post reply on HN