Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some critical issues with the SWE-bench dataset

arxiv.org

Post reply on HN
Print view View original

111–120 of 121 posts

Re: Some critical issues with the SWE-bench dataset

#111

Post by gcr »

OsrsNeedsf2P wrote: post #17
modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

I've been quite underwhelmed at Copilot's suggestions. Is Claude all that better?

Re: Some critical issues with the SWE-bench dataset

#112

Post by codelion »

comex wrote: post #73

Some of the examples in the paper seem to be wrong. For django-31056, they claim the AI-generated patch is "incomplete" because it's "missing critical parts of this logic, such as the try-except block and the check for a running event loop.". But if you look at the diff, that's clearly wrong. The try-except block and running check were already there before the patch. The human patch just indented them, making them ap…

Interesting analysis! I hadn't dug into the specific patch details like that. It's a good reminder that "correctness" isn't always the only dimension to evaluate these AI-generated patches – readability and idiomatic style definitely matter too, even if the functional outcome is the same.

I've been playing around with some automated code review tools recently, and it's surprising how often they flag things that are technically correct but just... unusual. Style matters, especially for maintainability.

Re: Some critical issues with the SWE-bench dataset

#113

Post by dr_kiszonka »

danielbln wrote: post #29
siva7 wrote: post #22

Earlier quoted context omitted.

o3-mini and gpt-4o are so piss poor in agent coding compared to claude that you don't even need a benchmark

Claude really is a step above the rest when it comes to agentic coding.

When I used it with Open Hands it was great but also quite expensive (~$8/hr). In Trea, it was pretty bad, but free. Maybe it depends on how the agents use it? (I was writing the same piece of software, a simple web crawler for a hobby RAG project.)

Re: Some critical issues with the SWE-bench dataset

#114

Post by _cs2017_ »

comex wrote: post #73

Some of the examples in the paper seem to be wrong. For django-31056, they claim the AI-generated patch is "incomplete" because it's "missing critical parts of this logic, such as the try-except block and the check for a running event loop.". But if you look at the diff, that's clearly wrong. The try-except block and running check were already there before the patch. The human patch just indented them, making them ap…

I can only confirm two mistakes in the apper: 1) As you say, the reversed(self.dict) is actually correct; 2) as another poster below said, hints are not part of the input. These two mistakes are so egregious given the objective of the paper that I'm convinced the authors are not qualified to write it.

IMHO, it is probably better to discard this paper, and wait for someone else to cover this important topic.

Re: Some critical issues with the SWE-bench dataset

#115

Post by brookst »

ilrwbwrkhv wrote: post #89
Xelynega wrote: post #68

Earlier quoted context omitted.

It's almost as if they're not trying to market to the people actually using the products, but trying to convince investors of features that don't exist

Its the good old Elon musk playbook spread out across the industry.

Someone should coin a term for this very new phenomenon. Maybe “vaporware”?

Re: Some critical issues with the SWE-bench dataset

#116

Post by alalv »

Something weird (or at least uncommon) that has caught my attention and I havent seen mentioned in the comments is that they cite the swe-bench paper author by first name in the abstract, Carlos et al, and then by last name (as it is usually done) in the paper, Jimenez et al.

Re: Some critical issues with the SWE-bench dataset

#117

Post by siva7 »

comex wrote: post #73

Some of the examples in the paper seem to be wrong. For django-31056, they claim the AI-generated patch is "incomplete" because it's "missing critical parts of this logic, such as the try-except block and the check for a running event loop.". But if you look at the diff, that's clearly wrong. The try-except block and running check were already there before the patch. The human patch just indented them, making them ap…

The paper should be then retracted.

Re: Some critical issues with the SWE-bench dataset

#118

Post by dimitri-vs »

hackernewds wrote: post #61
aprilthird2021 wrote: post #41

Earlier quoted context omitted.

Your last sentence feels kind of spot on. The lack of transparency around confidence in the answer makes it hard to use (and I know it would not be simple to add such a thing)

sounds like a skill issue to be honest. you could probably tell the assistant to just ask you questions when information is missing instead

Have you actually tried this? What happens is it will very often ask you questions at irrelevant times so you start ignoring the questions and it becomes wasted space.

Even OpenAI hasn't figured it out, because their Deep Research always asks questions before starting the search.

Re: Some critical issues with the SWE-bench dataset

#119

Post by jerpint »

OsrsNeedsf2P wrote: post #17
modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

I think the big thing overlooked is how much the human steering the models matters. If you know what you’re doing and what changes you need, cursor and other tools make you so productive.

If you don’t know what you’re doing, these things can sometimes produce good code, and sometimes produce things that don’t work at all

Re: Some critical issues with the SWE-bench dataset

#120

Post by sebzim4500 »

huac wrote: post #34
sebzim4500 wrote: post #28

Earlier quoted context omitted.

LLMs do not reliably reproduce their training data. This is quite easy to demonstrate, every LLM has been trained on all of wikipedia (at minimum) and yet there if you ask it a niche fact mentioned once on wikipedia it is highly likely to get it wrong.

that comment refers to the test time inference, i.e. what the model is prompted with, not to what it is trained on. this is, of course, also a tricky problem (esp over long context, needle in a haystack), but it should be much easier than memorization. anyways, another interpretation is that the model needs to also make a decision on if the code in the issue is a reliable fix or not too

Then I don't understand what he's suggesting. It is obviously not the case that 1/3 of the questions int he SWE-bench dataset have the solution in as part of the issue that is provided to the model. You can just download it and look. The solution is likely in the training data though.
Post reply on HN