Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some critical issues with the SWE-bench dataset

arxiv.org

Post reply on HN
Print view View original

71–80 of 121 posts

Re: Some critical issues with the SWE-bench dataset

#71

Post by dgunay »

OsrsNeedsf2P wrote: post #17
modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

I've been using LLMs for tab autocomplete for a while and just recently started trying out agentic coding AI (Copilot Edits and Cline). I think the disappointing shortfall of agentic AIs (at least for me) comes from the feedback loop being so much looser than the autocomplete style. With autocomplete, I don't have to actively think about what context to feed it, and I can gently correct it if it goes in the wrong direction on a line-by-line basis. With AI agents, they have a lot more leeway to generate a ton of code and reason themselves off the rails before you're able to step in and correct them. Now granted, I am also not very good yet at managing context and crafting prompts, but it feels a lot harder to get good at than simply dropping an AI autocompleter into an existing programming workflow. It's a new paradigm.

Re: Some critical issues with the SWE-bench dataset

#72

Post by ben_w »

TheDong wrote: post #32
OsrsNeedsf2P wrote: post #17

Earlier quoted context omitted.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

Different people seem to get wildly different results here, and I'm not sure what percentage is down to the type of software being built vs the usage patterns. In my case, I would guess less than 10% of the code I get out of AIs is useful. What sort of code are you getting those results with? Is it yet-another-react-frontend-button? Is it ebpf programs? Is it a parser in rust? For the latter two, I've found AI to hav…

> For the latter two, I've found AI to have pretty low rates, and for the former I haven't had the desire to try.

Similar. I've got a joke language project on the back burner, doing it properly requires going back over my 23 year old university notes on yacc etc., so I tried AI… the AI just makes a mess of it*.

For anything front end, even the original ChatGPT-3.5 model is basically magic (i.e. sufficiently advanced technology).

* I think the last time I touched it was just before o1 was announced; as o3 is now in the free tier of ChatGPT, I should try again…

Re: Some critical issues with the SWE-bench dataset

#73

Post by comex »

Some of the examples in the paper seem to be wrong.

For django-31056, they claim the AI-generated patch is "incomplete" because it's "missing critical parts of this logic, such as the try-except block and the check for a running event loop.". But if you look at the diff, that's clearly wrong. The try-except block and running check were already there before the patch. The human patch just indented them, making them appear as both - and +, while the AI patch didn't. To me, the AI patch seems correct. It's slightly less efficient than the human patch when DJANGO_ALLOW_ASYNC_UNSAFE is set, but slightly more efficient when it isn't (which is the common case!). The human patch does feel more natural, but the AI patch is fine. I'd grade it a tie between human and AI.

For django-32517, they claim that the human and AI patches "produce entirely different outputs", but actually they do exactly the same thing. The human version has `reversed(self.dict)`, while the AI version has `reversed(self.dict.keys())`. `reversed` treats the object as an iterator, and iterating over a dictionary in Python just gives you the keys, so it doesn't matter whether you call `.keys()` first. The human patch is more idiomatic, but it's also more confusing, as shown by the fact that it confused the authors of this paper. I'd grade it another tie.

Edit: I tried to sign up for OpenReview so I could leave a comment about this, but the system wouldn't let me register without completing a form that assumes you have an academic position. Perhaps I should email the authors.

Re: Some critical issues with the SWE-bench dataset

#74

Post by botanical76 »

dingnuts wrote: post #43
aprilthird2021 wrote: post #35

Earlier quoted context omitted.

My gut tells me the AIs will be best for small web projects that are greenfield. The kind a 1-3 person team could maintain. And my gut tells me they are the worst for the kinds of long-established software conglomerates many professionals work at, which have tons of internal services, integrated acquisitions, etc. etc. Ultimately the AI is good at what the average developer online is good at, probably full-stack web…

but that kind of code is so easy to write, and code is already way more terse than natural language! it's literally more typing to explain to an LLM how to write some greenfield web CRUD than it is to just type out the code, and if there's a lot of boilerplate it's faster to generate the repetitive parts with keyboard macros! where's the value everyone on this site and on LinkedIn (but NONE in my real or professional…

Have you tried the AI intellisense models like Copilot?

I don't understand the notion that it is faster to generate repetitive code with keyboard macros. I use Vim-mode exclusively, and while I'm not a Vim master, I don't think there's any set of macros that will do what Copilot can do.

It's not that Copilot is smart. It's that 60% of what I do doesn't require much intelligence to anticipate. It is the 40% that matters, the remainder can be trivially guessed, and this is exactly what Copilot does.

Maybe this will help: you need to imagine with an AI intellisense that with each keystroke, you are collapsing the possibility space down to a smaller, finite number of outcomes. You write exactly what code you need for the dumb AI to predict the rest of it.

There are a LOT of reasons why AI intellisense is not all there yet; it can be distracting; it can try to generate too much at once; none of the tools have LSP integrated, so it will provide bullshit suggestions of library methods that don't exist. This is all true, and yet it is still highly valuable in some domains, for some people.

That said, if you write x86 assembly for a living, you are probably out of luck.

(I write Kotlin, Java for Android apps and services, C++ that is tightly integrated with the SoC. Python and Bash for command-line tools that invoke REST APIs. Copilot is useful for these domains.)

Re: Some critical issues with the SWE-bench dataset

#75

Post by ryoshu »

hackernewds wrote: post #61
aprilthird2021 wrote: post #41

Earlier quoted context omitted.

Your last sentence feels kind of spot on. The lack of transparency around confidence in the answer makes it hard to use (and I know it would not be simple to add such a thing)

sounds like a skill issue to be honest. you could probably tell the assistant to just ask you questions when information is missing instead

Programming is easy. Asking the right question is hard.

People don't know what questions to ask.

Re: Some critical issues with the SWE-bench dataset

#76

Post by washadjeffmad »

bearjaws wrote: post #24

I would argue almost every popular benchmark quoted by the big LLM companies is tainted. OAI, xAI, Antropic, Google all score incredibly well, then you go to try and write code and its just okay . They claim it can do PHD level reasoning, but here I am not trusting it on basic computational thinking.

To be totally fair, using PhD as a barometer of anything without specifying what is like claiming that LLMs have encyclopedic knowledge while meaning a children's encyclopedia.

Re: Some critical issues with the SWE-bench dataset

#77

Post by duped »

TheDong wrote: post #32
OsrsNeedsf2P wrote: post #17

Earlier quoted context omitted.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

Different people seem to get wildly different results here, and I'm not sure what percentage is down to the type of software being built vs the usage patterns. In my case, I would guess less than 10% of the code I get out of AIs is useful. What sort of code are you getting those results with? Is it yet-another-react-frontend-button? Is it ebpf programs? Is it a parser in rust? For the latter two, I've found AI to hav…

I think it's frontend javascript versus everything else.

There's a few languages/tools I use often but am not an expert in and have been using Claude 3.5 to help me work with existing code. On paper this is a perfect use case. In practice it's like working with an intern that has google in front of them and enough jargon to convince me what they're saying isn't bullshit. Eventually, I'll be able to coax the answers I need out of it.

I'll say though the fact AI can't say "I don't know" and closely related "that is not possible in the context you've given me" combined with the inability to reason is what gives you results that look OK but are subtly trash.

Re: Some critical issues with the SWE-bench dataset

#78

Post by perrygeo »

The solution moving forward has to be private benchmark suites. I could see teams investing in their own set of programming challenges and periodically re-evaluating them - similar to how we would construct sets of live interview questions for candidates and qualitatively assess their ability.

It's so vital that it's not leaked and that it's fit-for-purpose and manually assessed. These general purpose, public benchmarks based on questionable metrics are effectively worthless to assess real programming skill.

Case in point, as others have mentioned here, Claude scores modestly on these benchmarks but vastly better than the alternatives in practice. I don't trust Claude fully but far more than OpenAI models; it's not even close. The IRL performance advantage is not reflected in any of these benchmarks.

Re: Some critical issues with the SWE-bench dataset

#79

Post by rco8786 »

OsrsNeedsf2P wrote: post #17
modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

That's been my experience too, but I would guess the problem of "here is a ton of context, produce a small amount of code" is significantly better suited for LLMs than "here is a problem, produce a ton of code".
Post reply on HN