Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some critical issues with the SWE-bench dataset

arxiv.org

Post reply on HN
Print view View original

31–40 of 121 posts

Re: Some critical issues with the SWE-bench dataset

#31

Post by azinman2 »

theturtletalks wrote: post #26
modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

I personally use Aider's Polyglot Benchmark [0] which is a bit low-key and not gamed just yet. It matches my experience too where Claude Sonnet 3.5 is the best and still beats the new reasoning models like o3-mini, DeepSeek, etc. 0. https://aider.chat/docs/leaderboards/

Would love if they put latency in this too.

Re: Some critical issues with the SWE-bench dataset

#32

Post by TheDong »

OsrsNeedsf2P wrote: post #17
modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

Different people seem to get wildly different results here, and I'm not sure what percentage is down to the type of software being built vs the usage patterns.

In my case, I would guess less than 10% of the code I get out of AIs is useful.

What sort of code are you getting those results with? Is it yet-another-react-frontend-button? Is it ebpf programs? Is it a parser in rust?

For the latter two, I've found AI to have pretty low rates, and for the former I haven't had the desire to try.

Re: Some critical issues with the SWE-bench dataset

#33

Post by wongarsu »

sva_ wrote: post #18
thegeomaster wrote: post #5

Earlier quoted context omitted.

...by piping it through the world's most inefficient echo function.

That reminds me of someone calling the Bitcoin blockchain the most expensive linked list in the world.

The difference is that Bitcoin is designed to be "just" an append-only* timestamped linked list, with some rules on how a new node can look like in order to be successfully appended. Making the creation of a canonical linked list possible between hostile actors is the whole innovation. The currency stuff is "just" a cool practical application tacked on the linked list

LLMs by contrast are not designed to just repeat what's already in the instructions, no matter which stance on LLM design you subscribe to

* exceptions apply

Re: Some critical issues with the SWE-bench dataset

#34

Post by huac »

sebzim4500 wrote: post #28
huac wrote: post #21

> 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments. Looking at the benchmark, https://www.swebench.com/ , about half of scored submissions score under 1/3 correct? So they're either not cheating, or not cheating effectively?

LLMs do not reliably reproduce their training data. This is quite easy to demonstrate, every LLM has been trained on all of wikipedia (at minimum) and yet there if you ask it a niche fact mentioned once on wikipedia it is highly likely to get it wrong.

that comment refers to the test time inference, i.e. what the model is prompted with, not to what it is trained on. this is, of course, also a tricky problem (esp over long context, needle in a haystack), but it should be much easier than memorization.

anyways, another interpretation is that the model needs to also make a decision on if the code in the issue is a reliable fix or not too

Re: Some critical issues with the SWE-bench dataset

#35

Post by aprilthird2021 »

TheDong wrote: post #32
OsrsNeedsf2P wrote: post #17

Earlier quoted context omitted.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

Different people seem to get wildly different results here, and I'm not sure what percentage is down to the type of software being built vs the usage patterns. In my case, I would guess less than 10% of the code I get out of AIs is useful. What sort of code are you getting those results with? Is it yet-another-react-frontend-button? Is it ebpf programs? Is it a parser in rust? For the latter two, I've found AI to hav…

My gut tells me the AIs will be best for small web projects that are greenfield. The kind a 1-3 person team could maintain.

And my gut tells me they are the worst for the kinds of long-established software conglomerates many professionals work at, which have tons of internal services, integrated acquisitions, etc. etc.

Ultimately the AI is good at what the average developer online is good at, probably full-stack web dev of projects from scratch.

Re: Some critical issues with the SWE-bench dataset

#36

Post by feznyng »

sebzim4500 wrote: post #28
huac wrote: post #21

> 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments. Looking at the benchmark, https://www.swebench.com/ , about half of scored submissions score under 1/3 correct? So they're either not cheating, or not cheating effectively?

LLMs do not reliably reproduce their training data. This is quite easy to demonstrate, every LLM has been trained on all of wikipedia (at minimum) and yet there if you ask it a niche fact mentioned once on wikipedia it is highly likely to get it wrong.

This is why I’m a bit skeptical of the o3 results. If it’s spending a bunch of time reasoning aren’t the chances of it simply regurgitating a solution it saw in its training data at some point in its output stream higher? It still needs to be clever enough to identify it as the correct answer but it’s not as impressive as an original solution.

Re: Some critical issues with the SWE-bench dataset

#37

Post by delusional »

semi-extrinsic wrote: post #6

So what we need is something like a versioned crowdsourced coding LLM eval dataset. Every quarter, you have a couple thousand volunteers provide 2 GitHub issues from the past 3 months, which are nontrivial to resolve, and where there exists strong test cases. Each volunteer then cross-checks 2 issues from other volunteers. The volunteers get 1 month free subscription to some AI service in return. This dataset is then…

And why would these "couple of thousand volunteers" help with this?

Re: Some critical issues with the SWE-bench dataset

#38

Post by nyrikki »

theturtletalks wrote: post #26
modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

I personally use Aider's Polyglot Benchmark [0] which is a bit low-key and not gamed just yet. It matches my experience too where Claude Sonnet 3.5 is the best and still beats the new reasoning models like o3-mini, DeepSeek, etc. 0. https://aider.chat/docs/leaderboards/

Quite the corpus for Exercism tasks that were almost certainly trained on, which could lead this to doing what we know LLM/LRM's are good at...approximate retrieval.

https://github.com/search?q=Exercism&type=repositories

Re: Some critical issues with the SWE-bench dataset

#39

Post by fooker »

sebzim4500 wrote: post #28
huac wrote: post #21

> 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments. Looking at the benchmark, https://www.swebench.com/ , about half of scored submissions score under 1/3 correct? So they're either not cheating, or not cheating effectively?

LLMs do not reliably reproduce their training data. This is quite easy to demonstrate, every LLM has been trained on all of wikipedia (at minimum) and yet there if you ask it a niche fact mentioned once on wikipedia it is highly likely to get it wrong.

Larger llms do pretty well with this.

Smaller ones don't.

Re: Some critical issues with the SWE-bench dataset

#40

Post by yunwal »

theturtletalks wrote: post #26
modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

I personally use Aider's Polyglot Benchmark [0] which is a bit low-key and not gamed just yet. It matches my experience too where Claude Sonnet 3.5 is the best and still beats the new reasoning models like o3-mini, DeepSeek, etc. 0. https://aider.chat/docs/leaderboards/

Are Exercism coding exercises really low key? I thought it was like the standard free platform for learning a new language now
Post reply on HN