Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some critical issues with the SWE-bench dataset

arxiv.org

Post reply on HN
Print view View original

61–70 of 121 posts

Re: Some critical issues with the SWE-bench dataset

#61

Post by hackernewds »

aprilthird2021 wrote: post #41
jandrese wrote: post #30

Earlier quoted context omitted.

Yeah, that's true in many fields with these AI agents. They demo well, but when you put them to actual work they fall right on their face. Even worse, the harder the task you set for them the more they lie to you. It's like hiring a junior dev from one of those highly regimented societies where it's more important to save face than to get the job done.

Your last sentence feels kind of spot on. The lack of transparency around confidence in the answer makes it hard to use (and I know it would not be simple to add such a thing)

sounds like a skill issue to be honest. you could probably tell the assistant to just ask you questions when information is missing instead

Re: Some critical issues with the SWE-bench dataset

#62

Post by vonneumannstan »

shayanh wrote: post #57

I found that this paper was submitted to ICLR, but got rejected: https://openreview.net/forum?id=pwIGnH2LHJ To me the analysis of SWE-Bench is a solid contribution and informative. My guess is that to meet conference's submission bar they had to come up with their own bench (SWE-Bench+), which wasn't thorough enough and the paper got rejected mainly because of that.

Acceptance or rejection at big ML Conferences doesn't seem to carry much signal either way anymore. Completely saturated by grift and poor quality so each paper should be evaluated independent of their Conference status imo.

Re: Some critical issues with the SWE-bench dataset

#63

Post by theturtletalks »

KaoruAoiShiho wrote: post #55
theturtletalks wrote: post #49

Earlier quoted context omitted.

Yes, but I use Cursor Composer Agent mode with Sonnet which is like Aider's architect mode where 1 LLM is instructing another one. Not to mention the new reasoning models can't use tool calling (except o3-mini which is not multi-modal).

Me too, cursor+sonnet is also my go to, I just didn't really understand what you were getting at by pointing out this benchmark. I guess it is significant that Sonnet is the actual line by line coder here. It is the best at that, and it's better than DeepSeek+any other combination and better than Any other reasoner+Sonnet.

Yes I've followed this benchmark for a while and before Deepseek + Sonnet Architect took the top spot, Sonnet was there alone followed by o1 and Gemini EXP. This is one of the few benchmarks where Sonnet is actually on top like my experience shows, other popular ones have 03-mini and DeepSeek r1 which fall short in my opinion.

Re: Some critical issues with the SWE-bench dataset

#65

Post by alfalfasprout »

TheDong wrote: post #32
OsrsNeedsf2P wrote: post #17

Earlier quoted context omitted.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

Different people seem to get wildly different results here, and I'm not sure what percentage is down to the type of software being built vs the usage patterns. In my case, I would guess less than 10% of the code I get out of AIs is useful. What sort of code are you getting those results with? Is it yet-another-react-frontend-button? Is it ebpf programs? Is it a parser in rust? For the latter two, I've found AI to hav…

Almost every time someone says "but most of my code nowadays is LLM generated" it's usually one of three things:

1. Very greenfield work where the LLM doesn't really have a lot of constraints to deal with and can fully control the setup + doesn't have to ingest a lot of existing context 2. Very small projects that largely follow established patterns (CRUD, frontends, etc.) 3. Well established implementation work (the kind of feature that's a simple JIRA ticket).

In my experience they're painfully bad at:

- Novel/niche work where there aren't really answers online to what you're trying to do - Complex refactoring - Architecting within existing constraints (other systems, etc.)

Re: Some critical issues with the SWE-bench dataset

#67

Post by softwaredoug »

OldGreenYodaGPT wrote: post #64

> solutions were directly provided in the issue report or the comments This is fine, many of my real tickets already explain the solution. A good ticket often offers a solution or where to start looking.

Yep that's fine for an issue, but a problem if you're trying to eval whether AIs can solve coding problems.

Re: Some critical issues with the SWE-bench dataset

#68

Post by Xelynega »

jandrese wrote: post #30
bearjaws wrote: post #24

I would argue almost every popular benchmark quoted by the big LLM companies is tainted. OAI, xAI, Antropic, Google all score incredibly well, then you go to try and write code and its just okay . They claim it can do PHD level reasoning, but here I am not trusting it on basic computational thinking.

Yeah, that's true in many fields with these AI agents. They demo well, but when you put them to actual work they fall right on their face. Even worse, the harder the task you set for them the more they lie to you. It's like hiring a junior dev from one of those highly regimented societies where it's more important to save face than to get the job done.

It's almost as if they're not trying to market to the people actually using the products, but trying to convince investors of features that don't exist

Re: Some critical issues with the SWE-bench dataset

#69

Post by rsynnott »

delusional wrote: post #37
semi-extrinsic wrote: post #6

So what we need is something like a versioned crowdsourced coding LLM eval dataset. Every quarter, you have a couple thousand volunteers provide 2 GitHub issues from the past 3 months, which are nontrivial to resolve, and where there exists strong test cases. Each volunteer then cross-checks 2 issues from other volunteers. The volunteers get 1 month free subscription to some AI service in return. This dataset is then…

And why would these "couple of thousand volunteers" help with this?

And how would you ensure that all of them were really volunteers and not colluding with the vendors? Like, tech companies cheating on benchmarks is an old, old story (personal favourite: in the dark ages, before 3D acceleration, some graphics card drivers, on detecting a 2D acceleration benchmark, would _simply draw the wrong thing_), and I wouldn’t trust at least three of the major players as far as I could throw them.

Re: Some critical issues with the SWE-bench dataset

#70

Post by ghaff »

Philip-J-Fry wrote: post #58
TheDong wrote: post #32

Earlier quoted context omitted.

Different people seem to get wildly different results here, and I'm not sure what percentage is down to the type of software being built vs the usage patterns. In my case, I would guess less than 10% of the code I get out of AIs is useful. What sort of code are you getting those results with? Is it yet-another-react-frontend-button? Is it ebpf programs? Is it a parser in rust? For the latter two, I've found AI to hav…

I'm pretty confident in my ability to write any code in my main language. But AI is still very useful in just filling out boiler plate, or noticing a pattern and filling out the rest of some repetitive code. Or say, I need to write wrapper around a common command-line utility. It's pretty good at generating the code for that. What I mostly enjoy using it for is just writing bash scripts for me. I hate writing bash bu…

>AI is still very useful in just filling out boiler plate

That's what I tend to find with English writing as well. It's not great. But sometimes you just need decent generic prose for an introduction or an explanation of something. If you know enough to adjust as needed, it can save time for something that readers are probably just skimming anyway. As I've written previously, about a year ago I was working on cleaning up a bunch of reference architectures and I used Google's Bard in that case to give me a rough draft of background intros for some of them which I modified as needed. Nothing miraculous but saved me a bit of time.

Post reply on HN