Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some critical issues with the SWE-bench dataset

arxiv.org

Post reply on HN
Print view View original

81–90 of 121 posts

Re: Some critical issues with the SWE-bench dataset

#81

Post by doug_durham »

aprilthird2021 wrote: post #35
TheDong wrote: post #32

Earlier quoted context omitted.

Different people seem to get wildly different results here, and I'm not sure what percentage is down to the type of software being built vs the usage patterns. In my case, I would guess less than 10% of the code I get out of AIs is useful. What sort of code are you getting those results with? Is it yet-another-react-frontend-button? Is it ebpf programs? Is it a parser in rust? For the latter two, I've found AI to hav…

My gut tells me the AIs will be best for small web projects that are greenfield. The kind a 1-3 person team could maintain. And my gut tells me they are the worst for the kinds of long-established software conglomerates many professionals work at, which have tons of internal services, integrated acquisitions, etc. etc. Ultimately the AI is good at what the average developer online is good at, probably full-stack web…

Your intuition runs counter to most folks experience. I work on complex machine learning training loops and loss functions. LLMs work great on that.

Re: Some critical issues with the SWE-bench dataset

#83

Post by 0x20cowboy »

modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

It matches my experience as well.

I find the models very useful to chat about library documentation or high level algorithm concepts, but I find the code it generates to be… I don’t know how else to say it… really bad and often out of context.

I know developers who blindly follow the hype and use them to generate production code. That scares the poop emoji out of me, and the code reads like an asset flipped 3D game.

Re: Some critical issues with the SWE-bench dataset

#84

Post by avs733 »

modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

It is worth reflecting, as much as HN seems to hate the social sciences, on this point. But the difficulty of measuring intelligence is a challenge that several fields have struggled with for decades. It is inherently hard because defining intelligence and building intelligence are very closely coupled. This both makes it hard to make unbiased measures as well making measures that don't affect the phenomenon basically NP hard, or known as the Flynn effect[0].

It also goes to how a lot of people misunderstand the replication crisis. 'Hard science' really should replicate - we should be able filter out sources fo error and variance because the phenomena (generally) isn't affected by our attempts to measure it. Making social science replicate often requires so much control that it is deabstracted from reality, meaning the effort at replication reduces the value and usefulness of the knowledge. Generalizable claims are hard because the sources of variance are so much larger adn more complex. Speaking as someone who went through a transition from engineering to social sciences, it is the concept that made it hard. I started my time in social sciences with a cool idea of a whole carrer based on just doing replication studies, because science. That was...useful and stupid at the same time.

[0] https://en.wikipedia.org/wiki/Flynn_effect

Re: Some critical issues with the SWE-bench dataset

#85

Post by cyanydeez »

alfalfasprout wrote: post #65
TheDong wrote: post #32

Earlier quoted context omitted.

Different people seem to get wildly different results here, and I'm not sure what percentage is down to the type of software being built vs the usage patterns. In my case, I would guess less than 10% of the code I get out of AIs is useful. What sort of code are you getting those results with? Is it yet-another-react-frontend-button? Is it ebpf programs? Is it a parser in rust? For the latter two, I've found AI to hav…

Almost every time someone says "but most of my code nowadays is LLM generated" it's usually one of three things: 1. Very greenfield work where the LLM doesn't really have a lot of constraints to deal with and can fully control the setup + doesn't have to ingest a lot of existing context 2. Very small projects that largely follow established patterns (CRUD, frontends, etc.) 3. Well established implementation work (the…

I assume it's the commoditized work that made India a success at outsourced activities.

Re: Some critical issues with the SWE-bench dataset

#86

Post by modeless »

OsrsNeedsf2P wrote: post #17
modeless wrote: post #3

> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

When I use Cursor I ask for specific small tasks that I know it should be able to handle. Larger, open-ended tasks fail almost universally for me.

Re: Some critical issues with the SWE-bench dataset

#87

Post by cyanydeez »

duped wrote: post #77
TheDong wrote: post #32

Earlier quoted context omitted.

Different people seem to get wildly different results here, and I'm not sure what percentage is down to the type of software being built vs the usage patterns. In my case, I would guess less than 10% of the code I get out of AIs is useful. What sort of code are you getting those results with? Is it yet-another-react-frontend-button? Is it ebpf programs? Is it a parser in rust? For the latter two, I've found AI to hav…

I think it's frontend javascript versus everything else. There's a few languages/tools I use often but am not an expert in and have been using Claude 3.5 to help me work with existing code. On paper this is a perfect use case. In practice it's like working with an intern that has google in front of them and enough jargon to convince me what they're saying isn't bullshit. Eventually, I'll be able to coax the answers I…

the reason AI is a giant f'n bubble is because doing an exhaustive review of results means _doing the actual work of millions of manhours_.

Instead, people squint their eyes at scrolling matrix text and convince themselves it must be true.

Re: Some critical issues with the SWE-bench dataset

#88

Post by nitwit005 »

semi-extrinsic wrote: post #6

So what we need is something like a versioned crowdsourced coding LLM eval dataset. Every quarter, you have a couple thousand volunteers provide 2 GitHub issues from the past 3 months, which are nontrivial to resolve, and where there exists strong test cases. Each volunteer then cross-checks 2 issues from other volunteers. The volunteers get 1 month free subscription to some AI service in return. This dataset is then…

If you know some way to get people to volunteer millions of dollars of free labor, there are better uses of their time than evaluating LLMs.

Re: Some critical issues with the SWE-bench dataset

#89

Post by ilrwbwrkhv »

Xelynega wrote: post #68
jandrese wrote: post #30

Earlier quoted context omitted.

Yeah, that's true in many fields with these AI agents. They demo well, but when you put them to actual work they fall right on their face. Even worse, the harder the task you set for them the more they lie to you. It's like hiring a junior dev from one of those highly regimented societies where it's more important to save face than to get the job done.

It's almost as if they're not trying to market to the people actually using the products, but trying to convince investors of features that don't exist

Its the good old Elon musk playbook spread out across the industry.

Re: Some critical issues with the SWE-bench dataset

#90

Post by [deleted] »

TheDong wrote: post #32
OsrsNeedsf2P wrote: post #17

Earlier quoted context omitted.

Anecdotal but I was always shocked to see Claude 3.5 perform so poorly in the benchmarks, when it generates 80% of my code in Cursor (and in cases it fails, no other model succeeds)

Different people seem to get wildly different results here, and I'm not sure what percentage is down to the type of software being built vs the usage patterns. In my case, I would guess less than 10% of the code I get out of AIs is useful. What sort of code are you getting those results with? Is it yet-another-react-frontend-button? Is it ebpf programs? Is it a parser in rust? For the latter two, I've found AI to hav…

[deleted]
Post reply on HN