Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

senior-swe-bench.snorkel.ai

Post reply on HN
Print view View original

101–110 of 130 posts

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#101

Post by sally_glance »

Eridrus wrote: post #54
sally_glance wrote: post #40

Earlier quoted context omitted.

Well actually there is a reasonably objective standard defining software quality criteria on the source code level (ISO 5055). They also define 29 criteria for maintainability: https://www.it-cisq.org/coding-rules/

See, this goes back to the, all software engineers besides me are wrong, because I see this list and do not think it is anywhere close to a sufficient list for good quality software. The thing about all these criteria is that sometimes they are important, sometimes they are not. This "standard" exists for the sake of code analysis vendors to be able to have some sort of shared taxonomy, but also provide a fig leaf of…

Very true. As with all standards, there will always be people who disagree. We still mostly follow them either because we're forced to or because the effort required to establish another standard doesn't outweigh the benefits.

Personally I've always been a proponent of project specific standards, but after many years of discussions about more or less individual preference I've come to think that maybe settling on something global isn't the worst idea. Not that I think it must be this one in particular, but it's not the worst start either.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#102

Post by fiso64 »

ricardobeat wrote: post #94
fiso64 wrote: post #87

Earlier quoted context omitted.

Yes it is relevant and testable. It's exactly what I meant by "a measurable increase in quality of the final product". In fact a proper test harness would reveal that problem. You are forgetting that with LLMs, testing software does not have to end at the usual unit/integration/e2e level.

But how is that testable? If your test is validating the rigidity, water resistance, etc, they will all pass even if the underlying material is a bad choice. Or the glue will degrade in six months. You can't test if a codebase will be extensible or maintainable as requirements change in the future, if the abstraction level or architecture is sound - that's down to code quality measures like the ones used here. LLMs a…

That's why we should simulate changing requirements, for example with an LLM roleplaying as a human who's co-developing with an agent. Simply asking the LLM to add one big feature is not enough. I don't see why we shouldn't be able to build a more advanced benchmark. Attempting to benchmark "taste" is not the way.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#104

Post by organsnyder »

jeffbee wrote: post #103

Staff SWE Bench: LLM doubts whether we should do any of this, calls the entire project into question, refuses to merge code, but is happy to delete it.

Principal version: similar, but also says the only acceptable approach is to do it like they did it at their last company.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#106

Post by pgwhalen »

organsnyder wrote: post #104
jeffbee wrote: post #103

Staff SWE Bench: LLM doubts whether we should do any of this, calls the entire project into question, refuses to merge code, but is happy to delete it.

Principal version: similar, but also says the only acceptable approach is to do it like they did it at their last company.

Distinguished version: write the outline of the slide deck for the talk you plan to give at conferences about it, without having shipped anything or even written code yet.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#107

Post by jfim »

I wonder how they're planning for the benchmark to stay relevant over time.

If the benchmark is to implement features that are part of an open source project, and LLMs have those changes as part of their training dataset, it seems that they could just give a verbatim or slightly modified version of the change in their training data.

And if one updates the benchmark to only incorporate code changes that are past the models knowledge cutoff, then the benchmark is less comparable over time, since the changes in the benchmark at time T and T+1 aren't the same.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#108

Post by sebastiennight »

LiamPowell wrote: post #63
rhdunn wrote: post #52

Earlier quoted context omitted.

This approach is effectively seeding the context with how you want the LLM to behave/operate ("senior reviewer", i.e. the style of the responses you want) and the context/domain in which the LLM is operating in ("SWE-Bench"). This is common in system prompts and frames the responses. For example, you'd get different responses saying: 1. you are a pirate writing sea shanties about programming; 2. you are a news report…

This is not actually what the reviewer prompt says, or perhaps it is, I don't know since they don't make it public. I'm just pointing out how it seems like a bad idea to ask a LLM to make a subjective judgement on things like "taste". If the SOTA LLM witting the code could not produce tasteful code then why would a different LLM be able to judge the "taste" of that code? Which LLM should we even use to judge taste? I…

> Is it giving an unfair advantage to Model X if we use Model X as the judge?

There have been studies that showed that models tended to rate responses from their own family of models better than equivalent responses from other families, eg. gpt-4 would prefer a response from gpt-3

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#109

Post by fsddfsdfssdf »

jeffbee wrote: post #103

Staff SWE Bench: LLM doubts whether we should do any of this, calls the entire project into question, refuses to merge code, but is happy to delete it.

You jest but I indeed find rejection an integral part of the job. Not plainly saying "no, get away", but backing up, requesting big picture views and trying to see if the overall organization is in need of and capable of carrying said project long-term feels like the absolute minimum that needs doing before we even begin.

I suspect LLMs can do this just fine and probably better than us, but they do need to be trained specifically for it and I have a hard time coming up with good sources of training data for it.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#110

Post by fsddfsdfssdf »

antonvs wrote: post #37
LiamPowell wrote: post #8

> You are a senior SWE-Bench reviewer, make no mistakes. I don't know what a better approach would look like while still remaining feasible, however this approach of telling a LLM to make a subjective judgement seems fundamentally flawed.

The “make no mistakes” admonition does seem pretty silly (it’s been skewered to death on yt), but… it’s easy to imagine how it might work. E.g. it could be interpreted as simply as “check your work”. Of course, no-one seems to be (publicly) doing the comparative measurements that might allow us to reach rational conclusions here.

Conversations in its training data that explicitly mentioned "make no mistakes" don't strike me as particularly rich sources of high-quality reasoning signals. They strike me as conversations with Pointy-haired Bosses.
Post reply on HN