Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

senior-swe-bench.snorkel.ai

Post reply on HN
Print view View original

71–80 of 130 posts

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#71

Post by nsingh2 »

iLoveOncall wrote: post #70
nsingh2 wrote: post #69

Earlier quoted context omitted.

Refusing to sufficiently specify a task and hoping the model guesses correctly is not being productive. Again, these models still don't really ask questions when they should. You have to explicitly tell them to. Specifying the problem is not extra work separate from solving it. If you skip that step, the ambiguity gets pushed into the model’s assumptions. Then you get a plausible looking answer to the wrong problem a…

My point is that it is much faster for me to solve the problem by writing the code than to write specifications detailed enough for the model to do the right thing in the right way.

A highly detailed specification is not what I mean here. It's closer to plugging in a few sentence descriptions (or a totally cluttered brain dump) and having the model interview you to help pin down critical details before continuing.

In my own work, it's usually been a few critical assumptions the model made silently (and I never even though of initially) that end up being the difference between passable results the first try, and me having to go back and fix things. Occasionally some questions force me to rethink the problem entirely.

I basically always begin any long-running session with this kind of brainstorming. I don't find the existing plan modes in Claude Code/Codex to be critical enough.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#72

Post by fiso64 »

iLoveOncall wrote: post #65
fiso64 wrote: post #55

I think benchmarks like this are too subjective and narrow to be useful. For example, whether a patch "bloats" the codebase really depends on the situation: If it's building a feature that will grow in the future, or refactoring code that has a long history of bugs, then a larger patch might in fact be good. It's not clear from the blog just how much context the LLM judge receives about the long term project goals an…

> Full disclosure, I'm not a software engineer Then maybe you should abstain, because your comment is a complete load of nonsense. Bad code is bad code regardless of the history or scope of the feature. Maintainability is important because you can never know if a feature will be built upon in the future or not. Bloat is bad regardless, because it increases the overall complexity of the whole software development life…

>Maintainability is important because you can never know if a feature will be built upon in the future or not.

Of course maintainability is important. It's almost like saying good code is important (duh). The issue is that what is or isn't maintainable depends on the problem at hand. Sometimes you need to build heavier abstractions or refactor existing code when implementing a feature because it will pay off later. Other times, that exact same approach is horrible over-engineering because a simple, direct fix was all that was needed, so in fact you introduced a maintenance burden. You cannot reliably decide whether a patch is "bloated" or "tasteful" when looking at a diff without knowing where the project is headed.

>You can write extremely poor code that has no bugs, it doesn't make it tasteful.

You can, but it becomes increasingly hard to do so as you try to add features and maintain it. Taste, whatever that is, should ultimately lead to a measurable increase in the quality of the final product; if it doesn't, then your definition of "taste" is irrelevant. What I'm proposing is to skip trying to measure this ill-defined concept and only assess the quality of the final product, after the agent spent a significant amount of time working on it, and a reviewer spent a significant amount of time testing it. Agents should be assessed on their ability to build entire projects (e.g., many large features or even an entire app), not just a single feature. If an agent has no taste, then its bad decisions will compound and result in it stalling, or its output having more bugs and performing worse, given a sufficiently large scope.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#73

Post by reactordev »

fooker wrote: post #39
nsingh2 wrote: post #31

Earlier quoted context omitted.

Why supply underspecified requirements in the first place? Both models are good at challenging assumptions/edge cases and asking questions to clarify, but seemingly only when explicitly asked (i.e. something like a "brainstorm" skill). I don't think either harnesses do enough to encourage the model to challenge all assumptions and ask questions, maybe because users might find it annoying. That step is basically a req…

> Why supply underspecified requirements in the first place? Because you'd not want to forever loop outside your home when asked to "while you're out, grab some eggs" :)

Meaning why not leave home with your grocery list?

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#74

Post by reactordev »

iLoveOncall wrote: post #70
nsingh2 wrote: post #69

Earlier quoted context omitted.

Refusing to sufficiently specify a task and hoping the model guesses correctly is not being productive. Again, these models still don't really ask questions when they should. You have to explicitly tell them to. Specifying the problem is not extra work separate from solving it. If you skip that step, the ambiguity gets pushed into the model’s assumptions. Then you get a plausible looking answer to the wrong problem a…

My point is that it is much faster for me to solve the problem by writing the code than to write specifications detailed enough for the model to do the right thing in the right way.

You should try transcribing while you speak. Then you can explain and articulate the task sufficiently that the model should have enough context to complete the task to your satisfaction. Since you won’t write it.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#75

Post by root-parent »

_345 wrote: post #23

This makes so much sense as to why I've always felt that Opus 4.8 was leagues ahead of GPT 5.5. It's so good at taking underspecified requirements and filling in the gaps with sensible approaches for your project

The best benchmarks are the ones you create yourself.

Its not my experience Opus is leagues ahead or even superior, but in any case, since GPT 5.5 has Instant, Medium, High, Extra High and Pro...Should the comparison be with GPT on Pro, instead of Extra High as it seems to be the case in the table?

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#76

Post by Terretta »

root-parent wrote: post #75
_345 wrote: post #23

This makes so much sense as to why I've always felt that Opus 4.8 was leagues ahead of GPT 5.5. It's so good at taking underspecified requirements and filling in the gaps with sensible approaches for your project

The best benchmarks are the ones you create yourself. Its not my experience Opus is leagues ahead or even superior, but in any case, since GPT 5.5 has Instant, Medium, High, Extra High and Pro...Should the comparison be with GPT on Pro, instead of Extra High as it seems to be the case in the table?

And, in turn, Opus with ultracode?

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#77

Post by eunos »

magnio wrote: post #20

I saw on Twitter that in an ML course at Tsinghua University, one of the tests asks students to write quizzes that fail the most LLM models as possible. What if we create a benchmark that works like this and assigns ELO scores? Models fight head-to-head by writing a question, a bug, or an incomplete implementation, which the opponent has to answer, fix, or finish.

That was Fudan I think

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#78

Post by d4rkp4ttern »

root-parent wrote: post #75
_345 wrote: post #23

This makes so much sense as to why I've always felt that Opus 4.8 was leagues ahead of GPT 5.5. It's so good at taking underspecified requirements and filling in the gaps with sensible approaches for your project

The best benchmarks are the ones you create yourself. Its not my experience Opus is leagues ahead or even superior, but in any case, since GPT 5.5 has Instant, Medium, High, Extra High and Pro...Should the comparison be with GPT on Pro, instead of Extra High as it seems to be the case in the table?

I didn’t know you could get the “Chat-GPT-5.5 Pro” (the one that’s been solving Erdos problems) inside codex-cli, or maybe I misunderstood?

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#79

Post by goosejuice »

iLoveOncall wrote: post #67
CSMastermind wrote: post #42

Earlier quoted context omitted.

Man I don't know if I'm living in a crazy bubble or something but GPT 5.5 is lightyears better than Opus 4.8 for me to the point where I'm honestly wondering how you're evaluating them or what kind of work you're doing. There's specific tasks that Opus does better on like Frontend Dev and Design but for anything else 5.5 just laps it.

It's just too funny to see people arguing about "no, it's my religion that's the right one!" on HackerNews. You guys are all a lost cause.

How is attempting to benchmark llms like religion?

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#80

Post by piterrro »

> Senior engineers build features without over-specified requirements

To me this already disqualifies the benchmark. That statement is missing the most critical piece about senior engineers: the senior engineers know how to obtain input for their work on their own whether that talking to customers or using metrics. Never ever they come up with stuff on their own - that’s junior behaviour.

Until a coding agent will be able to *gather* the input on its own, its never going to be „senior”

Post reply on HN