Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

senior-swe-bench.snorkel.ai

Post reply on HN
Print view View original

61–70 of 130 posts

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#61

Post by wwind123 »

olmo23 wrote: post #44
magnio wrote: post #20

I saw on Twitter that in an ML course at Tsinghua University, one of the tests asks students to write quizzes that fail the most LLM models as possible. What if we create a benchmark that works like this and assigns ELO scores? Models fight head-to-head by writing a question, a bug, or an incomplete implementation, which the opponent has to answer, fix, or finish.

How do you prevent degenerate strategies? I could trivially give a model a SHA256 hash and ask it to provide the source input. In class you'd probably want a rule saying at least one LLM should be able to figure out the answer, but in a head-to-head I'm not sure how to solve it.

Who knows. Maybe Mythos 5 already found a hole in SHA256, so this won't be too hard. :)

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#62

Post by wwind123 »

CSMastermind wrote: post #42
_345 wrote: post #23

This makes so much sense as to why I've always felt that Opus 4.8 was leagues ahead of GPT 5.5. It's so good at taking underspecified requirements and filling in the gaps with sensible approaches for your project

Man I don't know if I'm living in a crazy bubble or something but GPT 5.5 is lightyears better than Opus 4.8 for me to the point where I'm honestly wondering how you're evaluating them or what kind of work you're doing. There's specific tasks that Opus does better on like Frontend Dev and Design but for anything else 5.5 just laps it.

In my experience, for more mechanical refactoring work (like splitting a big source code file into multiple smaller ones), GPT 5.5 runs way faster than any of the Claude models. But for other tasks that require deeper reasoning, it's not that clear who is the winner.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#63

Post by LiamPowell »

rhdunn wrote: post #52
LiamPowell wrote: post #8

> You are a senior SWE-Bench reviewer, make no mistakes. I don't know what a better approach would look like while still remaining feasible, however this approach of telling a LLM to make a subjective judgement seems fundamentally flawed.

This approach is effectively seeding the context with how you want the LLM to behave/operate ("senior reviewer", i.e. the style of the responses you want) and the context/domain in which the LLM is operating in ("SWE-Bench"). This is common in system prompts and frames the responses. For example, you'd get different responses saying: 1. you are a pirate writing sea shanties about programming; 2. you are a news report…

This is not actually what the reviewer prompt says, or perhaps it is, I don't know since they don't make it public. I'm just pointing out how it seems like a bad idea to ask a LLM to make a subjective judgement on things like "taste". If the SOTA LLM witting the code could not produce tasteful code then why would a different LLM be able to judge the "taste" of that code?

Which LLM should we even use to judge taste? Is it giving an unfair advantage to Model X if we use Model X as the judge? Maybe we should use multiple models as the judge, but now the model that's best at recognising and praising its own code has an advantage. The whole thing is just an unsolvable problem when a LLM is the judge.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#65

Post by iLoveOncall »

fiso64 wrote: post #55

I think benchmarks like this are too subjective and narrow to be useful. For example, whether a patch "bloats" the codebase really depends on the situation: If it's building a feature that will grow in the future, or refactoring code that has a long history of bugs, then a larger patch might in fact be good. It's not clear from the blog just how much context the LLM judge receives about the long term project goals an…

> Full disclosure, I'm not a software engineer

Then maybe you should abstain, because your comment is a complete load of nonsense.

Bad code is bad code regardless of the history or scope of the feature. Maintainability is important because you can never know if a feature will be built upon in the future or not.

Bloat is bad regardless, because it increases the overall complexity of the whole software development lifecycle, for the whole team, forever (or until refactored out): It's harder to keep track of the code and how it works to write new requirements, it's harder to write, it's harder to read and review, it's harder to debug, etc.

You can write extremely poor code that has no bugs, it doesn't make it tasteful. This is simply a ridiculous statement.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#66

Post by iLoveOncall »

nsingh2 wrote: post #31
_345 wrote: post #23

This makes so much sense as to why I've always felt that Opus 4.8 was leagues ahead of GPT 5.5. It's so good at taking underspecified requirements and filling in the gaps with sensible approaches for your project

Why supply underspecified requirements in the first place? Both models are good at challenging assumptions/edge cases and asking questions to clarify, but seemingly only when explicitly asked (i.e. something like a "brainstorm" skill). I don't think either harnesses do enough to encourage the model to challenge all assumptions and ask questions, maybe because users might find it annoying. That step is basically a req…

> Why supply underspecified requirements in the first place?

Because the entire reason we use LLMs is to supposedly improve productivity?

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#67

Post by iLoveOncall »

CSMastermind wrote: post #42
_345 wrote: post #23

This makes so much sense as to why I've always felt that Opus 4.8 was leagues ahead of GPT 5.5. It's so good at taking underspecified requirements and filling in the gaps with sensible approaches for your project

Man I don't know if I'm living in a crazy bubble or something but GPT 5.5 is lightyears better than Opus 4.8 for me to the point where I'm honestly wondering how you're evaluating them or what kind of work you're doing. There's specific tasks that Opus does better on like Frontend Dev and Design but for anything else 5.5 just laps it.

It's just too funny to see people arguing about "no, it's my religion that's the right one!" on HackerNews.

You guys are all a lost cause.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#68

Post by victorbjorklund »

olmo23 wrote: post #44
magnio wrote: post #20

I saw on Twitter that in an ML course at Tsinghua University, one of the tests asks students to write quizzes that fail the most LLM models as possible. What if we create a benchmark that works like this and assigns ELO scores? Models fight head-to-head by writing a question, a bug, or an incomplete implementation, which the opponent has to answer, fix, or finish.

How do you prevent degenerate strategies? I could trivially give a model a SHA256 hash and ask it to provide the source input. In class you'd probably want a rule saying at least one LLM should be able to figure out the answer, but in a head-to-head I'm not sure how to solve it.

Maybe make the LLM:s write questions that they can solve (without seeing the question writing context) but not other LLm:s.

On the other hand then maybe a good strategy would be to write questions that the LLM just happen to have in a nich dataset in its training ”what did user5455 say to user6835?”

Nevermind my idea.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#69

Post by nsingh2 »

iLoveOncall wrote: post #66
nsingh2 wrote: post #31

Earlier quoted context omitted.

Why supply underspecified requirements in the first place? Both models are good at challenging assumptions/edge cases and asking questions to clarify, but seemingly only when explicitly asked (i.e. something like a "brainstorm" skill). I don't think either harnesses do enough to encourage the model to challenge all assumptions and ask questions, maybe because users might find it annoying. That step is basically a req…

> Why supply underspecified requirements in the first place? Because the entire reason we use LLMs is to supposedly improve productivity?

Refusing to sufficiently specify a task and hoping the model guesses correctly is not being productive. Again, these models still don't really ask questions when they should. You have to explicitly tell them to.

Specifying the problem is not extra work separate from solving it. If you skip that step, the ambiguity gets pushed into the model’s assumptions. Then you get a plausible looking answer to the wrong problem and have to waste time backing out of it.

LLMs are not magic machines that can read your mind.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#70

Post by iLoveOncall »

nsingh2 wrote: post #69
iLoveOncall wrote: post #66

Earlier quoted context omitted.

> Why supply underspecified requirements in the first place? Because the entire reason we use LLMs is to supposedly improve productivity?

Refusing to sufficiently specify a task and hoping the model guesses correctly is not being productive. Again, these models still don't really ask questions when they should. You have to explicitly tell them to. Specifying the problem is not extra work separate from solving it. If you skip that step, the ambiguity gets pushed into the model’s assumptions. Then you get a plausible looking answer to the wrong problem a…

My point is that it is much faster for me to solve the problem by writing the code than to write specifications detailed enough for the model to do the right thing in the right way.
Post reply on HN