Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

senior-swe-bench.snorkel.ai

Post reply on HN
Print view View original

91–100 of 130 posts

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#91

Post by eli »

charcircuit wrote: post #16
eli wrote: post #9

Earlier quoted context omitted.

I actually really like subjective benchmarks, so long as it's a human (ideally me) grading the results. LLM as judge never made much sense.

The issue is that you can't do unsupervised learning if you require humans.

Obviously there are advantages to not having to do work yourself.

But for a benchmark with the goal of picking a model to replace a human on some task? I really think the human should judge which is best.

I haven’t gotten very far yet but I had an idea for a personalized benchmark tool that walks through your git history and helps you craft prompts for tasks that bugs or features already implemented by hand so you can compare how different LLMs would do it.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#92

Post by HarHarVeryFunny »

Sounds more like vibe-bench.

For any professional work you care about the details.

Even for hobby work, if you are using LLMs then presumably it is to do the drudge work of coding, not making the decisions, and that goes doubly so if you are a senior developer. Sure the LLM can "fill in the details" and vibe code (or attempt to) you a compiler or whatever, but the whole reason you are doing a hobby project is presumably because you want to bring your experience to bear and build a GOOD compiler, not a generic one.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#93

Post by [deleted] »

piterrro wrote: post #80

> Senior engineers build features without over-specified requirements To me this already disqualifies the benchmark. That statement is missing the most critical piece about senior engineers: the senior engineers know how to obtain input for their work on their own whether that talking to customers or using metrics. Never ever they come up with stuff on their own - that’s junior behaviour. Until a coding agent will be…

[deleted]

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#94

Post by ricardobeat »

fiso64 wrote: post #87
ricardobeat wrote: post #81

Earlier quoted context omitted.

You buy a wooden dinner table, it is fully functional and looks perfect. It’s sturdy. You have dinner on it and it survives a few spills. A few months later you find out it is made of PU foam and printed waxed paper. A misplaced knee could bring it down. It’s likely to completely fall apart in a year. Is that irrelevant?

Yes it is relevant and testable. It's exactly what I meant by "a measurable increase in quality of the final product". In fact a proper test harness would reveal that problem. You are forgetting that with LLMs, testing software does not have to end at the usual unit/integration/e2e level.

But how is that testable? If your test is validating the rigidity, water resistance, etc, they will all pass even if the underlying material is a bad choice. Or the glue will degrade in six months.

You can't test if a codebase will be extensible or maintainable as requirements change in the future, if the abstraction level or architecture is sound - that's down to code quality measures like the ones used here. LLMs are very good at slightly cheating to pass tests even when the implementation is wrong. Introducing subjectivity - the kind of input a human will provide - leads to improved output.

https://senior-swe-bench.snorkel.ai/blog/2026-06-16-how-it-w...

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#95

Post by 0xbadcafebee »

FeepingCreature wrote: post #35
0xbadcafebee wrote: post #19

The "tasteful solves" is codified cargo culting. The software industry has a tendency to anthropomorphize software while playing to the ego of the programmer. The programmer imagines they are creating a "beautiful" artistic expression. Good code becomes "tasteful", as a software artist must have "good taste" to tell the good software from the bad software. Good quality lacks "bad smells", because a good artist has fi…

Taste is just quality by instinct. At sufficient (and not all that long) timescales, a tasteless product will be more and more difficult to make work at all.

So software engineering quality is vibes. All coding is vibe coding.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#97

Post by dang »

0xbadcafebee wrote: post #95
FeepingCreature wrote: post #35

Earlier quoted context omitted.

Taste is just quality by instinct. At sufficient (and not all that long) timescales, a tasteless product will be more and more difficult to make work at all.

So software engineering quality is vibes. All coding is vibe coding.

Could you please not post in the flamewar style to HN? We're trying to avoid that here, and we've had to ask you this many times over the years.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#98

Post by dang »

Madmallard wrote: post #12

[flagged]

Would you please stop posting like this? You've been doing it repeatedly, and it degrades the threads. In fact the majority of your recent comments have been this sort of shallow, dimissive, snarky stuff. That is not what this site is for, and destroys what it is for.

If you want to express your substantive points thoughtfully, that of course would be fine.

If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#99

Post by dang »

kerlenton wrote: post #64

[flagged]

Can you please not post AI-generated or AI-edited comments to HN? It's not allowed here - see https://news.ycombinator.com/newsguidelines.html#generated and https://news.ycombinator.com/item?id=47340079.

Of course, it's impossible to know for sure what was LLM processed or not, but some of your posts (like this one) are getting classified that way.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#100

Post by m3kw9 »

_345 wrote: post #23

This makes so much sense as to why I've always felt that Opus 4.8 was leagues ahead of GPT 5.5. It's so good at taking underspecified requirements and filling in the gaps with sensible approaches for your project

Better for vibe coders who always under specify. But at what point does it know you are under specifying but you have properly specified and it did it over your specification?
Post reply on HN