Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

senior-swe-bench.snorkel.ai

Post reply on HN
Print view View original

81–90 of 130 posts

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#81

Post by ricardobeat »

fiso64 wrote: post #72
iLoveOncall wrote: post #65

Earlier quoted context omitted.

> Full disclosure, I'm not a software engineer Then maybe you should abstain, because your comment is a complete load of nonsense. Bad code is bad code regardless of the history or scope of the feature. Maintainability is important because you can never know if a feature will be built upon in the future or not. Bloat is bad regardless, because it increases the overall complexity of the whole software development life…

>Maintainability is important because you can never know if a feature will be built upon in the future or not. Of course maintainability is important. It's almost like saying good code is important (duh). The issue is that what is or isn't maintainable depends on the problem at hand. Sometimes you need to build heavier abstractions or refactor existing code when implementing a feature because it will pay off later. O…

You buy a wooden dinner table, it is fully functional and looks perfect. It’s sturdy. You have dinner on it and it survives a few spills.

A few months later you find out it is made of PU foam and printed waxed paper. A misplaced knee could bring it down. It’s likely to completely fall apart in a year. Is that irrelevant?

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#82

Post by iLoveOncall »

goosejuice wrote: post #79
iLoveOncall wrote: post #67

Earlier quoted context omitted.

It's just too funny to see people arguing about "no, it's my religion that's the right one!" on HackerNews. You guys are all a lost cause.

How is attempting to benchmark llms like religion?

Re-read the comment I'm replying to, it's not talking about benchmarks, just models.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#83

Post by iLoveOncall »

reactordev wrote: post #74
iLoveOncall wrote: post #70

Earlier quoted context omitted.

My point is that it is much faster for me to solve the problem by writing the code than to write specifications detailed enough for the model to do the right thing in the right way.

You should try transcribing while you speak. Then you can explain and articulate the task sufficiently that the model should have enough context to complete the task to your satisfaction. Since you won’t write it.

My colleagues will thank me for speaking non-stop right next to them surely.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#84

Post by iLoveOncall »

fiso64 wrote: post #72
iLoveOncall wrote: post #65

Earlier quoted context omitted.

> Full disclosure, I'm not a software engineer Then maybe you should abstain, because your comment is a complete load of nonsense. Bad code is bad code regardless of the history or scope of the feature. Maintainability is important because you can never know if a feature will be built upon in the future or not. Bloat is bad regardless, because it increases the overall complexity of the whole software development life…

>Maintainability is important because you can never know if a feature will be built upon in the future or not. Of course maintainability is important. It's almost like saying good code is important (duh). The issue is that what is or isn't maintainable depends on the problem at hand. Sometimes you need to build heavier abstractions or refactor existing code when implementing a feature because it will pay off later. O…

I'll leave the conversation at the fact that it's painfully clear that you don't write software for a living.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#86

Post by fiso64 »

iLoveOncall wrote: post #84
fiso64 wrote: post #72

Earlier quoted context omitted.

>Maintainability is important because you can never know if a feature will be built upon in the future or not. Of course maintainability is important. It's almost like saying good code is important (duh). The issue is that what is or isn't maintainable depends on the problem at hand. Sometimes you need to build heavier abstractions or refactor existing code when implementing a feature because it will pay off later. O…

I'll leave the conversation at the fact that it's painfully clear that you don't write software for a living.

Yes, please do leave. The thing is that this isn't even necessarily about software engineering as much as it is about benchmarking/epistemology in general.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#87

Post by fiso64 »

ricardobeat wrote: post #81
fiso64 wrote: post #72

Earlier quoted context omitted.

>Maintainability is important because you can never know if a feature will be built upon in the future or not. Of course maintainability is important. It's almost like saying good code is important (duh). The issue is that what is or isn't maintainable depends on the problem at hand. Sometimes you need to build heavier abstractions or refactor existing code when implementing a feature because it will pay off later. O…

You buy a wooden dinner table, it is fully functional and looks perfect. It’s sturdy. You have dinner on it and it survives a few spills. A few months later you find out it is made of PU foam and printed waxed paper. A misplaced knee could bring it down. It’s likely to completely fall apart in a year. Is that irrelevant?

Yes it is relevant and testable. It's exactly what I meant by "a measurable increase in quality of the final product". In fact a proper test harness would reveal that problem. You are forgetting that with LLMs, testing software does not have to end at the usual unit/integration/e2e level.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#88

Post by mejutoco »

reactordev wrote: post #74
iLoveOncall wrote: post #70

Earlier quoted context omitted.

My point is that it is much faster for me to solve the problem by writing the code than to write specifications detailed enough for the model to do the right thing in the right way.

You should try transcribing while you speak. Then you can explain and articulate the task sufficiently that the model should have enough context to complete the task to your satisfaction. Since you won’t write it.

This assumes someone not articulate in writing will be articulate in talking. The most likely outcome is there will be more text with the same information. One can do a little interpretative dance as well but the clearer the requirements the better the result.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#89

Post by jghn »

piterrro wrote: post #80

> Senior engineers build features without over-specified requirements To me this already disqualifies the benchmark. That statement is missing the most critical piece about senior engineers: the senior engineers know how to obtain input for their work on their own whether that talking to customers or using metrics. Never ever they come up with stuff on their own - that’s junior behaviour. Until a coding agent will be…

I'd take this a step further, but that step also curls back to the other side a small bit.

The real skill is being able to both pull the necessary information from these sources as well as being able to intuit gaps in that knowledge based on their understanding of the business and their domain expertise & wisdom. Sometimes you can't get a perfect picture, sometimes the people who should know aren't able to tell you what they really need. You still need to do the right thing.

A benchmark like this can potentially do the second part. But I don't think any model would be good at it, for now.

Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

#90

Post by tangweigang »

piterrro wrote: post #80

> Senior engineers build features without over-specified requirements To me this already disqualifies the benchmark. That statement is missing the most critical piece about senior engineers: the senior engineers know how to obtain input for their work on their own whether that talking to customers or using metrics. Never ever they come up with stuff on their own - that’s junior behaviour. Until a coding agent will be…

[flagged]
Post reply on HN