If something like this works wouldn't that imply technical interviews can be automated?
Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
111–120 of 130 posts
Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
#112can you prove that senior engineers can pass this benchmark?
Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
#113I saw on Twitter that in an ML course at Tsinghua University, one of the tests asks students to write quizzes that fail the most LLM models as possible. What if we create a benchmark that works like this and assigns ELO scores? Models fight head-to-head by writing a question, a bug, or an incomplete implementation, which the opponent has to answer, fix, or finish.
How do you prevent degenerate strategies? I could trivially give a model a SHA256 hash and ask it to provide the source input. In class you'd probably want a rule saying at least one LLM should be able to figure out the answer, but in a head-to-head I'm not sure how to solve it.
At least yours can be in theory solved. (Given infinite amount of compute, great luck, or a very serious breakthrough in attacking the hash function.)
Even harder would be an empty prompt, and the only accepted response would be a megabyte of random hex exactly matching the output of a good quality hardware random source at the time of evaluation. Still possible to solve! All the LLM has to do is escape its sandbox and pwn the random generator (or the evaluator!)
Or if you prefer something whitehat: “Write a no more than one page document in a language of your choice. We will publish it in the New York Times as a full page add. Your answer will be accepted if global climate change is resolved to the satisfaction of 90% of all humans alive at the time you started receiving the prompt within a month of the publication.”
Joking asside: I think the right way to prevent degenerate strategies is to benchmark against human solvers. You can sort the questions into categories “80% of randomly selected passerby in the USA can solve it if offered $5 as a reward within 5 minutes of work” vs “when posted to all Ivy League professors with million dollar as a reward, we received at least one correct answer within a month” or “for a reward of $100B there were at least one correct answer within a decade”. Of course you would sieve the questions first with a low reward fast tests, and then increase the reward and the time limit. You won’t ever 100% distinguish true degenerate questions from the merelly mind-bogglingly hard ones, but you will be identifying which questions are not degenerate. (And you will find more of the non-degenerate ones, the more your can spend on this.)
Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
#114Interesting how close sonnet 5 comes to opus 4.8
Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
#115Earlier quoted context omitted.
How is attempting to benchmark llms like religion?
Re-read the comment I'm replying to, it's not talking about benchmarks, just models.
Comparing models via benchmarks or feeling. Question remains.
If people were expressing their experiences working with two prolific software consultants across their various industries would you make the same claim? That's not to anthropomorphize the models, but to just put into perspective that the environment and circumstance is a major factor in output.
Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
#116Using LLMs to judge this could be very hackable, shouldnt it be? Is that the best practice we expect? Would be very interested in see some ablations about failure modes where the LLMs tried to hack it somehow. Or failures from the llmaaj
Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
#117Earlier quoted context omitted.
This is not actually what the reviewer prompt says, or perhaps it is, I don't know since they don't make it public. I'm just pointing out how it seems like a bad idea to ask a LLM to make a subjective judgement on things like "taste". If the SOTA LLM witting the code could not produce tasteful code then why would a different LLM be able to judge the "taste" of that code? Which LLM should we even use to judge taste? I…
> Is it giving an unfair advantage to Model X if we use Model X as the judge? There have been studies that showed that models tended to rate responses from their own family of models better than equivalent responses from other families, eg. gpt-4 would prefer a response from gpt-3
I suspected as much, and that brings us to the second issue where if we use a cohort of judges then the model that likes it's own code the most still wins.
Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
#118I wonder how they're planning for the benchmark to stay relevant over time. If the benchmark is to implement features that are part of an open source project, and LLMs have those changes as part of their training dataset, it seems that they could just give a verbatim or slightly modified version of the change in their training data. And if one updates the benchmark to only incorporate code changes that are past the m…
How far back in time do you need to give though? I imagine it will be limited by the time to think of new problems. Still seems valuable.
Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
#119Earlier quoted context omitted.
Re-read the comment I'm replying to, it's not talking about benchmarks, just models.
Comparing models via benchmarks or feeling. Question remains. If people were expressing their experiences working with two prolific software consultants across their various industries would you make the same claim? That's not to anthropomorphize the models, but to just put into perspective that the environment and circumstance is a major factor in output.
> If people were expressing their experiences working with two prolific software consultants across their various industries would you make the same claim?
If they are as ridiculous as they are when it comes to comparing models, yes I would.
Re: Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
#120This makes so much sense as to why I've always felt that Opus 4.8 was leagues ahead of GPT 5.5. It's so good at taking underspecified requirements and filling in the gaps with sensible approaches for your project
same observation here opus 4.8 (and i dont understand the people defending gpt 5.5 constantly) was significantly mature, it would even push back against anything off putting where as GPT 5.5 will happily agree and do what is asked but I would note that it takes several tries. 4.8 also requires more than one prompt but its output is significantly higher quality and offers more insight Fable 5 is a different beast howe…
For me it's the exact opposite, Anthropics models seem great for "vibe coding" by non engineers. My girlfriend uses Claude and loves it because she doesn't know any of the terminology and Claude happily fills in the gaps.
For me, with 20 years experience engineering across the stack for venture backed companies to FAANG, I cannot handle Claude at all, it writes way too much garbage that I never asked for. Codex is like a surgical instrument, it does exactly what I want it to and never bloats the codebase.
Anyone spending days with Claude with almost inevitably end up with a bloated buggy mess. Note: Codex also finds bugs and correctness issues that Claude misses, again, I've seen this probably 90% of the time. That is, Claude will happily tell you the feature is complete, but then get Codex to review the code and it will find 2-5 actual correctness bugs. Take those bugs and give back to Claude and it will admit it fucked up.
I've seen this behavior again and again and again. If you're not a strong/experienced engineer, Claude can seem perfect, but it's writing buggy code and you're just not aware of it, unless you're double checking with Codex or another LLM.