Viewing profile — lieret

We disallow decompilation because we want this task to answer the question of "how well can LMs program from scratch if there's no structure given", not "how well can LMs stitch to…

Current evals test LMs on tasks: "fix this bug," "write a test" But we code to achieve goals: maximize revenue, cut costs, win users Meet CodeClash: LMs compete via their codebases…

[On swe-bench team] We read and analyzed a lot of trajectories but seems like only recently models have started to exploit this in a small fraction of instances. But yes, clearly s…

[On the SWE-bench team] As someone pointed out SWE-bench Verified is a subset of tasks that were reviewed to be solvable (i.e., have enough context in the task description) as well…

[Also on the SWE-bench team] Part of the reason why this didn't surface earlier was that it only seems to affect more recent models, maybe the result of reward hacking during postt…

What if your agent uses a different LM at every turn? We let mini-SWE-agent randomly switch between GPT-5 and Sonnet 4 and it scored higher on SWE-bench than with either model sepa…

I think gpt-5-mini should really help them. At least from these benchmark scores, there probably shouldn't be a huge performance degradation for letting gpt-5-mini drive most of th…

We evaluated the new GPT models with a minimal agent on SWE-bench verified. GPT-5 scores 65%, mini 60%, nano 35%. Still behind Opus 5 (68%), on par with Sonnet 4 (65%). But a lot c…

Sorry, I missed that! That's a little bit out of the scope of this project (because we were aiming for the bare minimum of what is needed to get a performative agent — and unfortun…

Hello from the SWE-bench/SWE-agent team at Princeton/Stanford. When we created the SWE-bench benchmark in 2023 from hundreds of real-life GitHub issues/pull requests, the highest s…

In 2024, we developed SWE-bench and SWE-agent at Princeton University and helped kickstart the coding agent revolution. Back then, LMs were optimized to be great at chatting, but n…