[dead]
Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
151–160 of 166 posts
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#152I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don’t go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR. The thing that takes the most time is finding the ex…
Don't you need to run each model several times and take some average since their outputs can change for a fixed prompt? How do you test for that?
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#153I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don’t go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR. The thing that takes the most time is finding the ex…
Don't you need to run each model several times and take some average since their outputs can change for a fixed prompt? How do you test for that?
Yes at temperature 1 it would need to be averaged. The mechanical scoring is hidden specs through and if they don’t pass that it’s not often a result of temp, but instruction following or verification (different flavor of instruction following given what they are told to do). What’s surprising is just how well they do on this dimension even with the variance. On my tasks anyway, maybe Rails work isn’t that complicated. It’s probably not that I am an amazing ticket writer.
I have run multiple with the same model through issues to see the output. What differs run to run on the same model is style, likelihood to find subtle bugs and chance to correctly guess author intent. I don’t score them on those things though, only note them.
To me, multiple vertical samples is more useful to the coder who isn’t benchmarking these as their job. The average is abstract to you if you’re not going to pointing them at a ticket more than once, but seeing their output over 10 different tickets will give you a window into model tendencies, strengths and weaknesses and help you understand how to write for them. Overall the exercise has made me a better issue writer.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#154Very cool work. Please also run on new DS Flash 4.1, an important oss model nowsday
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#155I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don’t go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR. The thing that takes the most time is finding the ex…
How do you grade them against the accepted PR?
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#156Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#157Earlier quoted context omitted.
I spent a while in big tech and remember several unique patterns of internal code based. Your comment inspired me to try to coax ChatGPT into spitting out code that was inspired by proprietary, private code. Surprisingly, it did it with no problem - I referenced an idiom from a tech company and it wrote code that really would have only been relevant for that vertical. When I asked how it learned the pattern, it said…
You cannot introspect model training by prompting the model. Whatever answer it hallucinated on your query "where do you know that from" has almost certainly very low bearing on reality.
True. It sort of doesn’t matter how it was trained. The fact that it knew the company based on an internal idiom is super surprising
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#158Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#159If anyone wants this kind of benchmark for their own codebase, happy to set you up with superconductor.com/benchmark
1. Import your own PRs 2. We find the original spec or infer the spec 3. Agents you select (eg claude code opus 5, codex gpt 6 astra high, pi kimi k3, etc) implement the spec (starting from the parent commit of the PR, with the git history is pruned so they can’t look up the implementation) 4. Three judge LLMs grade agent solution given spec, PR implementation, and a rubric 5. You see the scores
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#160The LLM vendors say their AI is going to kill us all and started calling their latest releases AGI. In the meanwhile realistic benchmarks like these ones, show they can only complete 10% to 15 % of the tasks, and now that Jon Skeet, Marc Gravell, BalusC, and Darin Dimitrov have gone on strike they will be progressively worst.