So TL;DR benchmarking in a completely non-reproducible manner ? "Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company". So basically pinky-promise benchmarking ? I'm not sure I follow the value here ?
Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
31–40 of 147 posts
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#32> Each task comes from a private production codebase that we licensed from a real-world company How does that work?
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#33I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.
Otherwise I wouldn’t see others in the exact same codebase struggle and underutilize agents while others thrive using the exact same ones.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#34Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#35- DeepSeek V4.1 Flash
- Kimi K3
- GLM 5.3 (and flash)
- hy4-preview
- Grok 4.6
All of these can be acessed using a $10/mo OpenCode Go subscription.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#36I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#37I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.
I do think it’s the wizard not the wand at this point given a decent model. These benchmarks don’t have the wizard. Otherwise I wouldn’t see others in the exact same codebase struggle and underutilize agents while others thrive using the exact same ones.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#38this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning…
I think the bigco folks saw all the "it wrote all this code but the tests didn't pass" or "it wrote the feature but it's super brittle" and tuned the newer model+harness combinations incredibly aggressively to try to turn a lazy prompt into "median Enterprise Architecture design suggestions" to bring up the baseline, but in a way that slows you down if you don't want that.
I'm not on big enough subscriptions to want to burn a lot time just evaluating Fable/Astra comparatively until they're cheaper, heh. I can steer any of the cheaper ones just fine anyway.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#39I think these benchmarks are not that useful, e.g. this suggests Fable is better than Astra, but in practice Astra is waaaaaay faster (like 5x; it's not even close), and also waaaay less annoying to talk to. There's only two or three sane options here - you can easily try them all and pick yourself.
I'm not sure what others are doing that they're getting such different results, but I'll take Codex every day of the week.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#40The fact that gemini 3.8 flash is so high up there just tells you this is an awful benchmark. Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change. EDIT: I was using gemini cli... it's not a harness issue lol
Hard disagree. I use 3.8 flash in Antigravity a lot, and thoroughly prefer it to most Pro-class models. It's really fast, and I've had it make crazy progress on compiler-like problems that previous models including Opus simply failed at. On ultra plan you can have it going for hours, and make incremental progress with good prompting for review interrupts. It solved a problem I couldn't solve for weeks in under 6 hour…
And it just BURNS tokens like crazy.