I'd love to see these: - DeepSeek V4.1 Flash - Kimi K3 - GLM 5.3 (and flash) - hy4-preview - Grok 4.6 All of these can be acessed using a $10/mo OpenCode Go subscription.
3 of those are already in there.
Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
41–50 of 147 posts
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#42this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning…
Anecdotally, +1. I’d also say this benchmark matches my experiences and how much I trust the model output
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#43I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.
Why would you get emotional over a model? They got you that good?
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#44I'd love to see these: - DeepSeek V4.1 Flash - Kimi K3 - GLM 5.3 (and flash) - hy4-preview - Grok 4.6 All of these can be acessed using a $10/mo OpenCode Go subscription.
Here's the list:
1 Fable 5.1 38.8%
2 GPT-6 Astra 33.8%
3 Gemini 3.8 Flash 31.2%
4 GLM 5.3 28.8%
5 Grok 4.6 23.8%
5 Muse Spark 1.3 23.8%
7 Kimi K3 18.8%
8 GPT-5.6 Sol 16.2%
See number 4, 5 and 7.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#45So TL;DR benchmarking in a completely non-reproducible manner ? "Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company". So basically pinky-promise benchmarking ? I'm not sure I follow the value here ?
Lots of private benchmarks already exist, where you have to trust the tester (ex Artificial Analysis, Arc-agi).
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#46Gemini 3.8 flash has been incredible for our agents. For us, It performs better than any other model except Fable.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#47The high score of Gemini 3.8 Flash vibes with my experience anecdotally. While it often goes off the rails with open-ended questions (which is a strength if taken with care), it is also a good at solving issues in a well-defined environment like an enterprise codebase.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#48Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#49this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning…
Do you find Fable significantly better than Opus at avoiding-overengineering? All of my recent testing of Anthropic models seems like they're tuned-to-hell to (a) be much slower than they need to be (running tests over and over during the loop vs at the end, say, even if those tests take a few minutes a pop) and (b) doing exactly that sort of "built a lot of fancy enterprisey feature-adjacent 'stuff'" even before nai…
i think this likely depends on workflow. for me, the first step is always a plan file artifact on disc, which i heavily review and go back and forth until satisfied. i often have to split the plan into multiple phases because agents are still poor at assessing how large a change will be. sometimes before even starting the plan the task is to create a harness for validation (a way for agent to check their work). the codebase you drop them in also greatly changes how much “over” engineering they do. i think if you are good about reviewing plan files and managing scope for your agent these sort of issues fade away. in general though, gpt models are faster, more token efficient, better technical writers, and yeah i still find Fable 1.0 to be a step up even though it’s slower because it’s actually a tad smarter (and the more important and dynamic the change / feature the more fable 1.0 shows its slight edge over astra)
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#50[deleted]