Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

withspecific.com

Post reply on HN
Print view View original

61–70 of 147 posts

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#61

Post by 0x457 »

thereitgoes456 wrote: post #19
bdlowery wrote: post #17

The fact that gemini 3.8 flash is so high up there just tells you this is an awful benchmark. Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change. EDIT: I was using gemini cli... it's not a harness issue lol

Why so brazenly confident? Isn’t it possible that the benchmark is correct, and your experience is correct too, but you haven’t tried all the thousand different modalities of work that programming encompasses and so maybe you don’t actually have standing to judge?

very outdated experience from me: when I first tried gemini something, in an existing rust codebase, it looked around for files that would indicate if its go, javascript, java or c++ project, then declared I must have asked it build a new app in javascript and proceeded to circle around to figure out how it can install node and npm on my machine.

So I totally believe that Gemini is just bad. Which is surprising because Gemma is very good for some tasks, but I never ever had any success with Gemini, be it in cli or chat thing or anything else that has gemini branding.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#63

Post by CuriouslyC »

jstummbillig wrote: post #28

I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.

Any bench that puts GLM 5.3 ahead of 5.6 Sol is highly sus. They've been my two daily drivers since release, and I like GLM 5.3, but it's definitely not better than Sol, it's more ~Terra, while being significantly slower.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#64

Post by kwamenum86 »

lmeyerov wrote: post #18

My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point. One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.

I spent a while in big tech and remember several unique patterns of internal code based. Your comment inspired me to try to coax ChatGPT into spitting out code that was inspired by proprietary, private code. Surprisingly, it did it with no problem - I referenced an idiom from a tech company and it wrote code that really would have only been relevant for that vertical. When I asked how it learned the pattern, it said “from my learned understanding of CompanyX’s internal coding conventions”. When I asked “how do you know about those internal conventions” ChatGPT said “I don’t have access to that internal code, I overstated what I know”. Internal coding conventions are the least of our worries at this point, cat is out of the bag.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#65

Post by zeven7 »

enraged_camel wrote: post #53
didgeoridoo wrote: post #36

Earlier quoted context omitted.

Sol failing mostly on “unverified assumptions” and rarely hitting “integration errors” seems about right to me. I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. If you give it a one-liner… you’re going to have a bad (SHA-256-hash-verified) time.

>> I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myse…

I agree somewhat with the way the agents behave but feel the opposite reaction. With Fable, I get exhausted because it's always dumping out paragraphs of text that explain one approach but have some secret gotcha thrown out in the last two sentences. Then I have to pause and consider the caveat and if it matters and it happens every single time Fable responds and that constantly needing to make a decision that could radically change the approach gives me decision fatigue. I much prefer how much more decisive Astra can be.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#67

Post by beefsack »

jstummbillig wrote: post #28

I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.

There's an issue with GPT-5.6 Sol where it sometimes starts mixing thinking with output and stops working[1]. Once it starts doing that, the session is essentially cooked and you need to do a bit of gymnastics if you want to recover it.

This happens to me more commonly in large projects (>100k LOC) and in those projects it seems to happen every few sessions. I feel this specific benchmark would be impacted by this more than the smaller contrived benchmarks.

[1]: https://github.com/openai/codex/issues/37524

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#68

Post by throwaway7783 »

mupuff1234 wrote: post #59
ramigb wrote: post #51

Earlier quoted context omitted.

Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe? From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly u…

If it works but the design is terrible - is that still success?

In the short term yes. Long term, no. But I guess you can say the same thing about a team of engineers taking shortcuts
Post reply on HN