Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

withspecific.com

Post reply on HN
Print view View original

81–90 of 147 posts

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#81

Post by [deleted] »

starchild3001 wrote: post #55
bdlowery wrote: post #17

The fact that gemini 3.8 flash is so high up there just tells you this is an awful benchmark. Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change. EDIT: I was using gemini cli... it's not a harness issue lol

There's no such things as gemini cli these days. It's called "agy" (short for antigravity). And if you don't know what that is, you're probably 3-6 months behind already. PS: Just Googled it to confirm: Gemini CLI was deprecated on May 19th, 2026. The correct harness is called agy or antigravity for Gemini 3.8 Flash. https://developers.googleblog.com/an-important-update-transi...

[deleted]

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#82

Post by [deleted] »

starchild3001 wrote: post #55
bdlowery wrote: post #17

The fact that gemini 3.8 flash is so high up there just tells you this is an awful benchmark. Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change. EDIT: I was using gemini cli... it's not a harness issue lol

There's no such things as gemini cli these days. It's called "agy" (short for antigravity). And if you don't know what that is, you're probably 3-6 months behind already. PS: Just Googled it to confirm: Gemini CLI was deprecated on May 19th, 2026. The correct harness is called agy or antigravity for Gemini 3.8 Flash. https://developers.googleblog.com/an-important-update-transi...

[deleted]

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#83

Post by astrostl »

starchild3001 wrote: post #55
bdlowery wrote: post #17

The fact that gemini 3.8 flash is so high up there just tells you this is an awful benchmark. Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change. EDIT: I was using gemini cli... it's not a harness issue lol

There's no such things as gemini cli these days. It's called "agy" (short for antigravity). And if you don't know what that is, you're probably 3-6 months behind already. PS: Just Googled it to confirm: Gemini CLI was deprecated on May 19th, 2026. The correct harness is called agy or antigravity for Gemini 3.8 Flash. https://developers.googleblog.com/an-important-update-transi...

The benchmark page itself asserts that it used Gemini CLI as a harness. I came to the comments just because I noticed the error. For my part — using agy — I found Gemini 3.8 Flash mid.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#85

Post by pdantix »

majormajor wrote: post #38
visiondude wrote: post #20

this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning…

Do you find Fable significantly better than Opus at avoiding-overengineering? All of my recent testing of Anthropic models seems like they're tuned-to-hell to (a) be much slower than they need to be (running tests over and over during the loop vs at the end, say, even if those tests take a few minutes a pop) and (b) doing exactly that sort of "built a lot of fancy enterprisey feature-adjacent 'stuff'" even before nai…

personally, i would not rely on opus 5 end to end as it'll start getting into walls of comment slop and shitting up the codebase similar to gpt 5.5's isRecord meme.

on the other hand, having fable plan and orchestrate with opus implemention + fable reviews, is my go-to. if you give fable your guidelines up front or in your {claude,agents}.md, it will keep opus on a tight leash. opus can still write great code almost on par with fable, but it needs to be tightly constrained.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#86

Post by doctorpangloss »

prometheus1992 wrote: post #34

Does this mean they ended up sharing those private codebases with OAI, Anthropic etc? Also, the ~30% number tracks with my experience. I thought I was going insane for expecting too much from the models but they are still bad, including astra. This morning it messed something pretty trivial while fixing an issue which I was shocked to see. Also2, benchmarks don't mean much these days.

the requests went into a pipeline that turns them into de-identified, but salient, training data, yeah. everywhere except maybe bedrock.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#87

Post by lmeyerov »

esikich wrote: post #73
kwamenum86 wrote: post #64

Earlier quoted context omitted.

I spent a while in big tech and remember several unique patterns of internal code based. Your comment inspired me to try to coax ChatGPT into spitting out code that was inspired by proprietary, private code. Surprisingly, it did it with no problem - I referenced an idiom from a tech company and it wrote code that really would have only been relevant for that vertical. When I asked how it learned the pattern, it said…

Well how do you know which statement is truthful? These LLMs confidently say they know things that they don't all the time.

For us it's ~easy: we do security investigation CTFs against log dumps / DBs like Splunk. We run the AI CTF as usual.. except without the DB nor internet access. If they still get some right, that's a problem. (And most public CTFs are contaminated now for that reason.)

There's a funny middle where they may not know the exact answer confidently, but do know where to get them, or aspects of the problem. That's still a potential lift in the competition . So we also check the traces to see if there is any knowledge during intermediate step DB queries that it shouldt have based on the previous steps.

I gave a recorded talk at BSides Las Vegas a few weeks ago on 'adversarial evals' that gets into the kind of cheating we've been having to deal with.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#88

Post by thefourthchime »

freakynit wrote: post #84

This is the first set of benchmarks which match my observations around gemini-3.8-flash perfectly. This model is a true hidden gem.

Really? I gave it a trivial HTML job, and it went off for fifteen minutes. It did eventually did a do a decent job, but I can't wait that long.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#90

Post by ttul »

We built a “code atlas” that provides the LLM with a semantically queryable map of how things connect and relate in a very large and sprawling codebase that evolved over 15 years. It tends to dramatically reduce the length of time models have to spend reading code while also making sure they are aware (within their context window) of nuances that are important that might be missed were they forced to just rely on reading the code in hundreds of repositories.

I strongly recommend trying this approach out yourself. The recipe is not rocket science. Get your coding agent to take a first cut at building the atlas itself, and then manually correct it. Once you’re happy that it got things right, put an MCP on it or a CLI or whatever. And your LLMs will know what to do from there.

Post reply on HN