Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

withspecific.com

Post reply on HN
Print view View original

91–100 of 149 posts

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#91

Post by hedgehog »

ttul wrote: post #90

We built a “code atlas” that provides the LLM with a semantically queryable map of how things connect and relate in a very large and sprawling codebase that evolved over 15 years. It tends to dramatically reduce the length of time models have to spend reading code while also making sure they are aware (within their context window) of nuances that are important that might be missed were they forced to just rely on rea…

How did you organize the atlas? I've tried a few things including embeddings and clustering files based on how often they change in the same commits, haven't yet found anything I want to bake into my tooling.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#92

Post by enraged_camel »

zeven7 wrote: post #65
enraged_camel wrote: post #53

Earlier quoted context omitted.

>> I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myse…

I agree somewhat with the way the agents behave but feel the opposite reaction. With Fable, I get exhausted because it's always dumping out paragraphs of text that explain one approach but have some secret gotcha thrown out in the last two sentences. Then I have to pause and consider the caveat and if it matters and it happens every single time Fable responds and that constantly needing to make a decision that could…

I haven't run into the verbosity issue since they added the "Concise" outputStyle, and Fable 5.1 has been even better about not outputting word slops.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#94

Post by m3kw9 »

ttul wrote: post #90

We built a “code atlas” that provides the LLM with a semantically queryable map of how things connect and relate in a very large and sprawling codebase that evolved over 15 years. It tends to dramatically reduce the length of time models have to spend reading code while also making sure they are aware (within their context window) of nuances that are important that might be missed were they forced to just rely on rea…

is this like Aider's repo map?

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#95

Post by asa123 »

nijave wrote: post #75
ramigb wrote: post #51

Earlier quoted context omitted.

Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe? From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly u…

Just earlier Opus 5 was supposed to migrate a Python codebase from kafka-python client-per-send to a client singleton. It decided it should add a partition key "that nobody asked for" which it admitted to at the end of the implementation. It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution. In fact, that was fairly cou…

One thing I've always been curious about is, often times it seems that models don't seem to have these addendums like "dont do excessive/random shit" by default? Or I suppose, if it did have something like that, and still screws up like in your example, it clearly isn't working, so assume that it doesn't.

But the phrase "don't do random shit", semantically, from a monkey's paw perspective, could imply curtailing the model's creativity and 'thinking out of the box' capacity, that might have existed in its 'reasoning' process. So I'm always concerned if it's possible that, adding these phrases might be part of the reason why a model performs dumber than it should.

I don't have empirical evidence to support that supposition though.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#96

Post by freakynit »

thefourthchime wrote: post #88
freakynit wrote: post #84

This is the first set of benchmarks which match my observations around gemini-3.8-flash perfectly. This model is a true hidden gem.

Really? I gave it a trivial HTML job, and it went off for fifteen minutes. It did eventually did a do a decent job, but I can't wait that long.

I'm using it through antigravity cli .. and in every single run (100's by now), this model was fast, and the outputs were of good quality. Of course, not at Astra or Fable level, but, close to like Sol-low level.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#97

Post by sureMan6 »

thefourthchime wrote: post #88
freakynit wrote: post #84

This is the first set of benchmarks which match my observations around gemini-3.8-flash perfectly. This model is a true hidden gem.

Really? I gave it a trivial HTML job, and it went off for fifteen minutes. It did eventually did a do a decent job, but I can't wait that long.

I gave it a trivial HTML job and it messed it up in several ways including being lazy and lying about results

It's probably very hit or miss like everything with LLMs but I was really surprised it performed that badly

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#98

Post by fireant »

kwamenum86 wrote: post #64
lmeyerov wrote: post #18

My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point. One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.

I spent a while in big tech and remember several unique patterns of internal code based. Your comment inspired me to try to coax ChatGPT into spitting out code that was inspired by proprietary, private code. Surprisingly, it did it with no problem - I referenced an idiom from a tech company and it wrote code that really would have only been relevant for that vertical. When I asked how it learned the pattern, it said…

You cannot introspect model training by prompting the model. Whatever answer it hallucinated on your query "where do you know that from" has almost certainly very low bearing on reality.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#100

Post by anon7000 »

nijave wrote: post #75
ramigb wrote: post #51

Earlier quoted context omitted.

Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe? From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly u…

Just earlier Opus 5 was supposed to migrate a Python codebase from kafka-python client-per-send to a client singleton. It decided it should add a partition key "that nobody asked for" which it admitted to at the end of the implementation. It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution. In fact, that was fairly cou…

This is why I like to really interrogate Claude on what it’s planning to do, what the implementation will look like etc before writing code. Without enough direction and time thinking about it, it can definitely go in an unintended direction
Post reply on HN