Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

withspecific.com

Post reply on HN
Print view View original

71–80 of 147 posts

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#71

Post by ImL1s »

beefsack wrote: post #67
jstummbillig wrote: post #28

I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.

There's an issue with GPT-5.6 Sol where it sometimes starts mixing thinking with output and stops working[1]. Once it starts doing that, the session is essentially cooked and you need to do a bit of gymnastics if you want to recover it. This happens to me more commonly in large projects (>100k LOC) and in those projects it seems to happen every few sessions. I feel this specific benchmark would be impacted by this mo…

[flagged]

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#72

Post by bluegatty »

dennisy wrote: post #57
ramigb wrote: post #51

Earlier quoted context omitted.

Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe? From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly u…

The reasons are highly project specific. The closer your project is to CRUD, the higher the chance of success.

yes ... 'the closer it is to normative, the higher the chances of success'.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#73

Post by esikich »

kwamenum86 wrote: post #64
lmeyerov wrote: post #18

My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point. One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.

I spent a while in big tech and remember several unique patterns of internal code based. Your comment inspired me to try to coax ChatGPT into spitting out code that was inspired by proprietary, private code. Surprisingly, it did it with no problem - I referenced an idiom from a tech company and it wrote code that really would have only been relevant for that vertical. When I asked how it learned the pattern, it said…

Well how do you know which statement is truthful? These LLMs confidently say they know things that they don't all the time.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#75

Post by nijave »

ramigb wrote: post #51
prometheus1992 wrote: post #34

Does this mean they ended up sharing those private codebases with OAI, Anthropic etc? Also, the ~30% number tracks with my experience. I thought I was going insane for expecting too much from the models but they are still bad, including astra. This morning it messed something pretty trivial while fixing an issue which I was shocked to see. Also2, benchmarks don't mean much these days.

Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe? From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly u…

Just earlier Opus 5 was supposed to migrate a Python codebase from kafka-python client-per-send to a client singleton. It decided it should add a partition key "that nobody asked for" which it admitted to at the end of the implementation.

It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution.

In fact, that was fairly counterproductive since this codebase dumps events to the topic per company so you'd get a lumpy distribution where only one partition is ever active at once (starving other consumers)

The fact it decided it should add this without prompting was a bit shocking and I'll probably try to make it explicitly clear in the context not to "do random shit that wasn't asked for"

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#76

Post by switchbak »

CuriouslyC wrote: post #63
jstummbillig wrote: post #28

I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.

Any bench that puts GLM 5.3 ahead of 5.6 Sol is highly sus. They've been my two daily drivers since release, and I like GLM 5.3, but it's definitely not better than Sol, it's more ~Terra, while being significantly slower.

I had exactly the same thoughts. I often have similar thoughts on other benchmark sites, where supposed performance is way off base from my experience.

I’m not sure what the methodology of these are, but they certainly don’t match what I experience. Maybe I need to look deeper for relevant benchmarks.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#78

Post by Grimblewald »

esikich wrote: post #73
kwamenum86 wrote: post #64

Earlier quoted context omitted.

I spent a while in big tech and remember several unique patterns of internal code based. Your comment inspired me to try to coax ChatGPT into spitting out code that was inspired by proprietary, private code. Surprisingly, it did it with no problem - I referenced an idiom from a tech company and it wrote code that really would have only been relevant for that vertical. When I asked how it learned the pattern, it said…

Well how do you know which statement is truthful? These LLMs confidently say they know things that they don't all the time.

much harder to do in OP's case, matching flavour then referencing that specific companies name when asked how it know to flavour this way? thats astronomically low for randomly selected plausible tokens without some data prior, like that companies codebase.

My own experience is opus being lousy at an extremely niche math task, but it was still easier for me to describe what it needed to do to get code and correct issues in its reasoning/working than to write myself. a minor model number change later and it's nailing everything, despite my opt-out. Its is astronomically unlikley others were working on this also, especially at that level, especially this application.

so, safe to say they _all_ train models on chats, the only difference being if you "opt out" you at least have some defence later when they steal your work and claim it as their models original output.

Post reply on HN