Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

withspecific.com

Post reply on HN
Print view View original

101–110 of 147 posts

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#101

Post by skhameneh »

Some things in this seem reasonable, but others just don’t make sense and there’s crucial details missing (like reasoning levels and what harness was used).

For example, I found Kimi K3 to use more tokens than some other models, which caused it to cost twice as much purely because of the token volume. This experience lines up with ArtificialAnalysis’s benchmarks, but not these.

There’s a number of other comparisons here that don’t match up with my experience or other benchmarks. By many accounts, this is the outlier.

I could attribute the differences to harnesses used or something like reasoning levels, but none of those details are published.

While this seems interesting, I can’t take this seriously.

Correction: The harnesses are listed as a column, I missed that. My other concerns and questions still remain, it’s unclear why some of their results are the outlier that does not match my experience, ArtificalAnalysis’s benchmarks, or some of the experiences of others commenting.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#102

Post by jaggederest »

asa123 wrote: post #95
nijave wrote: post #75

Earlier quoted context omitted.

Just earlier Opus 5 was supposed to migrate a Python codebase from kafka-python client-per-send to a client singleton. It decided it should add a partition key "that nobody asked for" which it admitted to at the end of the implementation. It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution. In fact, that was fairly cou…

One thing I've always been curious about is, often times it seems that models don't seem to have these addendums like "dont do excessive/random shit" by default? Or I suppose, if it did have something like that, and still screws up like in your example, it clearly isn't working, so assume that it doesn't. But the phrase "don't do random shit", semantically, from a monkey's paw perspective, could imply curtailing the…

I think of it as a tradeoff between creativity and specificity. Every instruction you give reduces creativity, and at best, increases specificity (I imagine a lot of prompts like "make no mistakes" do literally nothing but pollute context, but I haven't evaled them)

So if you're doing something very ordinary, fewer instructions result in better results. If you're doing something fairly off-piste, you have to give instructions to that effect and accept less creativity. For situations where you want it to do something extremely specific, tons of instructions and accept that you're going to get much closer output but much worse "intelligence"

Another way to think about it is Type 1 and Type 2 errors or sensitivity and specificity from statistical testing - do you want an agent that solves any problem but goes off the rails 10% of the time, or do you want an agent that can only solve 10% of the problems but nails them 100% of the time (sensitivity and specificity, respectively)

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#103

Post by arshxyz »

skhameneh wrote: post #101

Some things in this seem reasonable, but others just don’t make sense and there’s crucial details missing (like reasoning levels and what harness was used). For example, I found Kimi K3 to use more tokens than some other models, which caused it to cost twice as much purely because of the token volume. This experience lines up with ArtificialAnalysis’s benchmarks, but not these. There’s a number of other comparisons h…

They seem to be using the provider's harness for each

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#104

Post by glub »

I'm surprised Sol and Astra are leading "Unverified assumption" metric and Fable is better there.

I run Fable as my main model with Sol as advisor that watches every turn. Fable likes to throw around assumptions that it didn't check that are simply false, and Sol always goes to actually verify them and then alert Fable it's assuming things.

I've tried reversing this pairing with Fable as advisor. It'll just sit there going "sounds good"

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#107

Post by jeffybefffy519 »

jstummbillig wrote: post #28

I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.

Honestly matches my experience with Sol and the fact that all the random message boards appearing on the internet are attributed to Sol then it makes sense that its the one which makes up most requirements.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#108

Post by glub »

zeven7 wrote: post #65
enraged_camel wrote: post #53

Earlier quoted context omitted.

>> I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myse…

I agree somewhat with the way the agents behave but feel the opposite reaction. With Fable, I get exhausted because it's always dumping out paragraphs of text that explain one approach but have some secret gotcha thrown out in the last two sentences. Then I have to pause and consider the caveat and if it matters and it happens every single time Fable responds and that constantly needing to make a decision that could…

I'm the opposite. Every time I've let Sol/Astra be decisive, I ended up with an overengineered mess.

I much prefer getting alerted when there's more than 1 approach to the problem and it's discovered mid-implementation.

I don't want to do the grunt work of writing code, but I do want to know the architecture and be responsible for the decisions.

Fable is also very good at pushing back when I propose something that will cost me. E.g. I'm working on a configuration layer above nix to manage my homelab fleet declaratively, and I tend to get into "config as new language", where Fable just goes - let's not do that.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#109

Post by ghoshbishakh »

glub wrote: post #104

I'm surprised Sol and Astra are leading "Unverified assumption" metric and Fable is better there. I run Fable as my main model with Sol as advisor that watches every turn. Fable likes to throw around assumptions that it didn't check that are simply false, and Sol always goes to actually verify them and then alert Fable it's assuming things. I've tried reversing this pairing with Fable as advisor. It'll just sit there…

I suggest you flip them. The verifier role will always verify. You will see Sol making assumptions and Fable fixing them.

But I agree - Fable makes some spectacular assumptions (which are poor assumptions).

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#110

Post by LewisVerstappen »

glub wrote: post #104

I'm surprised Sol and Astra are leading "Unverified assumption" metric and Fable is better there. I run Fable as my main model with Sol as advisor that watches every turn. Fable likes to throw around assumptions that it didn't check that are simply false, and Sol always goes to actually verify them and then alert Fable it's assuming things. I've tried reversing this pairing with Fable as advisor. It'll just sit there…

What do you use as the harness?
Post reply on HN