Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

withspecific.com

Post reply on HN
Print view View original

131–140 of 147 posts

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#131

Post by janaksunil »

mupuff1234 wrote: post #59
ramigb wrote: post #51

Earlier quoted context omitted.

Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe? From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly u…

If it works but the design is terrible - is that still success?

this is a good question. what would make you reject an otherwise working PR on design grounds?

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#132

Post by janaksunil »

IshKebab wrote: post #12

I think these benchmarks are not that useful, e.g. this suggests Fable is better than Astra, but in practice Astra is waaaaaay faster (like 5x; it's not even close), and also waaaay less annoying to talk to. There's only two or three sane options here - you can easily try them all and pick yourself.

that's fair - for long horizon engineering tasks would speed still matter?

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#134

Post by janaksunil »

strobe wrote: post #32
demibabs wrote: post #11

> Each task comes from a private production codebase that we licensed from a real-world company How does that work?

lot of ads everywhere offering to buy your codebase of real product/star up even it long gone or failed (offer usually price per lines of code). So most likely that they have bunch of abandoned codebases between small and medium sizes and probably also some fake codebases as well.

we manually vet all codebases and companies

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#135

Post by janaksunil »

glub wrote: post #104

I'm surprised Sol and Astra are leading "Unverified assumption" metric and Fable is better there. I run Fable as my main model with Sol as advisor that watches every turn. Fable likes to throw around assumptions that it didn't check that are simply false, and Sol always goes to actually verify them and then alert Fable it's assuming things. I've tried reversing this pairing with Fable as advisor. It'll just sit there…

would love to chat and learn more about your set up! here's my email - janak@withspecific.com

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#136

Post by cbg0 »

traceroute66 wrote: post #13
kadoban wrote: post #7

Earlier quoted context omitted.

If it builds up history and perceived reliability, this type of thing can be valuable. You're giving up transparency for it being harder to game.

> You're giving up transparency for it being harder to game But then if we take that argument to its natural extreme, surely it means people should take the marketing bullshit published in the 100-page system cards published by Anthropic & co as "valuable" too ?

As long as the ones offering the benchmark aren't trying to sell you something and have no affiliation with one of the companies on the page I'll take it as opposed to having the benchmark rendered useless in 3 months when the next models drop.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#137

Post by traceroute66 »

InsideOutSanta wrote: post #24
traceroute66 wrote: post #15

Earlier quoted context omitted.

> How does that work? My gut feeling is that any serious real-world company with a proprietary codebase worth looking at would not be handing out the crown jewels to a third party. License or not. I don't doubt somebody licensed their codebase to them, I just have my doubts about who the "who" could be.

Code isn't worth all that much if you don't own the associated IP, mainly copyright. And even if you disagree with that premise, if you trust that they can keep the code secret, it's basically free money. At any rate, I'm not sure it matters whose codebase it is. I'd even say that a shitty codebase might make for a better test.

The point I'm making is that companies who are serious enough to want to keep their code-base in-house and off the various online repo services are also the kind of companies who are strict about what you can and cannot do with LLMs (if they permit use of LLMs at all).

So it does not make sense that the same companies would then magically sign-off on allowing their entire codebase to be spoon-fed into a whole bunch of LLMs for benchmarking.

Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

#140

Post by barbegal »

Without a human to benchmark against it's really tough to gauge how good these models are vs how good the task definitions and existing codebases are.

My intuition from the example full instructions are that the tasks are poorly specified which results in ~60% failures due to bad assumptions and missing requirements.

Post reply on HN