Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
1–10 of 147 posts
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#2A bit of a meta question: what are the most relevant benchmarks by now?
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#3I’ve been able to offload most tasks (coding or eles) to Codex since 5.3-codex with extra high thinking
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#4So TL;DR benchmarking in a completely non-reproducible manner ?
"Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company".
So basically pinky-promise benchmarking ?
I'm not sure I follow the value here ?
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#5Wake me up when September ends or when I can do this locally.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#6A bit of a meta question: what are the most relevant benchmarks by now?
Nvidia and OpenAI claimed AGI, but you still have a job.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#7So TL;DR benchmarking in a completely non-reproducible manner ? "Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company". So basically pinky-promise benchmarking ? I'm not sure I follow the value here ?
If it builds up history and perceived reliability, this type of thing can be valuable. You're giving up transparency for it being harder to game.
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#8A bit of a meta question: what are the most relevant benchmarks by now?
Epoch.ai has a global score and tracks many benchmarks: https://epoch.ai/benchmarks
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#9So TL;DR benchmarking in a completely non-reproducible manner ? "Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company". So basically pinky-promise benchmarking ? I'm not sure I follow the value here ?
Doesn’t it ultimately have to be this way, to prevent saturation?
Re: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
#10[dead]