121–130 of 151 posts
Okay, let's give software engineers a break for a bit and focus on obsoleting other high-linguistic context occupations.
RL environments building on top of each other will get these models there
needs people doing software development lifecycles to figure it out and implement
Kinda funny that their "cost-vs-performance" chart looks the same as the one for Composer 2.5[1], except that it includes Composer 2.5 at a completely different spot. What are the chances that CursorBench ranks Cursor's model highest, and Cognition's bench ranks Cognition's model highest? Both are to be RL'd from Kimi as a base model, BTW. I'd posit that it's not deliberate deception, but for both companies their tra…
Good observation. I actually started typing the same point that the chances are actually high because of train/eval overlap then realised you answered your own question with that same observation. It is interesting though! Perhaps in some way this means we should decide which eval set aligns best with our taste? Back to the blog post. This is an excellent write up of an excellent technical achievement. I have a lot o…
Earlier quoted context omitted.
Harness-wrapper tools that support multiple harnesses and allow sharing workspace features (skills, slash commands, etc.) between them will be meta.
Ironically, Devin Desktop is one of those tools. It supports any harness that supports ACP (which is most of them)—you can use Claude Code, Codex, OpenCode, etc from the Devin Desktop UI. I'm currently experimenting with OpenSpec[0] as the "framework" and using different subscriptions for different parts of the spec-driven process: Opus via Claude Code for exploration, Devin SWE for building, and GLM 5.2 via the Z.ai…
Earlier quoted context omitted.
Agreed on the likely mechanism. I'm not sure "overfitting" is even the right description. These things are of course absurdly complicated, and evaluating their quality down to a single number involves a lot of judgement and trade-offs. I think it's more "you get what you measure" which is true in human organizations too. Define a KPI and people work hard to make it go up, even if it's not quite right or has bad side-…
At this point I barely put any value in any of the benchmarks. I just use the models for coding (and related things like software product design/planning/ideation/etc.) tasks and judge them subjectively, and also see how others judge them subjectively on HN and Twitter.
…that are my own private internal suite on my own code bases where I can judge the output properly
I also measure wall clock time to completion which has been a surprising separator in practice.
We need more models that optimize for coding and that can be cheaper than frontier models, like what SWE 1.7 and composer 2.5 are trying to do. I don't think there's an effort to make something GLM-5.2 level but focused only on coding.
This isn't as easy as it sounds. Every ML model is struggling to balance between generalization and test performance. Taking a good model like GLM5.2 and just fine tuning it on coding can decrease real world performance due to mechanics like catastrophic forgetting. There is also other interesting behaviors were training on a broad training set can improve coding performance because there is positive transfer. There…
A company whose first demo was completely fraudulent announces that its model beats GPT-5.5, on its own benchmark? I’m gonna wait a little before I trust this. This whole company seems to optimize for raising money and impressing VCs. Lying about their products, ignoring consumer market to target enterprise, bragging about how they work their employees like slaves, and writing these posts full of intimidating technic…
review the top stash and tell me what's in it (grouped appropriately)
1.6 does this fine nearly instantly.
1.7 tried for 17s before I killed it
Earlier quoted context omitted.
> This tells me the benchmarks they use are cherry-picked. Which benchmarks would you have chosen instead, and why?
It honestly seems like there's not a great way to currently benchmark AI. The ideal way to run these benchmarks would be to give a 3rd party the model to run in an isolated environment so the prompts don't make their way back to the AI engineers. That seems doable for open weight models, but not for private models.
Set up a script that launches the harness for each model, prompts them to implement one of the tasks, let it churn until either tests pass or it hits some budget limit.
Then, most importantly, read the transcript and output and judge subjectively - I don't think this actually can be narrowed down to a score, although tokens burned to fix, whether it actually got the tests green etc are all good signals.
(I've done this, but so far only on a codebase that was too complicated with models that were too weak because I didn't want to spend more than a few dollars - results were inconclusive, planning on iterating on my personal benchmark in future)
Okay, let's give software engineers a break for a bit and focus on obsoleting other high-linguistic context occupations.
Like diplomacy? heh