Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CursorBench 3.1

cursor.com

Post reply on HN
Print view View original

11–20 of 107 posts

Re: CursorBench 3.1

#11

Post by mdasen »

I'm a bit skeptical.

Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price.

Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents. You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8 max gets 56, and Cursor 2.5 gets 16.

I don't doubt that Cursor works well for some people. It's beating DeepSeek v4 Pro in the DeepSWE benchmark and that's a very capable model. But I'm skeptical of the claims that it's a competitor for Opus 4.8 and GPT-5.5. It just seems convenient that their model does so well on their own benchmark while third party benchmarks have it far behind. Maybe it's a really great benchmark and a better measure than third party ones - I'd love for a cheap model to do as well as the expensive ones.

Re: CursorBench 3.1

#12

Post by famouswaffles »

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

Cursor sessions are pretty much what composer models are RL'd on. This bench and the training data are/should be basically the same distribution.

Re: CursorBench 3.1

#13

Post by shockembopper »

danfritz wrote: post #5
anilgulecha wrote: post #3

is composer 2.5 that good at that pricepoint? Seems like the gemini flash playbook of trying to get most bang for the buck.

It's my daily driver, it's fast affordable and with a bit of guidance gets the job done. I only reach for Claud when i need to plan something big or want to have a sparring partner to fire of some ideas. I think what a lot of people don't realize is that you don't need a fronteer model for 80% of coding tasks. Composer 2.5 is often more than good enough, less token hungry and way faster

I have been doing the same for quite a while now. Composer 2.5 is incredible when you’re working in the loop.

Re: CursorBench 3.1

#14

Post by BugsJustFindMe »

I've used both Composer 2.5 and GPT 5.5 (both in Cursor and in Codex) extensively, and their claim that Composer 2.5 is anywhere close in performance to GPT 5.5 is absolutely farcical. It's faster, but it's nowhere near as good.

And given that you can only use Composer with a Cursor monthly subscription, cost comparisons are pointless since an equivalently priced OpenAI subscription gets you just as much usage of the better model.

Re: CursorBench 3.1

#16

Post by datadrivenangel »

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

For lighter interactive agentic coding, where you type stuff into an IDE and a minute or three later get results back for review, composer 2.5 is honestly pretty great. The results get notably worse for larger tasks though.

Re: CursorBench 3.1

#17

Post by __natty__ »

It's hard to believe Composer 2.5 is that good. I tried to compare it with GLM 5.2 or Opus 4.6 and it lacked thinking about the problem and critical reasoning. It's great for executing plans made by other models, but even then it does some weird code manipulation that is far from how other files around actually work.

Re: CursorBench 3.1

#19

Post by gkbrk »

verse wrote: post #18

backwards X axis? is there a reason for that? it looks ridiculous

It looks very natural, cheaper is better after all. Performance axis going up, and cheapness axis going up match each other.
Post reply on HN