CursorBench 3.1 • Hacker News Board

Re: CursorBench 3.1

#71

Post by extr » Thu, Jul 02, 2026, 4:19 PM UTC

burmanm wrote: post #38

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

DeepSWE is slightly flawed in the sense that is uses only its own harness and that causes issues on models that are not correctly supported by it. There's huge amount of evidence that the harness plays a big role in how these models work and yet DeepSWE entirely removes that (and has probably only tested that it works fine with some favourite model of them). There's also issues with cost calculation (as that harness…

I think DeepSWE is flawed in a different way: the tasks look like someone took a bunch of big highly technical PRs they found really well done, and inverted it into specs for agents to autistically execute. This is not really how people use agents in practice IMO. And it's why DeepSWE is so generous to OAI models, rigid task execution is the thing they're best at. I think FrontierCode matches the vibes a lot better.

Re: CursorBench 3.1

#72

Post by leerob » Thu, Jul 02, 2026, 4:58 PM UTC

CuriouslyC wrote: post #67

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

Not hard to understand what's going on here. They RL'd around patterns in their data and specific capabilities, so of course they'd construct a benchmark that's aligned with the training set. Ironically, their benchmark might be more accurate than artificial analysis for a narrow slice of things that Cursor's Eigencustomer is really interested in. Otherwise I'd take it as just another data point.

(I work at Cursor) CursorBench includes many evals from actual engineering tasks from the Cursor team, which include our private codebase. This codebase is held-out from training so models haven't seen it, including Composer.

Re: CursorBench 3.1

#73

Post by bmurphy1976 » Thu, Jul 02, 2026, 5:00 PM UTC

__natty__ wrote: post #17

It's hard to believe Composer 2.5 is that good. I tried to compare it with GLM 5.2 or Opus 4.6 and it lacked thinking about the problem and critical reasoning. It's great for executing plans made by other models, but even then it does some weird code manipulation that is far from how other files around actually work.

I'm not using Cursor at the moment, but when I did (not too long ago) my experience was similar. Plan with Opus, implement with Composer, clean up with Opus.

Composer did a competent but not amazing job with a good plan. What I really liked though is it was fast! Opus could take 30 minutes to do something Composer would get done in 5-10 minutes. Of course the output wasn't perfect, but that's why I'd do a cleanup pass using Opus or Codex.

It's all a balance though, constantly changing and completely dependent on the problem you're solving. I just remain flexible and adapt my process to what's working best in the moment.

Re: CursorBench 3.1

#74

Post by leerob » Thu, Jul 02, 2026, 5:03 PM UTC

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

(I work at Cursor) When Composer 2.5 launched, we initially scored very competitively on AA's composite benchmark. I believe 3rd place overall. They have recently updated to use DeepSWE, which has more of a focus on very long-horizon tasks, and Composer isn't as good at those yet. We're aware and working on this for our next model.

Overall, some benchmarks show Composer doing well, others not so much. We think the model is very capable at the given price point. There's lots to improve! If you see any specific behaviors or places the model isn't very good, lmk here or can email me lrobinson at cursor.com.

Re: CursorBench 3.1

#75

Post by leerob » Thu, Jul 02, 2026, 5:06 PM UTC

mi_lk wrote: post #58

Cursor: Find me another benchmark where Composer 2.5 is a top 10 frontier coding model

(I work at Cursor) We score well on Terminal-Bench and SWE-bench Multilingual. DeepSWE, not so great yet, as it's more for very long-horizon tasks. We're planning to include more public benchmarks in our next model release.

Re: CursorBench 3.1

#76

Post by artooro » Thu, Jul 02, 2026, 5:14 PM UTC

leerob wrote: post #74

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

(I work at Cursor) When Composer 2.5 launched, we initially scored very competitively on AA's composite benchmark. I believe 3rd place overall. They have recently updated to use DeepSWE, which has more of a focus on very long-horizon tasks, and Composer isn't as good at those yet. We're aware and working on this for our next model. Overall, some benchmarks show Composer doing well, others not so much. We think the mo…

I never run long horizon tasks. So Composer 2.5 is great.

Re: CursorBench 3.1

#77

Post by giancarlostoro » Thu, Jul 02, 2026, 5:35 PM UTC

leerob wrote: post #74

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

(I work at Cursor) When Composer 2.5 launched, we initially scored very competitively on AA's composite benchmark. I believe 3rd place overall. They have recently updated to use DeepSWE, which has more of a focus on very long-horizon tasks, and Composer isn't as good at those yet. We're aware and working on this for our next model. Overall, some benchmarks show Composer doing well, others not so much. We think the mo…

How does it compare to a $100 Claude subscription at $60? Especially in terms of how much of it I can use, because I havent found anything that is in the US that can get me similar usage as Claude at $100 per month or less, really open to alternatives.

Grok build only gave me roughly 10 hours of use for $40 for the entire month...

I don't even care about long horizon, can I use it a reasonable amount of time through the month? I use AI for hobby projects, Claude gets me quite far, but I tire of dropping $100 every month. I'm not sending my money to some Chinese firm that now has access to my computer.

Re: CursorBench 3.1

#78

Post by forgot-my-pw » Thu, Jul 02, 2026, 5:38 PM UTC

leerob wrote: post #74

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

(I work at Cursor) When Composer 2.5 launched, we initially scored very competitively on AA's composite benchmark. I believe 3rd place overall. They have recently updated to use DeepSWE, which has more of a focus on very long-horizon tasks, and Composer isn't as good at those yet. We're aware and working on this for our next model. Overall, some benchmarks show Composer doing well, others not so much. We think the mo…

Even with the new benchmark, Composer 2.5 seems to be just a bit worse than Opus 4.7. So I assume it's going to be about similar with Sonnet 5.0 at 1/6 of the cost.

Re: CursorBench 3.1

#79

Post by ai_slop_hater » Thu, Jul 02, 2026, 5:46 PM UTC

leerob wrote: post #74

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

(I work at Cursor) When Composer 2.5 launched, we initially scored very competitively on AA's composite benchmark. I believe 3rd place overall. They have recently updated to use DeepSWE, which has more of a focus on very long-horizon tasks, and Composer isn't as good at those yet. We're aware and working on this for our next model. Overall, some benchmarks show Composer doing well, others not so much. We think the mo…

Don't lie. You forked a Chinese model.

Re: CursorBench 3.1

#80

Post by BugsJustFindMe » Thu, Jul 02, 2026, 6:43 PM UTC

leerob wrote: post #74

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

(I work at Cursor) When Composer 2.5 launched, we initially scored very competitively on AA's composite benchmark. I believe 3rd place overall. They have recently updated to use DeepSWE, which has more of a focus on very long-horizon tasks, and Composer isn't as good at those yet. We're aware and working on this for our next model. Overall, some benchmarks show Composer doing well, others not so much. We think the mo…

> We think the model is very capable at the given price point.

The "price point" comparison is a lie though because Composer is only available with a monthly Cursor subscription, and Cursor's external-model-per-token charges for other models are not representative of what other models' monthly subscribers get. An OpenAI $200 subscription gets you at least as much GPT 5.5 as a $200 Cursor subscription gets you Composer 2.5.