Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CursorBench 3.1

cursor.com

Post reply on HN
Print view View original

41–50 of 107 posts

Re: CursorBench 3.1

#41

Post by WinstonSmith84 »

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

that benchmark seems to match my experience. GPT 5.5 is significantly better than Opus 4.8, last time I tried composer 2.5 it was truly dumb, and Fable to me looks to be on par with GPT 5.5 but .. different overall ... The best is to have a LLM-peer-review between GPT and Opus (now Fable) for best outcome.

Re: CursorBench 3.1

#42

Post by maxdo »

muzani wrote: post #39
mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

Anecdotally, I find Composer 2.5 to be useless. I do use light LLMs like Claude Haiku and some of Cursor's older free models, but Composer is negative productivity for me.

The opposite , I use for everything like trigger and monitor a 10 steps release process using composer , a very capable model

Re: CursorBench 3.1

#43

Post by maxdo »

The most interesting part is costs . Gpt 5.5 and sonnet 5 cost same amount of money as GLM 5.2 but are more capable models

Re: CursorBench 3.1

#44

Post by budsniffer952 »

tekacs wrote: post #6

I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO. Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to k…

>Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of

Is a single thing in your post demonstrable, or are we just supposed to take your word for it? Because all of this stuff sounds laughably subjective.

Re: CursorBench 3.1

#45

Post by cbg0 »

o10449366 wrote: post #2

I feel like this benchmark reiterates my disbelief that anyone uses the latest Anthropic models for any productive work. They seem to be the best at burning tokens and spawning unnecessary subagents even for well-defined and tightly scoped tasks. Can we get a count of people that have had Claude read irrelevant documents or perform unnecessary web searches even when told not to from the beginning? I'm starting to won…

I've had no problems like the ones you've mentioned while using Opus 4.8. It does overthink stuff with higher effort levels but that's kind of expected.

Re: CursorBench 3.1

#46

Post by anon7000 »

datadrivenangel wrote: post #16
mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

For lighter interactive agentic coding, where you type stuff into an IDE and a minute or three later get results back for review, composer 2.5 is honestly pretty great. The results get notably worse for larger tasks though.

Agreed. It’s worse than Opus of course. But Opus takes more than 10x longer to give you something to look at. I’m not kidding, I “benchmarked” a real ticket I was working on. Opus 4.7 took more than 30min. Opus 4.8 took over an hour. Composer 2.5 took 5min on the exact same prompt & local setup. My subjective review is that composer’s code was only like 10-20% worse. It still worked, it was just a bit less clean and a little more hacky. But it’s not like Opus is flawless either. At the end of the day, if it takes an hour to get to draft code I can look at and iterate on… that’s fucking impossible for me. Unless it did an excellent job. But as long as I still need to review and follow up with changes, Opus is just too slow. It’s really frustrating because it’s a lot slower than it was 6mo ago, and not noticeably better. Fable seems a step in the right direction but is $$$$

Re: CursorBench 3.1

#47

Post by tekacs »

pbowyer wrote: post #9
tekacs wrote: post #6

I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO. Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to k…

> I'm only using Opus at all because GPT-5.5 in the subscriptions only has a small (400k, but 258k effective) context window. Do you find that makes a difference in your work? I've been using 5.5 high/xhigh to optimize and benchmark a C codebase, and just reading the initial code virtually fills the first context window. A session will auto-compact 5-15 times, but it seems to do okay in spite of that because the task…

> I think for programming the strength of GPT over Opus is winning here over the context window.

On this, absolutely!

I more often use Opus for planning than for implementation. In those cases I really do need the very large context window, because the agent has to read in a bunch of my code base and a bunch of previous plan files and product context and such, to understand what we're talking about.

And then I need to go back and forth with it over a really extended period: getting into a bunch of details, asking it to load how things already work so that we can discuss options for evolution of those, etc.

For that kind of thing, compaction completely destroys its effectiveness because even if you try to serialize out all the decisions made in the conversation into a plan file, the agent still loses e.g. the plan files and code files that it's read in that are adding sharp edges to its understanding of the scope of what's being planned.

For implementation or something like what you're describing in the vein of benchmarking, often I can get away with compaction. Although even then, if the agent needs to have a lot "loaded" into its head, to implement something very, very subtle, complex or far-reaching, in those cases it can be really detrimental if it compacts.

Re: CursorBench 3.1

#48

Post by justachillguy »

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

Naturally, given it’s their benchmark they have overfitted their model somewhat to it.

Re: CursorBench 3.1

#49

Post by apothegm »

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

Composer writes the worst, stupidest, most naive and straight up brains-dead code you could imagine. Fast and cheap is about all it’s got going for it. I mostly use it for “sort these lines alphabetically” and stuff that’s a smidge too complex for regex find/replace.

Re: CursorBench 3.1

#50

Post by vorticalbox »

maxdo wrote: post #42
muzani wrote: post #39

Earlier quoted context omitted.

Anecdotally, I find Composer 2.5 to be useless. I do use light LLMs like Claude Haiku and some of Cursor's older free models, but Composer is negative productivity for me.

The opposite , I use for everything like trigger and monitor a 10 steps release process using composer , a very capable model

this is my finding too, i have moved to it fully for most of the plan/coding.

for most tasks is capable and very cheap, for a days worth of tasks is costing about $10

Post reply on HN