Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CursorBench 3.1

cursor.com

Post reply on HN
Print view View original

61–70 of 107 posts

Re: CursorBench 3.1

#61

Post by urbsgpw »

vorticalbox wrote: post #50
maxdo wrote: post #42

Earlier quoted context omitted.

The opposite , I use for everything like trigger and monitor a 10 steps release process using composer , a very capable model

this is my finding too, i have moved to it fully for most of the plan/coding. for most tasks is capable and very cheap, for a days worth of tasks is costing about $10

Same here, maybe I'm underusing it a bit, because for anything that is a bit more complex i tend to err on the safe side and go with anthropic, but i wonder if thats just a placebo effect because i pay more for it.

I do feel that they've really upped their game with composer this year though.

Re: CursorBench 3.1

#62

Post by tekacs »

smokel wrote: post #51
budsniffer952 wrote: post #44

Earlier quoted context omitted.

> Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of Is a single thing in your post demonstrable, or are we just supposed to take your word for it? Because all of this stuff sounds laughably subjective.

Most interesting things in software engineering are (laughably) subjective. Just check out any conversation on dynamic vs static typing, talk to a Rust zealot, or ask a backend engineer if microservices were a mistake. It's unfortunate, and it makes it hard to have proper discussions on these subjects. It would be worthwhile to figure out how we can have more constructive arguments.

Thanks very much for saying this!

Frankly, it feels like we should just sidestep arguments entirely and just all contribute our messy data/reports, and then see how we can meld all of it together, to find the best answers for our individual situations.

Probably a good use of frontier AI, melding all of that!

Re: CursorBench 3.1

#63

Post by mwigdahl »

cbg0 wrote: post #45
o10449366 wrote: post #2

I feel like this benchmark reiterates my disbelief that anyone uses the latest Anthropic models for any productive work. They seem to be the best at burning tokens and spawning unnecessary subagents even for well-defined and tightly scoped tasks. Can we get a count of people that have had Claude read irrelevant documents or perform unnecessary web searches even when told not to from the beginning? I'm starting to won…

I've had no problems like the ones you've mentioned while using Opus 4.8. It does overthink stuff with higher effort levels but that's kind of expected.

Same (including the overthinking issue).

Re: CursorBench 3.1

#64

Post by andai »

Interesting that Opus 4.7 does better than 4.8. Too bad they didn't test 4.6, too. I witnessed a man here mocked yesterday for insisting it was better than its successors!

Although, the benchies are always tricksy ... On DeepSWE, GPT-5.5 beats Opus-4.8, by a fair margin, but on FrontierCode, the situation is the other way around.

The only benchmark you can trust is your actual workload!

Re: CursorBench 3.1

#65

Post by andai »

tekacs wrote: post #6

I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO. Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to k…

>it tends to leave big, dangerous holes hiding inside implementations unless babied.

A brainwave: perhaps GLM or DeepSeek could be integrated into the mix for the purposes of red-teaming the code. Fable has been blinded to security by design[0], and the open models are pretty decent at it.

[0] It's not clear what the situation with GPT-5.6 will be but the blog suggests similarly over-cautious safety filters.

Amusingly the posts for recent Opus releases brag that they successfully made it worse at security! "during its [Opus 4.7] training we experimented with efforts to differentially reduce these ["cyber"] capabilities"

Re: CursorBench 3.1

#66

Post by daft_pink »

tekacs wrote: post #6

I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO. Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to k…

I agree why they reverse the x axis makes this graph very hard to understand for the casual observer.

Re: CursorBench 3.1

#67

Post by CuriouslyC »

mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

Not hard to understand what's going on here. They RL'd around patterns in their data and specific capabilities, so of course they'd construct a benchmark that's aligned with the training set.

Ironically, their benchmark might be more accurate than artificial analysis for a narrow slice of things that Cursor's Eigencustomer is really interested in. Otherwise I'd take it as just another data point.

Re: CursorBench 3.1

#68

Post by tekacs »

andai wrote: post #65
tekacs wrote: post #6

I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO. Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to k…

>it tends to leave big, dangerous holes hiding inside implementations unless babied. A brainwave: perhaps GLM or DeepSeek could be integrated into the mix for the purposes of red-teaming the code. Fable has been blinded to security by design[0], and the open models are pretty decent at it. [0] It's not clear what the situation with GPT-5.6 will be but the blog suggests similarly over-cautious safety filters. Amusingl…

I definitely use GPT-5.5 as a counterpart to validate these exact sorts of things in Anthropic models' implementations, in the (now-rarer) cases where I allow Anthropic's models _to_ implement.

And yeah, it's a bit depressing to think that 5.6 might be similarly nerfed. Less secure software for us all, I guess... except BigCorps. :(

Re: CursorBench 3.1

#70

Post by rc1 »

tekacs wrote: post #6

I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO. Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to k…

> I'm pretty baffled by their choice of axes

To put their own model out in front?

Post reply on HN