Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CursorBench 3.1

cursor.com

Post reply on HN
Print view View original

51–60 of 107 posts

Re: CursorBench 3.1

#51

Post by smokel »

budsniffer952 wrote: post #44
tekacs wrote: post #6

I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO. Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to k…

> Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of Is a single thing in your post demonstrable, or are we just supposed to take your word for it? Because all of this stuff sounds laughably subjective.

Most interesting things in software engineering are (laughably) subjective.

Just check out any conversation on dynamic vs static typing, talk to a Rust zealot, or ask a backend engineer if microservices were a mistake.

It's unfortunate, and it makes it hard to have proper discussions on these subjects. It would be worthwhile to figure out how we can have more constructive arguments.

Re: CursorBench 3.1

#52

Post by mrngld »

xyzsparetimexyz wrote: post #37

I wish all these sites would show pareto frontier graphs of cost/performance. That's the main 2 things that matter (I guess you could make it 3D with a speed param as well). https://paraplouis.github.io/llm-pareto-frontier/ is the best of these graphs I've seen but it doesn't update as frequently as I'd like.

That site is useless though because thinking tokens (and caching) and the efficiency thereof aren't accounted for. GLM5.2 is promoted by every 50 Cent Party the PLA can muster on the internet but it falls short because of its extremely verbose thinking. Anthropic models have the same problem but starting from a much higher base of real intelligence.

Which is exactly why every credible comparison now represents cost associated with completing a task, not arbitrary input and output token costs.

Re: CursorBench 3.1

#53

Post by simondotau »

apothegm wrote: post #49
mdasen wrote: post #11

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents . You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8…

Composer writes the worst, stupidest, most naive and straight up brains-dead code you could imagine. Fast and cheap is about all it’s got going for it. I mostly use it for “sort these lines alphabetically” and stuff that’s a smidge too complex for regex find/replace.

I primarily use composer. I wanted to build something from scratch recently and, thinking I was missing out on something, I got Opus to build it. I wasn't blown away. I gave the same prompts to composer and the code it came up with different but similar in quality. I ended up progressing with the composer code because it was easier to progress with improvements due to its faster response time.

Re: CursorBench 3.1

#54

Post by apwheele »

__natty__ wrote: post #17

It's hard to believe Composer 2.5 is that good. I tried to compare it with GLM 5.2 or Opus 4.6 and it lacked thinking about the problem and critical reasoning. It's great for executing plans made by other models, but even then it does some weird code manipulation that is far from how other files around actually work.

I read these and think it is just the jagged edge. I do not doubt your personal experience, I have used Composer 2.5 (via Grok and the credits I get with my X premium account) the past month.

I am not building rockets, but have been quite impressed. All the models do dumb things sometimes, it has done the work I have asked it to pretty well though and has done to me some impressive work.

It is fast on Grok, for other models I have worked extensively with I think it is better than gemini 3.1 (3.5 and antigravity for me is worse than the prior gemini cli). And is comparable to Opus 4.6. (Have not used the more recent models in Claude Code.)

Re: CursorBench 3.1

#55

Post by mrngld »

o10449366 wrote: post #2

I feel like this benchmark reiterates my disbelief that anyone uses the latest Anthropic models for any productive work. They seem to be the best at burning tokens and spawning unnecessary subagents even for well-defined and tightly scoped tasks. Can we get a count of people that have had Claude read irrelevant documents or perform unnecessary web searches even when told not to from the beginning? I'm starting to won…

Now that enterprise customers are pay-as-you-go with tokens I suspect we'll see renewed interest in OpenAI and their focus on token efficiency. At least I hope so if the alternative is abandoning the tools entirely.

Re: CursorBench 3.1

#56

Post by simondotau »

shockembopper wrote: post #13
danfritz wrote: post #5

Earlier quoted context omitted.

It's my daily driver, it's fast affordable and with a bit of guidance gets the job done. I only reach for Claud when i need to plan something big or want to have a sparring partner to fire of some ideas. I think what a lot of people don't realize is that you don't need a fronteer model for 80% of coding tasks. Composer 2.5 is often more than good enough, less token hungry and way faster

I have been doing the same for quite a while now. Composer 2.5 is incredible when you’re working in the loop.

When you normalise for time and money, Composer 2.5 is way, way, way, way better than anything else out there. Yes it requires more babysitting, but that's a good thing.

Re: CursorBench 3.1

#57

Post by abirch »

smokel wrote: post #51
budsniffer952 wrote: post #44

Earlier quoted context omitted.

> Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of Is a single thing in your post demonstrable, or are we just supposed to take your word for it? Because all of this stuff sounds laughably subjective.

Most interesting things in software engineering are (laughably) subjective. Just check out any conversation on dynamic vs static typing, talk to a Rust zealot, or ask a backend engineer if microservices were a mistake. It's unfortunate, and it makes it hard to have proper discussions on these subjects. It would be worthwhile to figure out how we can have more constructive arguments.

"Have you ever noticed that anybody driving slower than you is an idiot, and anyone going faster than you is a maniac?" -- George Carlin

Re: CursorBench 3.1

#59

Post by kandros »

I like Composer a lot as a general-purpose workhorse, but putting it over gpt5.5 medium makes the whole graph lose trust to me, asme witg GLM so low

Re: CursorBench 3.1

#60

Post by tekacs »

budsniffer952 wrote: post #44
tekacs wrote: post #6

I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO. Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to k…

> Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of Is a single thing in your post demonstrable, or are we just supposed to take your word for it? Because all of this stuff sounds laughably subjective.

It's all closed code, so I don't have a great way of showing you, but this is all pretty easy to test for yourself, and a good chunk of it is fairly objective:

On performance: just grab CC + Codex and try Opus 4.8 xhigh and GPT 5.5 xhigh side by side. Ask them a trivial question about something that's already in their context. Opus will churn for 30 seconds, and GPT 5.5 will respond in about three seconds. If you try the same with Fable 5 you'll notice way better adaptive thinking than Opus (it'll quicker than Opus, even on xhigh – although often still slower than 5.5).

I have many, many times done 'Opus xhigh, Opus max and GPT xhigh all tried to implement something' – Opus max is... hours and hours. Opus xhigh is usually ~1.5-2x GPT 5.5 xhigh. This feels like a pretty straightforward generalization of the first point. Again, just try racing three agents and see what you get.

As far as 'right on the edge of what they're able to do', my specific tasks don't matter. Just find something that no matter how hard you try, with however many agents or combinations thereof, with arbitrarily detailed plans, agents can't seem to implement without massively mistakes or a hollowing-out of 'the point' of the implementation... and then try it on the 'following generation' of models. I've been doing this repeatedly with coding agents since I turned aider into a CC-like coding agent in early 2025 (this was my second one, my first modern-style coding agent was in Jan 2025): https://github.com/Aider-AI/aider/pull/3781

A couple of examples of the latter thing that I tend to work on are database internals (indexes, query planner stuff, etc.; I built the DB in full before agents, it just works on it with me), very advanced UIs (try making a beautiful Rolex-like interactive visualization of the internals of a mechanical watch with Opus and see how far it gets – not very), and 'hardcore product questions' (all agents kinda suck at schema – Fable far less than prior ones). I have dozens and dozens of these that they can't do, though.

Post reply on HN