Earlier quoted context omitted.
Don't lie. You forked a Chinese model.
They didn't fork it, they post-trained on top of it.
You can read our full technical report here: https://cursor.com/blog/composer-2-technical-report
CursorBench 3.1
91–100 of 107 posts
Re: CursorBench 3.1
#92Earlier quoted context omitted.
Composer writes the worst, stupidest, most naive and straight up brains-dead code you could imagine. Fast and cheap is about all it’s got going for it. I mostly use it for “sort these lines alphabetically” and stuff that’s a smidge too complex for regex find/replace.
It’s starting to feel like people need to say what language/stack and problem space they’re working in. It would be interesting to see why we’re seeing such wild variance.
True - I wrote my own comment here - https://news.ycombinator.com/item?id=48766275
Didn't notice yours until now.
Re: CursorBench 3.1
#93I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO. Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to k…
> it tends to leave big, dangerous holes hiding inside implementations unless babied
it's fascinating that I used these same exact words to express my distaste for Composer and my preference for Opus. I suspect, the domains and problems we are trying to solve need to be shared. I wrote about it here: https://news.ycombinator.com/item?id=48766275
Would love to reach out to discuss more, if you're ok with it, or absolutely feel free to do the same as my email's in the profile like yours!
Re: CursorBench 3.1
#94If I understand the graph correctly;
Fable is using less tokens to achive that same tasks compared to sonet and opus. If so that is a good thing. It feels like we for a while there was spitting out tokens to get a better result. If the model themselves are getting better without generating more tokens that feels like a real win.
Q1: Why is number of steps relevant in this graph? What does it tell us?
Q2: and why have they flipped the horizontal graph so that 0 is to the right and not at origo? Is that some kind of new smart thing? can't say i have seen it before
Re: CursorBench 3.1
#95Earlier quoted context omitted.
I'm not using Cursor at the moment, but when I did (not too long ago) my experience was similar. Plan with Opus, implement with Composer, clean up with Opus. Composer did a competent but not amazing job with a good plan. What I really liked though is it was fast ! Opus could take 30 minutes to do something Composer would get done in 5-10 minutes. Of course the output wasn't perfect, but that's why I'd do a cleanup pa…
Interesting. If I may: What was this "clean up" pass? A code review? A code review with specialized prompt? A focused review to check for edge cases / logic errors / api misuse? Or, something else specific to the codebase? Have you settled on what the clean up pass should look like? Or, do you keep experimenting with it? In case one might not have been aware: Composer 2 was Kimi Base 2.5 post-trained (RL'd) by Cursor…
I end every project with a long interrogation session. Why did you do this? Is there a better approach you didn't consider? Do the naming conventions follow the project idioms? Justify the decision you made here, providing evidence. I disagree with your approach, etc. etc.
Doesn't matter which model wrote the code, they all make mistakes. This is the same stuff I'd do with any junior engineer's PR and it leads to better quality outcomes (something I care about and am finding hard to let go).
Fable is particularly good at having this back and forth I'm discovering.
Re: CursorBench 3.1
#96Very skeptical about the composer accuracy. I have been using it for 6 months now and it is very fast, especially compared to anthropic models, but the result it produces, especially with more difficult tasks is very shallow. It feels like it just finds the cheapest way to deliver the task.
Re: CursorBench 3.1
#97AI is getting expensive.
Re: CursorBench 3.1
#98Earlier quoted context omitted.
Agreed. It’s worse than Opus of course. But Opus takes more than 10x longer to give you something to look at. I’m not kidding, I “benchmarked” a real ticket I was working on. Opus 4.7 took more than 30min. Opus 4.8 took over an hour. Composer 2.5 took 5min on the exact same prompt & local setup. My subjective review is that composer’s code was only like 10-20% worse. It still worked, it was just a bit less clean and…
Opus is so damn slow that it's forced me to work on multiple items at a time, whenever possible. Waiting for one session? Just switch to another!
Yeah, only way to use Opus is with 2-3 different sessions running side by side, or ask it to be the main conductor and delegate tasks to sub agents. Definitely a new way of thinking for me, but I'm liking it to some degree!
Cursor's Composer 2.5 is refreshingly fast by comparison, but I don't trust it as much as Opus. And Composer doesn't have 1 million context...
Re: CursorBench 3.1
#99AI is getting expensive.
I see what you're saying. And it's still crazy valuable for the price.
Compared with how much it can do in such little time, it's still far less than even a junior engineer.