Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

cognition.com

Post reply on HN
Print view View original

111–120 of 151 posts

Re: SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

#111

Post by oofbey »

pants2 wrote: post #19

Kinda funny that their "cost-vs-performance" chart looks the same as the one for Composer 2.5[1], except that it includes Composer 2.5 at a completely different spot. What are the chances that CursorBench ranks Cursor's model highest, and Cognition's bench ranks Cognition's model highest? Both are to be RL'd from Kimi as a base model, BTW. I'd posit that it's not deliberate deception, but for both companies their tra…

Agreed on the likely mechanism. I'm not sure "overfitting" is even the right description. These things are of course absurdly complicated, and evaluating their quality down to a single number involves a lot of judgement and trade-offs. I think it's more "you get what you measure" which is true in human organizations too. Define a KPI and people work hard to make it go up, even if it's not quite right or has bad side-effects.

Re: SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

#112

Post by RussianCow »

modeless wrote: post #100

What is the actual per token price? The benchmarks look similar to Grok 4.5 also released today and priced at $2/M input tokens and $6/M output tokens.

The regular one (not the fast variant) is free but slow. The "Lightning" variant (which uses Cerebras and gets supposedly 1000 TPS) costs $12.50/M output, $2.5/M input, $1/M cached input. So it's quite a bit more expensive than SWE 1.6.

Re: SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

#113

Post by mrinterweb »

akshaydeshraj wrote: post #58

Would have been worth a consideration if it could have been used beyond it's own harness. Unfortunately, doesn't seem to be the case. https://x.com/theodormarcu/status/2074896486047834380

I really don't want harness lock-in. I am trying to decouple myself from Claude Code now. I love the model of OpenRouter and being able to switch models at will let's your harness focus on your personal tooling and you can easily switch to the flavor of the month LLM with a single slash command instead of rewiring your entire workflow to use a harness to use a model.

I like Cerabras, but I really wish they would make more of their hosted models generally available.

Re: SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

#114

Post by cogman10 »

petesergeant wrote: post #87
kgeist wrote: post #75

On artificialanalysis.ai, Kimi 2.7 Code is way worse than GLM 5.2 at everything (general intelligence, coding, agentic tasks). But here, both Kimi 2.7 and its derivative SWE-1.7 are ahead of GLM 5.2. This tells me the benchmarks they use are cherry-picked.

> This tells me the benchmarks they use are cherry-picked. Which benchmarks would you have chosen instead, and why?

It honestly seems like there's not a great way to currently benchmark AI.

The ideal way to run these benchmarks would be to give a 3rd party the model to run in an isolated environment so the prompts don't make their way back to the AI engineers.

That seems doable for open weight models, but not for private models.

Re: SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

#115

Post by qingcharles »

kgeist wrote: post #75

On artificialanalysis.ai, Kimi 2.7 Code is way worse than GLM 5.2 at everything (general intelligence, coding, agentic tasks). But here, both Kimi 2.7 and its derivative SWE-1.7 are ahead of GLM 5.2. This tells me the benchmarks they use are cherry-picked.

Composer 2.5 is worse than both; I use it all day for simple stuff. It's Kimi 2.6 in a new outfit.

Re: SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

#116

Post by blauditore »

znsnjanwnwwn wrote: post #81
ryandvm wrote: post #26

Okay, let's give software engineers a break for a bit and focus on obsoleting other high-linguistic context occupations.

And to do that you’ll need development so until we’re all out of a job they’ll keep pushing. Once automating is automated it’s done.

Any day now... Just a bit more space in the context window, trust me bro.

Re: SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

#117

Post by RussianCow »

lnenad wrote: post #104
unshavedyak wrote: post #82

Earlier quoted context omitted.

It's a lossy conversion though. "Mistake" is relative to the stated goals and specifications which are often heavily lacking. So unless you write with a high degree of architectural and implementation specificity then it might make very high quality code that is still not what you wanted.

You can ask for a complete feature/app/business. Or you can split up the work into verifiable/testable pieces and rely on a high quality AI to deliver. As time goes by the pieces will get larger as capability grows. I still trust myself and my experience when arch is involved, but AI has been great at tackling lower level stuff. And with Fable I don't really care it takes a while for it to complete, as I know I can t…

But some requirements you don't realize you have until you start building. With a fast model, you can surface those really quickly and have more time to iterate and explore different solutions. With a slower but smarter model, you just hope that what it produces after an hour is what you were imagining.

And yes, with Fable, the chance of that is higher than with SWE/Composer, but in my experience it's not so much higher that the extra time and cost is worth it. But it certainly depends on your goals and what you're building.

Re: SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

#118

Post by RussianCow »

2001zhaozhao wrote: post #86
akshaydeshraj wrote: post #58

Would have been worth a consideration if it could have been used beyond it's own harness. Unfortunately, doesn't seem to be the case. https://x.com/theodormarcu/status/2074896486047834380

Harness-wrapper tools that support multiple harnesses and allow sharing workspace features (skills, slash commands, etc.) between them will be meta.

Ironically, Devin Desktop is one of those tools. It supports any harness that supports ACP (which is most of them)—you can use Claude Code, Codex, OpenCode, etc from the Devin Desktop UI.

I'm currently experimenting with OpenSpec[0] as the "framework" and using different subscriptions for different parts of the spec-driven process: Opus via Claude Code for exploration, Devin SWE for building, and GLM 5.2 via the Z.ai Coding Plan for verification. I don't love having to mix and match harnesses, but in practice it's barely more effort than switching models.

[0]: https://openspec.dev/

Re: SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

#119

Post by jstummbillig »

andriy_koval wrote: post #88
jstummbillig wrote: post #59

Earlier quoted context omitted.

Defining what "coding" means now, and how quickly we fall off the capability cliff seems increasingly important. Today my "coding" sessions often enough begin with real life problems, where I discuss domain or inter-domain things, ranging from business, economics, psychology, etc. Being able to do all of that with one model is something I am willing to pay a premium for. Of course not having to pay the premium, becau…

> Today my "coding" sessions often enough begin with real life problems intuition is that your sessions consists of 10% of domain related reasoning, and 90% of code plumbing. Those 90% could be moved to cheap and efficient specialized and focused model.

Possibly! It's just hard to reason about from the outside. When does the model benefit from all the ambient knowledge? Idk.

Regardless, it's fairly obvious to me that none of what I do now will require "frontier models" for much longer. Models are getting better more quickly than my problems are getting harder.

Re: SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence

#120

Post by meowface »

oofbey wrote: post #111
pants2 wrote: post #19

Kinda funny that their "cost-vs-performance" chart looks the same as the one for Composer 2.5[1], except that it includes Composer 2.5 at a completely different spot. What are the chances that CursorBench ranks Cursor's model highest, and Cognition's bench ranks Cognition's model highest? Both are to be RL'd from Kimi as a base model, BTW. I'd posit that it's not deliberate deception, but for both companies their tra…

Agreed on the likely mechanism. I'm not sure "overfitting" is even the right description. These things are of course absurdly complicated, and evaluating their quality down to a single number involves a lot of judgement and trade-offs. I think it's more "you get what you measure" which is true in human organizations too. Define a KPI and people work hard to make it go up, even if it's not quite right or has bad side-…

At this point I barely put any value in any of the benchmarks. I just use the models for coding (and related things like software product design/planning/ideation/etc.) tasks and judge them subjectively, and also see how others judge them subjectively on HN and Twitter.
Post reply on HN