Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 Price Increase: What It Costs

openrouter.ai

Post reply on HN
Print view View original

51–60 of 80 posts

Re: GPT-5.5 Price Increase: What It Costs

#51

Post by languid-photic »

thepasch wrote: post #45
languid-photic wrote: post #42

Earlier quoted context omitted.

I agree! So far we have been native harnessmaxxing, which simplifies things a lot. The configuration space around open models is much larger. Eg which models, capability heterogeneity, which harness, networking, data egress / privacy, etc. If anyone is getting very good production code out of open models, I'd love to do a user interview to better understand your setup. Email is in my bio.

With how much vendor harnesses are now actively steering the agent with their own instructions on top of user prompts, I think it’d be super interesting to see a comparison of one of the already tested models - so Opus 4.7 or GPT-5.5 - across a range of different harnesses that aren’t their native. OpenCode, Pi, Hermes, Kilo Code. The most popular coding-focused harnesses, basically.

Agreed. Harness is really important. Especially since many labs are now post-training agents directly in their native harness.

(Which is why my prior is that third party harnesses would not perform as well. But I haven't actually measured this.)

Re: GPT-5.5 Price Increase: What It Costs

#52

Post by nothinkjustai »

olao99 wrote: post #11
iceKirin wrote: post #7

I feel that the recent iterations of LLM haven't provided an intuitive qualitative leap. Have they entered a bottleneck period so quickly?

For what is worth I find GPT 5.5 qualitatively different than 5.4 and 5.3 If I had to collapse the nature of the difference in one sentence it'd be that the 5.5 does more what I'm asking it to do versus doing a small aspect of what I'm asking then stopping. 5.4 required a lot of "continue" encouragement. 5.5 just "gets it" a bit more What is boils down to for me is that even though it's more expensive I would much ra…

They probably just tell it to do more in the prompt lmao

Re: GPT-5.5 Price Increase: What It Costs

#53

Post by languid-photic »

digdugdirk wrote: post #43
languid-photic wrote: post #34

Earlier quoted context omitted.

Yes! It depends on the extent of changes needed. If the changes needed are small, I'll apply the best implementation as a foundation and then just iterate directly. If the changes needed are drastic, it usually signals that there was sth wrong/ambiguous/etc in the spec (or the ensemble was too weak, which is rarely the case). In cases like this, I improve the spec and then rerun. If it's in the middle, I'll usually a…

How does that get integrated into the scoring system? I'm imagining a scenario where a cheaper model may get close, but only needs a small follow up to get the desired result. How would this score in comparison to a larger model that got it right the first time - even if it may have been much more expensive overall?

We also use a secondary signal from blinded multi-verifier reviews. Each verifier ranks the candidates, and those verification outcomes serves as an additional quality signal. It's somewhat similar to consensus labeling.

Btw, this also helps manage scale. Eg you have 15 diffs to review. Run a few verifiers to get a short list, then review directly and apply the best.

Re: GPT-5.5 Price Increase: What It Costs

#55

Post by throwaway219450 »

auspiv wrote: post #32
iceKirin wrote: post #7

I feel that the recent iterations of LLM haven't provided an intuitive qualitative leap. Have they entered a bottleneck period so quickly?

GPT-5.5 is a solid leap with Codex or other harnesses. Opus 4.7 I still don't understand how people use... I tried it for a day or two, have tried it for a few hours every week or so since release, and still use 4.6 as daily driver (with xhi thinking).

As with these daily opinion threads, ymmv. I find GPT's code to be competent, but its voice isn't great. If Claude can be a little too cool, GPT-5.x often reads like 90s era movie hacker technobabble. This has got to be RLHF/alignment and the sort of tone that people like. Also anecdotally I used xhigh for a while and turned it down to medium because it would take so long to do even simple jobs. The instruction following is quite good with 5.5 so there isn't too much need to let it wander off.

Re: GPT-5.5 Price Increase: What It Costs

#56

Post by motbus3 »

languid-photic wrote: post #19

We track performance vs. the all-in cost of completing real engineering tasks, rather than cost per token. [1] Cost per token is a bit misleading because, as others have noted, different models use tokens in different ways. (Aside - This is also why TPS isn't a great metric). We found that 5.5 is about 1.5-2x more expensive overall. On a "Pareto" basis, we only find 5.5 xhigh worth it. At the lower reasoning levels,…

But what situation seems to good to enable xhigh?

Re: GPT-5.5 Price Increase: What It Costs

#57

Post by boh »

New model releases are now like new iPhones--mostly imperceivable improvements with a higher price tag. That's one of the major benefits to open source: you can "freeze" what model you're using. Often it's the model that you know that wins over the one that is different enough that you have to start from scratch with every major update. Most businesses require cost control and predictability over a cutting edge with limited evidence of profitable output outside of tech.

Re: GPT-5.5 Price Increase: What It Costs

#58

Post by jere »

NitpickLawyer wrote: post #41
jere wrote: post #25

Earlier quoted context omitted.

You mean the company that just doubled their rate limits? https://www.anthropic.com/news/higher-limits-spacex

They only did that after they "found" ~300k H100 equivalent compute. Before signing that deal they were severely compute constrained. Especially visible when EU tz was still active and US east would wake up.

I don't disagree. It's just a weird way to describe them currently when they just announced massively increasing limits.

Re: GPT-5.5 Price Increase: What It Costs

#59

Post by BugsJustFindMe »

languid-photic wrote: post #50
BugsJustFindMe wrote: post #44

Earlier quoted context omitted.

It feels pretty weird that your ratings have: gpt-5-4-high > gpt-5-4-xhigh gpt-5-4-high > gpt-5-5-high gpt-5-4 > gpt-5-5 gpt-5-2-high > gpt-5-2-xhigh No other ratings I've seen show that.

Yes, the signal we are measuring is quite different from most evals. We are measuring sth much closer to: when multiple agents compete on the same spec, which one produces the patch that holds up best in code review? Most evals are static / synthetic, and for code, generally stop at tests. Test evals are weak proxies for quality since it's difficult to encode qualities like scope creep/churn, codebase fit, maintainab…

Ok, but my point is that the claims you make about more reasoning performing worse seems kinda suspicious and I haven't seen any analysis exploring why that would happen.

Re: GPT-5.5 Price Increase: What It Costs

#60

Post by cyberpunk »

thepasch wrote: post #45
languid-photic wrote: post #42

Earlier quoted context omitted.

I agree! So far we have been native harnessmaxxing, which simplifies things a lot. The configuration space around open models is much larger. Eg which models, capability heterogeneity, which harness, networking, data egress / privacy, etc. If anyone is getting very good production code out of open models, I'd love to do a user interview to better understand your setup. Email is in my bio.

With how much vendor harnesses are now actively steering the agent with their own instructions on top of user prompts, I think it’d be super interesting to see a comparison of one of the already tested models - so Opus 4.7 or GPT-5.5 - across a range of different harnesses that aren’t their native. OpenCode, Pi, Hermes, Kilo Code. The most popular coding-focused harnesses, basically.

OpenCode seems to give me better results than codex-cli, i’d be interested in seeing this too!
Post reply on HN