[dead]
GPT-5.5 Price Increase: What It Costs
31–40 of 80 posts
Re: GPT-5.5 Price Increase: What It Costs
#32I feel that the recent iterations of LLM haven't provided an intuitive qualitative leap. Have they entered a bottleneck period so quickly?
GPT-5.5 is a solid leap with Codex or other harnesses. Opus 4.7 I still don't understand how people use... I tried it for a day or two, have tried it for a few hours every week or so since release, and still use 4.6 as daily driver (with xhi thinking).
Re: GPT-5.5 Price Increase: What It Costs
#33We track performance vs. the all-in cost of completing real engineering tasks, rather than cost per token. [1] Cost per token is a bit misleading because, as others have noted, different models use tokens in different ways. (Aside - This is also why TPS isn't a great metric). We found that 5.5 is about 1.5-2x more expensive overall. On a "Pareto" basis, we only find 5.5 xhigh worth it. At the lower reasoning levels,…
Interesting! I've been thinking about how to create a similar type of evaluation system for myself. How do you handle tweaks to agentic tasks? Say that a model gets pretty close to what you want, so you just need a quick follow up prompt to the original response?
Re: GPT-5.5 Price Increase: What It Costs
#34We track performance vs. the all-in cost of completing real engineering tasks, rather than cost per token. [1] Cost per token is a bit misleading because, as others have noted, different models use tokens in different ways. (Aside - This is also why TPS isn't a great metric). We found that 5.5 is about 1.5-2x more expensive overall. On a "Pareto" basis, we only find 5.5 xhigh worth it. At the lower reasoning levels,…
Interesting! I've been thinking about how to create a similar type of evaluation system for myself. How do you handle tweaks to agentic tasks? Say that a model gets pretty close to what you want, so you just need a quick follow up prompt to the original response?
Yes! It depends on the extent of changes needed.
If the changes needed are small, I'll apply the best implementation as a foundation and then just iterate directly.
If the changes needed are drastic, it usually signals that there was sth wrong/ambiguous/etc in the spec (or the ensemble was too weak, which is rarely the case). In cases like this, I improve the spec and then rerun.
If it's in the middle, I'll usually apply the best and write a follow on spec.
Re: GPT-5.5 Price Increase: What It Costs
#35I feel that the recent iterations of LLM haven't provided an intuitive qualitative leap. Have they entered a bottleneck period so quickly?
Call me cynical but for me these are mostly pricing changes, the change in quality is imperceptible. I believe after a few iterations we will be closer to the real cost.
Re: GPT-5.5 Price Increase: What It Costs
#36This doesn't seem to be controlling for the number of turns in any way. Am I missing something? Stronger models needing fewer turns to achieve a task feels like a prime source of efficiency gains for agentic coding, more so than individual responses being shorter.
OpenRouter may see you fire hundreds of requests at them, but they have no idea that "these 50 requests here at 4PM are for task A", "those 100 requests there does task B", etc. So it's a shallow analysis at the "overall request shape" level.
Re: GPT-5.5 Price Increase: What It Costs
#37I feel that the recent iterations of LLM haven't provided an intuitive qualitative leap. Have they entered a bottleneck period so quickly?
I do a lot of OCaml and I found 5.5 to be much better, but that's kind of an esoteric language thing
Re: GPT-5.5 Price Increase: What It Costs
#38I feel that the recent iterations of LLM haven't provided an intuitive qualitative leap. Have they entered a bottleneck period so quickly?
[deleted]
Re: GPT-5.5 Price Increase: What It Costs
#39We track performance vs. the all-in cost of completing real engineering tasks, rather than cost per token. [1] Cost per token is a bit misleading because, as others have noted, different models use tokens in different ways. (Aside - This is also why TPS isn't a great metric). We found that 5.5 is about 1.5-2x more expensive overall. On a "Pareto" basis, we only find 5.5 xhigh worth it. At the lower reasoning levels,…
would be interesting to see some other labs:
- deepseek v4 pro
- glm 5.1
- kimi k2.6
- qwen 3.6 max
- xiaomi 2.5 pro
- minimax 2.7
- grok
Re: GPT-5.5 Price Increase: What It Costs
#40Has any enterprising hacker here yet graphed price vs "output" over time since 2023, taking "quality" into account? That's got to be a very tricky analysis given how subjective quality is. But I'm sure there are people trying to pin it down.
artificial analysis has an intelligence benchmark