Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

tokens.billchambers.me

Post reply on HN
Print view View original

561–570 of 620 posts

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#561

Post by Dumbledumb »

sillyfluke wrote: post #557
vanviegen wrote: post #524

Earlier quoted context omitted.

> In a subsequent round, it generally can't meaningfully introspect on its prior internal state It can't do any better in the moment it's making the choices. Introspection mostly amounts to back-rationalisation, just like in humans. Though for humans, doing so may help learning to make better future decisions in similar situations.

I don't understand why people don't just say "This is wrong. try again." or "This is wrong because xyz. try again." This anthropologizing by asking why seems a bit pointless when you know how LLMs work, unless you've empirically had better results from a specific make and version of LLM by asking why in the past. It's theoretically functionally equivalent to asking a brand new LLM instance with your chat history why…

I definitely do this, along with the compulsion sometimes to tell the agent how a problem was fixed in the end, when investigating myself after the model failing to do so. Just common courtesy after working on something together. Let’s rationalize this as giving me an opportunity to reflect and rubberduck the solution.

Regarding not just telling „try again“: of course you are right to suggest that applying human cognition mechanisms to llm is not founded on the same underlying effects.

But due to the nature of training and finetuning/rf I don’t think it is unreasonable that instructing to do backwards reflection could have a positive effect. The model might pattern match this with and then exhibit a few positive behaviors. It could lead it doing more reflection within the reasoning blocks and catch errors before answering, which is what you want. These will have attention to the question of „what caused you to make this assumption“, also, encouraging this behavior. Yes, both mechanisms are exhibited through linear forward going statical interpolation, but the concept of reasoning has proven that this is an effective strategy to arrive at a more grounded result than answering right away.

Lastly, back to anthro. it shows that you, the user, is encouraging of deeper thought an self corrections. The model does not have psychological safety mechanisms which it guards, but again, the way the models are trained causes them to emulate them. The RF primes the model for certain behavior, I.e. arriving at answer at somepoint, rather than thinking for a long time. I think it fair to assume that by „setting the stage“ it is possible to influence what parts of the RL activate. While role-based prompting is not that important anymore, I think the system prompts of the big coding agents still have it, suggesting some, if slight advantage, of putting the model in the right frame of mind. Again, very sorry for that last part, but anthro. does seem to be a useful analogy for a lot of concepts we are seeing (the reason for this being in the more far of epistemological and philosophical regions, both on the side of the models and us)

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#562

Post by d0gsg0w00f »

xstas1 wrote: post #444
rectang wrote: post #400

Earlier quoted context omitted.

https://github.blog/changelog/2026-04-16-claude-opus-4-7-is-... > Over the coming weeks, Opus 4.7 will replace Opus 4.5 and Opus 4.6 in the model picker for Copilot Pro+[...] > This model is launching with a 7.5× premium request multiplier as part of promotional pricing until April 30th.

Promotional pricing? Are they saying that after the promotion, it will cost more than 7.5x??

Hmm, maybe they're discouraging copilot+Claude through pricing, nudging people to anthropic suite of tools. That sucks. I've been super happy with copilot+opus/sonnet.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#563

Post by KronisLV »

mrandish wrote: post #455
codethief wrote: post #430

Earlier quoted context omitted.

> > Why are you handwaving things away though? I've got you on max effort. I even patched the system prompts to reduce this. In my experience, prompts like this one, which 1) ask for a reason behind an answer (when the model won't actually be able to provide one), 2) are somewhat standoff-ish, don't work well at all. You'll just have the model go the other way. What works much better is to tell the model to take a st…

> when the model won't actually be able to provide one This is key. In my experience, asking an LLM why it did something is usually pointless. In a subsequent round, it generally can't meaningfully introspect on its prior internal state, so it's just referring to the session transcript and extrapolating a plausible sounding answer based on its training data of how LLMs typically work. That doesn't necessarily mean th…

> This is key. In my experience, asking an LLM why it did something is usually pointless. In a subsequent round, it generally can't meaningfully introspect on its prior internal state, so it's just referring to the session transcript and extrapolating a plausible sounding answer based on its training data of how LLMs typically work.

Yep, I've gotten used to treating the model output as a finished, self-contained thing.

If it needs to be explained, the model will be good at that, if it has an issue, the model will be good at fixing it (and possibly patching any instructions to prevent it in the future). I'm not getting out the actual reason why things happened a certain way, but then again, it's just a token prediction machine and if there's something wrong with my prompt that's not immediately obvious and perhaps doesn't matter that much, I can just run a few sub-agents in a review role and also look for a consensus on any problems that might be found, for the model to then fix.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#564

Post by girvo »

alex_sf wrote: post #467
danny_codes wrote: post #251

Earlier quoted context omitted.

I just don’t see how they’ll be able to make a profit. Open models have the same performance on coding tasks now. The incentives are all wrong. Why pay more for a model that’s no better and also isn’t open? It’s nonsense

Open models, in actual practice, don't match up to even one or two generation prior models from Anthropic/OpenAI/Google. They've clearly been trained on the benchmarks. Entirely possible it was by mistake, but it's definitely happening.

GLM 5.1 is absolutely on par with Sonnet 4.5, sometimes better in practice (it holds abstractions over longer context windows better)

It’s about the only one that is at that level though to be fair. They’re all still useful, still!

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#565

Post by andai »

irthomasthomas wrote: post #559
andai wrote: post #239

For a fair comparison you need to look at the total cost, because 4.7 produces significantly fewer output tokens than 4.6, and seems to cost significantly less on the reasoning side as well. Here is a comparison for 4.5, 4.6 and 4.7 (Output Tokens section): https://artificialanalysis.ai/?models=claude-opus-4-7%2Cclau... 4.7 comes out slightly cheaper than 4.6. But 4.5 is about half the cost: https://artificialanalysi…

The link you are commenting on shows data from actual prompts from real users, and the COST of the average prompt increased 37%. I do not think synthetic benchmarks are a rebuttal to real usage data.

The cost of the input tokens, not the reasoning or output.

Agree though that benchmarks aren't very helpful w.r.t. estimating real world performance or costs.

What we'd need are people giving the same real world tasks to 4.6 and 4.7 and measuring time, quality and costs.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#566

Post by andai »

scrollop wrote: post #519
andai wrote: post #239

For a fair comparison you need to look at the total cost, because 4.7 produces significantly fewer output tokens than 4.6, and seems to cost significantly less on the reasoning side as well. Here is a comparison for 4.5, 4.6 and 4.7 (Output Tokens section): https://artificialanalysis.ai/?models=claude-opus-4-7%2Cclau... 4.7 comes out slightly cheaper than 4.6. But 4.5 is about half the cost: https://artificialanalysi…

That is not what anthropic says- "Opus 4.7 thinks more at higher effort levels, particularly on later turns in agentic settings. This improves its reliability on hard problems, but it does mean it produces more output tokens. " https://www.anthropic.com/news/claude-opus-4-7

That's a good point. AA's Cost Efficiency section says the opposite: you can hover to see the breakdown between input, reasoning and output tokens.

I'm not sure where that discrepancy comes from (is Anthropic using different benchmarks?).

There's a few different theories but all we have now are synthetic benchmarks, anecdotes and speculation.

(Benchmarks are misleading, I think our best bet now is for individuals to run real world tests, giving the same task to each model, and compare the quality, cost and time.)

The input cost inflation however is real, and dramatic.

I would have expected them to lower input costs proportionally, because otherwise you're getting less intelligence per dollar even with the smarter model. Think that would be the smartest thing for them to do, at least PR wise. And maybe a bit of free usage as an apology :)

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#567

Post by retsibsi »

3dfd wrote: post #426
braebo wrote: post #403

Earlier quoted context omitted.

Which open model has the same performance as Opus 4.7?

They dont have to be parity today. If the frontier models reach a point of barely any noticeable improvements the trade off changes. You do not need a perfect substitute if you are getting it for free... People will factor in future expectations about the development of open source vs frontier models. Why do you think OAI and anthropic are pushing hard on marketing? its for this reason. They want to get contractual c…

The person they were responding to said "Open models have the same performance on coding tasks now." AFAIK this is bullshit, but I'd love to be corrected if I'm wrong.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#568

Post by andai »

zozbot234 wrote: post #297
andai wrote: post #109

Earlier quoted context omitted.

Yeah I've been arriving at the same thing. The other models give me way more usage but they don't seem to have enough common sense to be worth using as the main driver. If I can have Claude write up the plan, and the other models actually execute it, I'd get the best of both worlds. (Amusingly, I think Codex tolerates being invoked by Claude (de facto tolerated ToS violation), but not the other way around.)

I don't think there's any ToS violation involved? AIUI you can use GPT models with any harness, at least at present. You could nonetheless have Codex write up the plan to an .md file for Claude (perhaps Sonnet or even Haiku?) to execute.

I meant with the subscriptions, which are heavily subsidized. API billing is free game.

Anthropic's been against 3rd party usage of the subs, including, I believe, 3rd party software invoking Claude Code.

OpenAI, from what I heard, is the only company tolerating this type of usage (for now).

So my point was, if you get a subscription from both, you could let Claude "drive" Codex, but not the reverse.

And while it sounds silly, it might actually be the only way to get real work done at the moment, with usage being exhausted so quickly.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#569

Post by Terretta »

mrandish wrote: post #455
codethief wrote: post #430

Earlier quoted context omitted.

> > Why are you handwaving things away though? I've got you on max effort. I even patched the system prompts to reduce this. In my experience, prompts like this one, which 1) ask for a reason behind an answer (when the model won't actually be able to provide one), 2) are somewhat standoff-ish, don't work well at all. You'll just have the model go the other way. What works much better is to tell the model to take a st…

> when the model won't actually be able to provide one This is key. In my experience, asking an LLM why it did something is usually pointless. In a subsequent round, it generally can't meaningfully introspect on its prior internal state, so it's just referring to the session transcript and extrapolating a plausible sounding answer based on its training data of how LLMs typically work. That doesn't necessarily mean th…

How I think about this is…

From early GPT days to now, best way to get a decently scoped and reasonably grounded response has always been to ask at least twice (early days often 7 or 8 times).

Because not only can it not reflect, it cannot "think ahead about what it needs to say and change its mind". It "thinks" out loud (as some people seem to as well).

It is a "continuation" of context. When you ask what it did, it still doesn't think, it just* continues from a place of having more context to continue from.

The game has always been: stuff context better => continue better.

Humans were bad at doing this. For example, asking it for synthesis with explanation instead of, say, asking for explanation, then synthesis.

You can get today's behaviors by treating "adaptive thinking" like a token budgeted loop for context stuffing, so eventually there's enough context in view to produce a hopefully better contextualized continuation from.

It seems no accident we've hit on the word "harness" — so much that seems impressive by end of 2025 was available by end of 2023 if "holding it right". If (and only if!) you are an expert in an area you need it to process: (1) turn thinking off, (2) do your own prompting to "prefill context", and (3) you will get superior final response. Not vibing, just staff-work.

---

* “just” – I don't mean "just" dismissively. Qwen 3.5 and Gemma 4 on M5 approaches where SOTA was a year ago, but faster and on your lap. These things are stunning, and the continuations are extraordinary. But still: Garbage in, garbage out; gems in, gem out.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#570

Post by Forgeties79 »

mrandish wrote: post #455
codethief wrote: post #430

Earlier quoted context omitted.

> > Why are you handwaving things away though? I've got you on max effort. I even patched the system prompts to reduce this. In my experience, prompts like this one, which 1) ask for a reason behind an answer (when the model won't actually be able to provide one), 2) are somewhat standoff-ish, don't work well at all. You'll just have the model go the other way. What works much better is to tell the model to take a st…

> when the model won't actually be able to provide one This is key. In my experience, asking an LLM why it did something is usually pointless. In a subsequent round, it generally can't meaningfully introspect on its prior internal state, so it's just referring to the session transcript and extrapolating a plausible sounding answer based on its training data of how LLMs typically work. That doesn't necessarily mean th…

> This is key. In my experience, asking an LLM why it did something is usually pointless.

That kind of strikes me as a huge problem. Working backwards from solutions (both correct and wrong) can yield pretty critical information and learning opportunities. Otherwise you’re just veering into “guess and check” territory.

Post reply on HN