Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

tokens.billchambers.me

Post reply on HN
Print view View original

551–560 of 620 posts

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#551

Post by wallst07 »

mrandish wrote: post #455
codethief wrote: post #430

Earlier quoted context omitted.

> > Why are you handwaving things away though? I've got you on max effort. I even patched the system prompts to reduce this. In my experience, prompts like this one, which 1) ask for a reason behind an answer (when the model won't actually be able to provide one), 2) are somewhat standoff-ish, don't work well at all. You'll just have the model go the other way. What works much better is to tell the model to take a st…

> when the model won't actually be able to provide one This is key. In my experience, asking an LLM why it did something is usually pointless. In a subsequent round, it generally can't meaningfully introspect on its prior internal state, so it's just referring to the session transcript and extrapolating a plausible sounding answer based on its training data of how LLMs typically work. That doesn't necessarily mean th…

It's worked for me when I ask why with a stated goal of preventing the same error the next time.

"Why did you guess at the functions signature and get it wrong, what information were you using and how can we prevent it next time."

Is that not the right approach?

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#552

Post by wallst07 »

holoduke wrote: post #395
glerk wrote: post #148

I'd be ok with paying more if results were good, but it seems like Anthropic is going for the Tinder/casino intermittent reinforcement strategy: optimized to keep you spending tokens instead of achieving results. And yes, Claude models are generally more fun to use than GPT/Codex. They have a personality. They have an intuition for design/aesthetics. Vibe-coding with them feels like playing a video game. But the resu…

You have to guide an ai. Not let roam freely. If you got skills to guide you can make it output high quality

That is 100% correct as a foundation.

However when you guide the AI as a constant, and the model behaves MUCH differently (given a baseline guide), that is where the problem lies.

It's as if your 'guidance' has to be variable on how well the model is behaving. Analogy is a junior dev who is sometimes excellent, and sometimes shows up drunk for work and you have no breathalyzer.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#554

Post by ozgrakkurt »

The design of this thing is atrocious. There should be a clear way to see what the +X% thing means. Is 4.7 using more or is 4.6 using more.

Also there should be time distribution for the queries and a way to filter by query time. This is because Anthropic is reported to change the model quality arbitrarily in the background.

Also there is no unit in table column headers. For example "Request 4.7" is this the amount of tokens 4.7 consumes? Is it output/input/reasoning etc.

Really difficult to make sense of this.

People get offended if what they are doing is labeled as slop but this is unfortunately the level of quality I expect from AI related content or code.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#555

Post by ozgrakkurt »

xpe wrote: post #537
ozgrakkurt wrote: post #522

Earlier quoted context omitted.

Can you explain this literature review process? I don't believe you can do a same quality job with an LLM in 5 minutes.

I don't mean writing a literature review. I mean reviewing the literature to find what I need. My point is that this was not practical with "fancy search" three years ago by which I mean Google-like search engines. My example: I wanted to get a sense for the feasibility of doing a project that blends Gaussian Processes, active learning, and pairwise comparisons. So I want to dig into the literature to find out what i…

It is a better investment to read about those things for a bit in my experience. It should not be scary or niche to take some time and read a textbook or a high quality paper.

There is no replacement for reading textbooks or high quality papers.

If you are saying that you didn't do this kind of thing anyway and now you can do it. Then I would question the definition of the action you are doing because it is not the same in my opinion.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#556

Post by muyuu »

hereme888 wrote: post #255

> Opus 4.7 (Adaptive Reasoning, Max Effort) cost ~$4,406 to run the Artificial Analysis Intelligence Index, ~11% less than Opus 4.6 (Adaptive Reasoning, Max Effort, ~$4,970) despite scoring 4 points higher. This is driven by lower output token usage, even after accounting for Opus 4.7's new tokenizer. This metric does not account for cached input token discounts, which we will be incorporating into our cost calculati…

uhmmm so that is the p they are hacking? it would actually explain a lot

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#557

Post by sillyfluke »

vanviegen wrote: post #524
mrandish wrote: post #455

Earlier quoted context omitted.

> when the model won't actually be able to provide one This is key. In my experience, asking an LLM why it did something is usually pointless. In a subsequent round, it generally can't meaningfully introspect on its prior internal state, so it's just referring to the session transcript and extrapolating a plausible sounding answer based on its training data of how LLMs typically work. That doesn't necessarily mean th…

> In a subsequent round, it generally can't meaningfully introspect on its prior internal state It can't do any better in the moment it's making the choices. Introspection mostly amounts to back-rationalisation, just like in humans. Though for humans, doing so may help learning to make better future decisions in similar situations.

I don't understand why people don't just say "This is wrong. try again." or "This is wrong because xyz. try again." This anthropologizing by asking why seems a bit pointless when you know how LLMs work, unless you've empirically had better results from a specific make and version of LLM by asking why in the past. It's theoretically functionally equivalent to asking a brand new LLM instance with your chat history why the original gave such an answer...Do you want the correct result or do you actually care about knowing why?

>Introspection mostly amounts to back-rationalisation, just like in humans.

That's the best case scenario. Again, let's stop anthropologizing. The given reasons why may be incompatible with the original answer upon closer inspection...

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#558

Post by jbrooks84 »

couchdb_ouchdb wrote: post #175

Comments here overall do not reflect my experience -- i'm puzzled how the vast majority are using this technology day to day. 4.7 is absolute fire and an upgrade on 4.6.

Amen, yes it uses more tokens and thinks longer but it's amazing

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#559

Post by irthomasthomas »

andai wrote: post #239

For a fair comparison you need to look at the total cost, because 4.7 produces significantly fewer output tokens than 4.6, and seems to cost significantly less on the reasoning side as well. Here is a comparison for 4.5, 4.6 and 4.7 (Output Tokens section): https://artificialanalysis.ai/?models=claude-opus-4-7%2Cclau... 4.7 comes out slightly cheaper than 4.6. But 4.5 is about half the cost: https://artificialanalysi…

The link you are commenting on shows data from actual prompts from real users, and the COST of the average prompt increased 37%. I do not think synthetic benchmarks are a rebuttal to real usage data.
Post reply on HN