Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

tokens.billchambers.me

Post reply on HN
Print view View original

481–490 of 620 posts

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#481

Post by tclancy »

noodletheworld wrote: post #479
codethief wrote: post #430

Earlier quoted context omitted.

> > Why are you handwaving things away though? I've got you on max effort. I even patched the system prompts to reduce this. In my experience, prompts like this one, which 1) ask for a reason behind an answer (when the model won't actually be able to provide one), 2) are somewhat standoff-ish, don't work well at all. You'll just have the model go the other way. What works much better is to tell the model to take a st…

> What works much better is to tell the model to take a step back and re-evaluate. I desperately hate that modern tooling relies on “did you perform the correct prayer to the Omnissiah” > to add some entropy to get it away from the local optimum Is that what it does? I don't think thats what it does, technically. I think thats just anthropomorphizing a system that behaves in a non deterministic way. A more menaingful…

Other than AI (and possibly npm packaging) where do you feel you have to rely on prayer? Additionally, most of human history has been the story of scientific advancement to a different point where people rely on prayer, so maybe suck it up buttercup is the best advice here? &emdash;

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#482

Post by squeaky-clean »

manmal wrote: post #191
httgbgg wrote: post #173

Earlier quoted context omitted.

Only the first one. Ideally now there is no second prompt.

Are you aware that every tool call produces output which also counts as input to the LLM?

Are you aware that a lot of model tool calls are useless and a smarter model could avoid those?

Are you aware that output tokens are priced 5x higher than input tokens?

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#483

Post by iammrpayments »

sync wrote: post #81
tiffanyh wrote: post #52

I was using Opus 4.7 just yesterday to help implement best practices on a single page website. After just ~4 prompts I blew past my daily limit. Another ~7 more prompts & I blew past my weekly limit. The entire HTMl/CSS/JS was less than 300 lines of code. I was shocked how fast it exhausted my usage limits.

Which plan are you on? I could see that happening with Pro (which I think defaults to Sonnet?), would be surprised with Max…

Opus is not available for claude code in pro

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#484

Post by krackers »

stingraycharles wrote: post #462
krackers wrote: post #407

Earlier quoted context omitted.

>pay for reinitializing the cache Why can't they save the kv cache to disk then later reload it to memory?

It’s a shitload of data, and it only works if all the tokens are 100% identical, i.e. all the attention values are exactly the same. Typically it’s cached for about 5 minutes, you can pay extra for longer caches.

If I have a conversation with claude then come back 30 minutes later to resume the conversation, the KV values for that prefill prefix are going to be exactly the same. That's the whole point of this caching in the first place.

If you're willing to incur a latency penalty on a "cold resume" (which is fine for most use-cases), why couldn't they just move it to disk. The size of the KV cache should scale on the order of something like (context_length * n_layers * residual_length). I think for a standard V3-MoE model at 1M token length, this should be on the order of 100G at FP16? And you can surely play tricks with KV compression (e.g. the recent TurboQuant paper). It doesn't seem like an outrageous amount of data to put onto cheap scratch HDD (and it doesn't grow indefinitely since really old conversations can be discarded).

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#485

Post by ipaddr »

colechristensen wrote: post #275
aerhardt wrote: post #261

Earlier quoted context omitted.

I don't think it can write something that's of a fraction of the quality of Kafka. But GP's argument ("limit the space to text") could be taken to imply - and it seems to be a common implication these days - that LLMs have mastered the text medium, or that they will very soon. > it can't write a foundational piece of literature Why not, if this a pure textual medium, the corpus includes all the great stories ever wri…

I don't know what to tell you. It's more than a little absurd to make the qualification of being able to do something to be that the output has to be considered a great work of art for generations.

If the end result is most books will be written by AI you need the possiblity of that qualification. If its only capable of certain types of book then we will need endless amounts of that.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#486

Post by squeaky-clean »

ACCount37 wrote: post #355
bigyabai wrote: post #317

Earlier quoted context omitted.

I can ask a nine-year-old human brain to solve that problem with a box of Crayola and a sheet of A4 printer paper. In-context learning is professedly not "good enough" to approximate continuous learning of even a child.

You're absolutely wrong! You can also ask an LLM to solve that problem by spelling the word out first. And then it'll count the letters successfully. At a similar success rate to actual nine-year-olds. There's a technical explanation for why that works, but to you, it might as well be black magic. And if you could get a modern agentic LLM that somehow still fails that test? Chances are, it would solve it with no inst…

This is false. You can ask it to spell out strawberry and count the letters and it will still say 2 (it's unable to actually count the letters by the way). The only way to get a model that believes strawberry has 2 R's to consistently give the correct answer is to ask it to code the problem and return the output.

In fact, asking a model not to repeat the same mistake makes it more likely to commit that mistake again, because it's in it's context.

I think anyone who uses LLMs a lot will tell your that your steps 3 and 4 are fictional.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#487

Post by sky2224 »

subarctic wrote: post #353
sky2224 wrote: post #326

Earlier quoted context omitted.

This is part of the reason why I'm really worried that this is all going to result in a greater economic collapse than I think people are realizing. I think companies that are shelling out the money for these enterprise accounts could honestly just buy some H100 GPUs and host the models themselves on premises. Github CoPilot enterprise charges $40 per user per month (this can vary depending on your plan of course), b…

Out of curiosity, how many concurrent users could you get with a hosting setup at that price? If let's say 10% of those 1000 users were using it at the same time would it handle it? What about 30% or 100%?

You made a good point that I didn't think through fully. It's the concurrent user aspect that heavily impacts things. Currently, you'd probably need quite a bit more investment to the point of having a mini data center to do what I'm proposing.

However, we've been seeing advancements in compressing context and capabilities of smaller models that I don't think it'd be too far off to see something like what I'm talking about within the next 5 years.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#488

Post by squeaky-clean »

8note wrote: post #433
bigyabai wrote: post #317

Earlier quoted context omitted.

I can ask a nine-year-old human brain to solve that problem with a box of Crayola and a sheet of A4 printer paper. In-context learning is professedly not "good enough" to approximate continuous learning of even a child.

why is the breakdown from words to letters your highest priority thing to add to the training data? what problem does this allow you to solve that you couldnt otherwise?

This comment is tangential to their point that a transformer architecture can or cannot be functionally equivalent to a human brain. Practicality of those limitations is a different discussion

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#489

Post by [deleted] »

andersa wrote: post #413
bashtoni wrote: post #412

Earlier quoted context omitted.

Only if you set `ENABLE_PROMPT_CACHING_1H`, which was mentioned in the release notes for a recent Claude Code release but doesn't seem to be in the official docs.

Bruh. It's getting hard to track down all these MAKE_IT_ACTUALLY_WORK settings that default to off for no reason.

[deleted]

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#490

Post by xvector »

matheusmoreira wrote: post #389
andai wrote: post #239

For a fair comparison you need to look at the total cost, because 4.7 produces significantly fewer output tokens than 4.6, and seems to cost significantly less on the reasoning side as well. Here is a comparison for 4.5, 4.6 and 4.7 (Output Tokens section): https://artificialanalysis.ai/?models=claude-opus-4-7%2Cclau... 4.7 comes out slightly cheaper than 4.6. But 4.5 is about half the cost: https://artificialanalysi…

It thinks less and produces less output tokens because it has forced adaptive thinking that even API users can't disable. Same adaptive thinking that was causing quality issues in Opus 4.6 not even two weeks ago. The one bcherny recommended that people disable because it'd sometimes allocate zero thinking tokens to the model. https://news.ycombinator.com/item?id=47668520 People are already complaining about low quali…

Adaptive thinking is optional
Post reply on HN