Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring Claude 4.7's tokenizer costs

claudecodecamp.com

Post reply on HN
Print view View original

481–490 of 540 posts

Re: Measuring Claude 4.7's tokenizer costs

#481

Post by siffin »

thrwaway55 wrote: post #404
fragmede wrote: post #200

Earlier quoted context omitted.

That post doesn't address the human factor of cost, and I don't mean that in a good way. Even if AI costs more than a human, it's tireless, doesn't need holidays, is never going to have to go to HR for sexual harassment issues, won't show up hungover or need an advance to pay for a dying relative's surgery. It can be turned on and off with the flip of a switch. Hire 30 today, fire 25 of them next week. Spin another 5…

Why do you think it can't sexually harass someone or drive people to suicide. There are already lawsuits coming in on it causing suicides. This is an architecture that people are increasing begging to give network connectivity that can't differentiate its system prompt from user input

I thought it already was used to sexually harass people by creating naked versions of them.

Re: Measuring Claude 4.7's tokenizer costs

#482

Post by losvedir »

ericol wrote: post #266

I did some work yesterday with Opus and found it amazing. Today we are almost on non-speaking terms. I'm asking it to do some simple stuff and he's making incredible stupid mistakes: This is the third time that I have to ask you to remove the issue that was there for more than 20 hours. What is going on here? and at the same time the compacting is firing like crazy. (What adds ~4 minute delays every 1 - 15 minutes) |…

> This is the third time that I have to ask you to remove the issue that was there for more than 20 hours. What is going on here?

I don't know if you're giving this as something you've actually given Claude, but I don't think it's a good way of using Claude.

It's not a collaborator who's having a bad day where a little empathy might make him feel better and realize his error. It's a token generator based on a prompt which includes all chat history. If you have three examples of the bad approach in the history, in a format that looks like Claude doing work, it will totally pollute it! And even worse with auto-compaction where you don't know exactly what of those false starts is getting summarized into its context.

You have to treat this like a tool and understand how it works.

If Claude is going down a wrong path it's better to cancel and rewind and improve the previous addition to the prompt. You don't want it to generate a bunch of misleading tokens for itself and leave it in the context window indefinitely!

Re: Measuring Claude 4.7's tokenizer costs

#483

Post by adi_kurian »

skissane wrote: post #412
noosphr wrote: post #409

Earlier quoted context omitted.

Anything but the simplest tooling is not transferable between model generations, let alone completely different families.

> Anything but the simplest tooling is not transferable between model generations, let alone completely different families. It is transferable-yes, you will get issues if you take prompts and workflows tuned for one model and send them to another unchanged. But, most of the time, fixing it is just tinkering with some prompt templates People port solutions between models all the time. It takes some work, but the amoun…

Yep. My approach has been, if I can’t reliably get something to 90+% with a flash / nano / haiku, then it’s not viable for any accuracy critical work. (I don’t know of or have the luck of having any other work.) Starting out with the pro / opus for any production classification work has always been a trick.

Re: Measuring Claude 4.7's tokenizer costs

#484

Post by uberman »

jstummbillig wrote: post #467
uberman wrote: post #2

On actual code, I see what you see a 30% increase in tokens which is in-line with what they claim as well. I personally don't tend to feed technical documentation or random pros into llms. Given that Opus 4.6 and even Sonnet 4.6 are still valid options, for me the question is not "Does 4.7 cost more than claimed?" but "What capabilities does 4.7 give me that 4.6 did not?" Yesterday 4.6 was a great option and it is to…

I don't understand how people measure how much more or less work they need to do. It's not that gpt-4o was incapable of exuding enormous amounts of code quickly, it's that the tokens were relativ garbage. How do you have an opinion on 4.6/4.7 here? It's less clear but I could totally see that 4.7 or beyond leads to project completion 20% faster, by removing dead ends, foot guns, less backtracking, etc. How to tell /…

My personal opinion here based on observations not empirical tested. 4.5 could generate code, but I often ran out of context and the results were regularly incomplete. The result was that I had to spend as much time proofing and debugging as I did making direct progress.

4.6 has what in practice seems to an almost unlimited context window and rarely produces incomplete or flat out wrong results. That is a big step forward though i do burn through quota much faster.

I have not formed an option yet how what 4.7 does for me other than to say I have observed my quota being consumed faster. To be fair, I have not put 4.7 to a challenging task yet.

It honestly surprises me that someone who regularly uses Claude would not have an opion about 4.6 or even Opus vs Sonnet at this point. The lift at least for me was obvious.

Re: Measuring Claude 4.7's tokenizer costs

#485

Post by naasking »

_pdp_ wrote: post #29

IMHO there is a point where incremental model quality will hit diminishing returns. It is like comparing an 8K display to a 16K display because at normal viewing distance, the difference is imperceptible, but 16K comes at significant premium. The same applies to intelligence. Sure, some users might register a meaningful bump, but if 99% can't tell the difference in their day-to-day work, does it matter? A 20-30% cost…

Diminishing returns are inevitable, agreed, but it's not clear we're near that point yet.

Re: Measuring Claude 4.7's tokenizer costs

#486

Post by tclancy »

michaelbuckbee wrote: post #278
fragmede wrote: post #200

Earlier quoted context omitted.

That post doesn't address the human factor of cost, and I don't mean that in a good way. Even if AI costs more than a human, it's tireless, doesn't need holidays, is never going to have to go to HR for sexual harassment issues, won't show up hungover or need an advance to pay for a dying relative's surgery. It can be turned on and off with the flip of a switch. Hire 30 today, fire 25 of them next week. Spin another 5…

More importantly it collapses mythical-man-month communication overhead.

Hang on, tell me how, because I am not picking up what you are putting down. At a minimum, wouldn’t this require working from a perfectly written spec that has already accounted for the discovery of changes that would need to be made from the original perfect spec?

Re: Measuring Claude 4.7's tokenizer costs

#487

Post by aenis »

AnonymousPlanet wrote: post #473
aenis wrote: post #459

Earlier quoted context omitted.

I am not sure how others are doing this, but here is our process: - meaningful test coverage - internal software architecture was explicitly baked into the prompts, and we try to not go wild with vibing, but, rather, spec it well, and keep Claude on a short leash - each feature built was followed by a round of refactoring (with Claude, but with an oversight of an opinionated human). we spend 50% building, 50% refacto…

The part about refactoring is very interesting and reassuring. I sometimes think I'm holding it wrong when I end up refactoring most of the agent's code towards our "opinionated" style, even after laying it out in md files. Thank you very much for this insight.

Thanks! In our limited experience, Claude does not focus that much on guardrails and code quality when building a feature - but can be pretty focused on code quality and architecture when asked to do just that. So, one a few hours to iterate a feature, a few hours to refactor. Rinse and repeat.

Re: Measuring Claude 4.7's tokenizer costs

#488

Post by Majromax »

yorwba wrote: post #172
Majromax wrote: post #13

Earlier quoted context omitted.

While that's a nice effort, the inter-run variability is too high to diagnose anything short of catastrophic model degradation. The typical 95% confidence interval runs from 35% to 65% pass rates, a full factor of two performance difference. Moreover, on the companion codex graphs ( https://marginlab.ai/trackers/codex-historical-performance/ ), you can see a few different GPT model releases marked yet none correspond…

Yes, MarginLab only tests 50 tasks a day, which is too few to give a narrower confidence interval. On the other hand, this really calls into question claims of performance degradation that are based on less intensive use than that. Variance is just so high that long streaks of bad luck are to be expected and plausibly the main source of such complaints. Similarly, it's unlikely you can measure a significant performan…

> Similarly, it's unlikely you can measure a significant performance difference between models like GPT 5.4-xhigh and GPT 5.2 unless you have a task where one of them almost always fails or one almost always succeeds

That feels like a concession to the limited benchmarking framework. 5.4-xhigh is supposed to be (and is widely believe to be) a better model than 5.2, so if that's invisible in the benchmarking scores then the protocol has problems. The test probably should include cases that should be 'easy passes' or 'near always failures', and then paired testing could offer greater precision on improvements or degradations.

Conversely, if model providers also don't do this then they could be accidentally 'benchmaxxing' if they use protocols like this to set dynamic quantization levels for inference. All you really need for a credible observation of problems from 'less intensive use' is a problem domain that isn't well-covered by the measured and monitored benchmark.

Re: Measuring Claude 4.7's tokenizer costs

#489

Post by j_bum »

losvedir wrote: post #482
ericol wrote: post #266

I did some work yesterday with Opus and found it amazing. Today we are almost on non-speaking terms. I'm asking it to do some simple stuff and he's making incredible stupid mistakes: This is the third time that I have to ask you to remove the issue that was there for more than 20 hours. What is going on here? and at the same time the compacting is firing like crazy. (What adds ~4 minute delays every 1 - 15 minutes) |…

> This is the third time that I have to ask you to remove the issue that was there for more than 20 hours. What is going on here? I don't know if you're giving this as something you've actually given Claude, but I don't think it's a good way of using Claude. It's not a collaborator who's having a bad day where a little empathy might make him feel better and realize his error. It's a token generator based on a prompt…

Yep I bewilders me when I see instructions like this. Go bad and edit your previous message if you didn’t et what you want!

I think this is a direct result of OpenAI and Anthropic humanizing these models too much.

I want C-3PO by my side helping me work, not a machine acting emotional.

But that’s what they’ve given us, and now a huge fraction of the username treats these tools like a human.

Post reply on HN