Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring Claude 4.7's tokenizer costs

claudecodecamp.com

Post reply on HN
Print view View original

521–530 of 540 posts

Re: Measuring Claude 4.7's tokenizer costs

#521

Post by solenoid0937 »

smt88 wrote: post #520
solenoid0937 wrote: post #497

Earlier quoted context omitted.

There was no nerf - this meme needs to die.

What exactly happened then? How did we all have this collective hallucination?

Collective hallucinations are common. Mandela effect, people thinking FB is listening to your microphone because they see relevant ads, etc

This is a common phenomenon that all humans pattern match to things we expect. When we learn a new vocabulary word you see it everywhere for the next two days. When we think Claude might be nerfed, we overindex on every instance of Claude underperforming.

The only way to account for this is credulous, hard data. Like benchmarks over time. To this day no one has provided evidence that Claude Code, when fixed to the same thinking level, has had degraded performance.

Re: Measuring Claude 4.7's tokenizer costs

#522

Post by al_borland »

solenoid0937 wrote: post #521
smt88 wrote: post #520

Earlier quoted context omitted.

What exactly happened then? How did we all have this collective hallucination?

Collective hallucinations are common. Mandela effect, people thinking FB is listening to your microphone because they see relevant ads, etc This is a common phenomenon that all humans pattern match to things we expect. When we learn a new vocabulary word you see it everywhere for the next two days. When we think Claude might be nerfed, we overindex on every instance of Claude underperforming. The only way to account…

Are there any good ways to benchmark models over time that don't fall victim to Goodhart's law? It seems that once the benchmark is defined, the AI will train on it, and it will become effectively meaningless.

I read many articles about AIs doing extremely well on various tests in graduate or PhD level programs. But these tests are well defined. A professor put the same models though his freshman CS class and most of them failed.

Re: Measuring Claude 4.7's tokenizer costs

#523

Post by snoman »

altern8 wrote: post #211
9dev wrote: post #145

Earlier quoted context omitted.

But that objective measure is exactly what we’re lacking in programming: There is often many ways to skin a cat, but the model only takes one. Without knowing about those it didn’t take, how do you judge the quality of a new model?

I would say following instructions. If Claude understood what you mean better without you having to over explain it would be an improvement

I agree with you, but my gut tells me that a lot of people don’t know what a good outcome should/could look like and are accepting whatever it delivers.

Re: Measuring Claude 4.7's tokenizer costs

#524

Post by nwienert »

motbus3 wrote: post #234

I've been using 4.6 models since each of them launched. Same for 4.5. 4.6 performers worse or the same in most of the tasks I have. If there is a parameter that made me use 4.6 more frequently is because 4.5 get dumber and not because 4.6 seemed smarter.

Agree on all counts, 4.5 was a monster, 4.6 a clear regression, and then 4.5 was dumbed down so I moved on.

Re: Measuring Claude 4.7's tokenizer costs

#525

Post by sho »

rohansood15 wrote: post #449
sho wrote: post #448

Taking the article's 5% accuracy improvement at face value: if true, then it's more than worth the token inflation IMO. That's because of tool call chains, where errors compound and accumulate, and small improvements in accuracy get greatly magnified. Again, the article's numbers are likely a rather crude approximation, but taking 85% accuracy (claude 4.6) vs 90% (4.7) as inputs: 4.6 1 iteration 85% 4.7 1 iteration 9…

You're assuming errors cannot be retried/recovered. They can.

You're assuming errors are a clear failure that can be identified and retried, rather than a silent drift from user intent that simply feeds bad but well-formed results into the next step. They're not. Well, of course sometimes they are, but the much more insidious failure mode is doing the wrong thing in the right way.

Re: Measuring Claude 4.7's tokenizer costs

#527

Post by hnarturpl »

fuzzy2 wrote: post #444
thiht wrote: post #434

Earlier quoted context omitted.

> IC becomes an EM What?

IC = Individual Contributor EM = Engineering Manager The idea being that you no longer contribute yourself, you manage agents doing that.

That’s exactly what I’ve become. A monkey typing prompts and pressing Enter to confirm plans and actions.

Obviously I am exaggerating but my days shifted from figuring out issues and coming up with solutions to explaining the issue to Claude and supervising the work.

What worries me is two things:

1. Current models were mostly trained on human work. Do we have enough training materials created now for the models to progress or they’ll be training or other models output? That cannot end well. 2. I’ve started as a Junior Engineer and had opportunity to learn and become Senior. The job market for junior is really bad cause businesses plan just for couple quarters ahead. They replaced juniors with AI. Who’s gonna replace seniors? And don’t say AI ;)

Re: Measuring Claude 4.7's tokenizer costs

#528

Post by solenoid0937 »

al_borland wrote: post #522
solenoid0937 wrote: post #521

Earlier quoted context omitted.

Collective hallucinations are common. Mandela effect, people thinking FB is listening to your microphone because they see relevant ads, etc This is a common phenomenon that all humans pattern match to things we expect. When we learn a new vocabulary word you see it everywhere for the next two days. When we think Claude might be nerfed, we overindex on every instance of Claude underperforming. The only way to account…

Are there any good ways to benchmark models over time that don't fall victim to Goodhart's law? It seems that once the benchmark is defined, the AI will train on it, and it will become effectively meaningless. I read many articles about AIs doing extremely well on various tests in graduate or PhD level programs. But these tests are well defined. A professor put the same models though his freshman CS class and most of…

These models don't learn continuously, they are a static snapshot one training is finished. You only need a new benchmark once new models are published (or you need a private benchmark, in which case you don't need to update the benchmark at all)

Re: Measuring Claude 4.7's tokenizer costs

#529

Post by snoman »

tetha wrote: post #134
uberman wrote: post #2

On actual code, I see what you see a 30% increase in tokens which is in-line with what they claim as well. I personally don't tend to feed technical documentation or random pros into llms. Given that Opus 4.6 and even Sonnet 4.6 are still valid options, for me the question is not "Does 4.7 cost more than claimed?" but "What capabilities does 4.7 give me that 4.6 did not?" Yesterday 4.6 was a great option and it is to…

Yeah that was an interesting discovery in a development meeting. Many people were chasing after the next best model and everything, though for me, Sonnet 4.6 solves many topics in 1-2 rounds. I mainly need some focus on context, instructions and keeping tasks well-bounded. Keeping the task narrow also simplifies review and staying in control, since I usually get smaller diffs back I can understand quickly and manage…

I wonder if there are different use cases. You sound like you’re using an LLM in a similar way to me. I think about the problem and solution, describe what I need implemented, provide references in the context (“the endpoint should be structured like this one…”) and then evaluate the output.

It sounds like other folks are more throwing an LLM at the problem to see what it comes up with. More akin to how I delegate a problem to one of my human engineers/architects. I understand, conceptually, why they might be doing that but I know that I stopped trying that because it didn’t produce quality. I wonder if the newer models are better at handling that ambiguity better.

Re: Measuring Claude 4.7's tokenizer costs

#530

Post by fragmede »

mh- wrote: post #510
jessermejia wrote: post #456

Earlier quoted context omitted.

There's an interesting analysis here: https://github.com/anthropics/claude-code/issues/42796 >The most striking row is user prompts: 5,608 in February vs 5,701 in March. The human put in the same effort. But the model consumed 80x more API requests and 64x more output tokens to produce demonstrably worse results.

Sorry, "this" referred to the parent comment's claim. > models starting becoming "moody" due to their proprietors arbitrarily modifying their performance capabilities The tokenizer changes are measurable, the above is quite difficult to quantify. There are a few sites floating around that purport to, but all of them have fatal flaws in their methodology.

Unfortunately, LLM performance isn't an exact science and some observations are going to be subjective. Observations like ChatGPT being "lazy" in the Winter. Wanting to form opinions based on hard data, aka science, and not vibes is entirely reasonable but doesn't make the vibes a figment of imagination. Or as Jeff Bezos put it, "When the data and the anecdotes disagree, the anecdotes are usually right." And while he's not a scientist, his success does put some weight behind that quote. (as does digging deeper in what he meant by that.)
Post reply on HN