The "multiplier" on Github Copilot went from 3 to 7.5. Nice to see that it is actually only 20-30% and Microsoft wanting to lose money slightly slower. https://docs.github.com/fr/copilot/reference/ai-models/suppo...
oh wow, that is very telling!
Measuring Claude 4.7's tokenizer costs
471–480 of 540 posts
Re: Measuring Claude 4.7's tokenizer costs
#472I did some work yesterday with Opus and found it amazing. Today we are almost on non-speaking terms. I'm asking it to do some simple stuff and he's making incredible stupid mistakes: This is the third time that I have to ask you to remove the issue that was there for more than 20 hours. What is going on here? and at the same time the compacting is firing like crazy. (What adds ~4 minute delays every 1 - 15 minutes) |…
> he’s making .. mistakes Claude and other LLMs do not have a gender; they are not a “he”. Your LLM is a pile of weights, prompts, and a harness; anthropomorphising like this is getting in the way. You’re experiencing what happens when you sample repeatedly from a distribution. Given enough samples the probability of an eventual bad session is 100%. Just clear the context, roll back, and go again. This is part of the…
[dead]
Re: Measuring Claude 4.7's tokenizer costs
#473Earlier quoted context omitted.
Who's going to review that output for accuracy? We'll leave performance and security as unnecessary luxuries in this age and time. In my experience, even Claude 4.6's output can't be trusted blindly it'll write flawed code and would write tests that would be testing that flawed code giving false sense of confidence and accomplishment only to be revealed upon closer inspection later. Additionally - it's age old known…
I am not sure how others are doing this, but here is our process: - meaningful test coverage - internal software architecture was explicitly baked into the prompts, and we try to not go wild with vibing, but, rather, spec it well, and keep Claude on a short leash - each feature built was followed by a round of refactoring (with Claude, but with an oversight of an opinionated human). we spend 50% building, 50% refacto…
The part about refactoring is very interesting and reassuring. I sometimes think I'm holding it wrong when I end up refactoring most of the agent's code towards our "opinionated" style, even after laying it out in md files. Thank you very much for this insight.
Re: Measuring Claude 4.7's tokenizer costs
#474[dead]
Re: Measuring Claude 4.7's tokenizer costs
#475Earlier quoted context omitted.
> It was easy to get comfortable with using the best model at the highest setting for everything for a while, but as the models continue to scale and reasoning token budgets grow, that's no longer a safe default unless you have unlimited budgets. For a while I used Cerebras Code for 50 USD a month with them running a GLM model and giving you millions of tokens per day . It did a lot of heavy lifting in a software mig…
I have this exact same fear as an IC. I wonder if Engineering Managers have this same fear, or they’re used to having to distribute complex tasks to senior engineers and gamble with seeming less risky tasks to juniors that may leave ticking time bombs in their code. Just the nature of code written by agents or humans?
Yes, that is absolutely a dynamic in managing an engineering team, and I'd argue that knowing the right person to give a particular task to, and how much detail they're going to need to get it done, is what separates good engineering managers from bad ones.
Re: Measuring Claude 4.7's tokenizer costs
#476[dead]
Re: Measuring Claude 4.7's tokenizer costs
#477Earlier quoted context omitted.
Who's going to review that output for accuracy? We'll leave performance and security as unnecessary luxuries in this age and time. In my experience, even Claude 4.6's output can't be trusted blindly it'll write flawed code and would write tests that would be testing that flawed code giving false sense of confidence and accomplishment only to be revealed upon closer inspection later. Additionally - it's age old known…
I am not sure how others are doing this, but here is our process: - meaningful test coverage - internal software architecture was explicitly baked into the prompts, and we try to not go wild with vibing, but, rather, spec it well, and keep Claude on a short leash - each feature built was followed by a round of refactoring (with Claude, but with an oversight of an opinionated human). we spend 50% building, 50% refacto…
Very nice insight, that’s where the value is, even with a lot of time refactoring, testing and reviewing the compressed code phase is so much gziped than it’s still worth it to use an imperfect LLM. Even with humans we have all those post phases so great structure around the code generation leads to a lot of gains.
It depends on industries and what’s being developed for sure
Re: Measuring Claude 4.7's tokenizer costs
#478Earlier quoted context omitted.
> It was easy to get comfortable with using the best model at the highest setting for everything for a while, but as the models continue to scale and reasoning token budgets grow, that's no longer a safe default unless you have unlimited budgets. For a while I used Cerebras Code for 50 USD a month with them running a GLM model and giving you millions of tokens per day . It did a lot of heavy lifting in a software mig…
I have this exact same fear as an IC. I wonder if Engineering Managers have this same fear, or they’re used to having to distribute complex tasks to senior engineers and gamble with seeming less risky tasks to juniors that may leave ticking time bombs in their code. Just the nature of code written by agents or humans?
Yes, that definitely happens as an EM. You want your Senior/Staff engineers to architect out the new high-risk functionality into a doc for review. Then that Staff engineer either implements or has a junior/senior under their wing helping implement some of the scaffolding.
In this [common] paradigm the Staff Engineer acts as a architect/programmer and project manager in one. The EM should be there to guide and unblock.
Re: Measuring Claude 4.7's tokenizer costs
#479Earlier quoted context omitted.
Well it's all tradeoffs, right? 6 months for 9 FTEs is 54 man months. 2 months for 2 FTEs is 4 man months. Even if one FTE spent two extra months perusing every line of code and reviewing, that's still 6 man months, resulting in almost 10x speed. Let's say you dont review. Those two extra months probably turns into four extra months of finding bugs and stuff. Still 8 man months vs 54. Of course this is all assuming t…
I do greenfield in fluid dynamics and Claude doesn't help: I need to be able to justify each line of my code (the physics part) and using Claude doesn't help. On the UI side Claude helps a lot. So for me I'd say I have a 25% productivity increment. I work like this: I put the main architecture of the code in place by hand, to get a "feel" for it. Once that is done, I ask Claude to make incremental changes, review the…
My observation is so far, LLMs are not good at scientific computing.
Re: Measuring Claude 4.7's tokenizer costs
#480Earlier quoted context omitted.
This is hard to say definitively. The new Nvidia Vera Rubin chips are 35-50x more efficient on a FLOPS/ megawatt basis. TPU/ ASICS/ AMD chips are making similar less dramatic strides. So a service ran at a loss now could be high margin on new chips in a year. We also don’t really know that they are losing money on the 200/ month subscriptions just that they are compute constrained. If prices increase might be because…
What is your source on 35x more efficient? That seems like a wild performance improvement that I would have hears about. My research shows claims of 10x efficiency, but that number is very questionable.
https://hashrateindex.com/blog/nvidia-vera-rubin-nvl72-specs...
Honestly some of this info is quite hard to parse. I think the efficiency is ~35X on the system level but 10X on the hardware level. I think this is due to Nvidia bringing in Groq in addition to chip improvements.