21–30 of 53 posts
Claude's Law: "Token consumption grows faster than the cost per token falls." The Red Queen's Haiku Run faster, she said— each cheaper token consumed to hold the same place Mr. Meeseeks' Law: "An agent that cannot finish a task spawns another agent to help. No task reveals its difficulty until it is attempted; as such, the cost of any unattended task can exceed it's value"
I find it kind of funny that all these companies were token-maxing while the AI companies are giving services at huge discounts costing the AI companies tons of money just so the people can get on leader boards at work. How much has Anthropic and OpenAI spent on just people wanting to get on the leader board at work (or worse, how many trees have been burned down just to get on the leader board at work.)
Isn't inference cheap? Why are AI labs charging so much for it?
If you mean cheaper than training, sure
Google has a lot of systems to make a very large monorepo manageable so builds and code search don't take forever. The build system is Blaze (on which Bazel is based), which has a Pythonic syntax and was once Python but that hasn't been the case (AFAIK) for over a decade. This means you build a massive digraph of build artifacts. By "large" I mean somewhere between 100M and 1B vertices (guessing). Loading that became a significant problem for a build so there's heavy caching around that. There's also heavy caching around build artifacts (ie Forge).
So, part of the issue with every developer using Claude is that you have a ton of inefficiency becasue everybody has a significant context. And what is context really? It's not too dissimilar to the build graph and/or code search you already have.
So the infra I would be working on would be some kind of "global context" or "context cache". Now a lot of context changes when you do a local change but a lot doesn't. As an ordinary engineer, you aren't generally modifying /base. You're modifying leaf nodes or branches for very few leaf nodes.
The reasons I see to do this are:
1. Cost-savings by deduplication;
2. Speed if context is partially-cached;
3. You avoid issues of sending out your codes to third-parties. In the case of Google or Amazon, if they use Claude at all, they would probably only be using their own clouds so they avoid this. But Uber doesn't have that luxury;
4. You avoid any issues of people using your prompts for responses for training and leaking any potential sensitie information that way;
5. You can use off-peak resources for a lot of this work;
6. You can control resources within your own pervasive resource management (in the case of Google); and
7. You can more easily integrate into internal tooling.
I also think that expanding compute power is the biggest risk to Anthropic (and OpenAI). There's a vast difference between a model you need a cluster of NVidia's finest to run vs one you can run on a Macbook Pro. We aren't there yet on a Macbook Pro but it'll only be a few years we are.
It's funny the convos I now have with Sonnet that I wasn't having with Opus. I feel like most of us here are starting to be told to draw down some of our 1M Opus xtrahigh thinking tokens :) Is anyone using a local router to deal with that? Something thats like "don't even bother with sonnet for this task, just go with Opus". I wonder if Haiku could even do that math and recommend the model you should be in?
I’m confused why a business would allow (non-data-science/agent harness devs) to pay per token instead of eg an Anthropic business premium seat? A monthly subscription seems pretty straight forward for the accountants, no?
That no longer exists.. Anthropic business is now seat + usage billed. https://claude.com/pricing#team-&-enterprise