71–79 of 79 posts
An interesting parameter that I don't read about a lot is vocab size. A larger vocab means you will need to generate less tokens for the same word on average, also the context window will be larger. This means that a model with a large vocab might be more expensive on a per token basis, but would generate less tokens for the same sentence, making it cheaper overall. This should be taken into consideration when compar…
An interesting parameter that I don't read about a lot is vocab size. A larger vocab means you will need to generate less tokens for the same word on average, also the context window will be larger. This means that a model with a large vocab might be more expensive on a per token basis, but would generate less tokens for the same sentence, making it cheaper overall. This should be taken into consideration when compar…
All in all this is something I was looking for or was roughly going to do to compare costs. Cool stuff.
Earlier quoted context omitted.
I love the deep dive! On the other hand, this situation is bad and the idea we should ignore it is misguided. Below, we show that given only 1 tokenizer is used, any outputs collapse to a function that is constant, the per token cost. This is why it's shameful: I get that people can't believe its just C100K, but it is, and the writers know that, and they know at that point there is no function, just a constant. > (es…
> Tangential critiques are preferable? Not at all. Reasoned arguments and adding information (the examples I gave were what I thought your main points were) to the discussion are preferable to (what seemed to be) character attacks. Your comment here, as an example, was mostly great. It provides the same level of usefulness to anyone reading it (highlighting that the computation is just C100K and that people will be m…
1. They only support GPT3.5 and GPT4.0. Note here: [1], and that gpt-4o would get swallowed into gpt-4-0613.
2. This will lead to massive, significant, embarrassingly large error in calculations. Tokenizers are not mostly the same, within 10% error.
# Explicating #1, Responsive to ex. "It's not just C100K though. It is for a few models [0]".
The link is to Tiktoken, OpenAI's tokenization library. There are literally more than GPT3.5 and GPT4.0 there, but they're just OpenAI's models, no one else's, none of the others in the long list in their documentation, and certainly not 400.
Most damning? There's only 2 other tokenizers, long deprecated, used only for deprecated models not served anymore, thus you're not calculating costs with them. The only live ones are c100k and o200k. As described above, and shown in [1], their own code kneecaps the o200k and will use c100k
# Explicating #2
Let me know what you'd want to see if you're curious about the 30%+ error thing. I don't want to guess at a test suite that would make you confident you need to revise a prior that there's only +/- 10% difference between arbitrary tokenizers.
For context, I run about 20 unit tests, for each of the big 5 providers, with the same prompts, to capture their input and output token counts to make sure I'm billing accurately.
# Conclusion
Just to save you time, I think the best way I can provide some value is token count experiments demonstrating error. You won't be able to talk me down to "eh, lets just say its +/- 10%, thats good enough for most people!" --- It matters, if it didn't, they'd explicate at least some of this. Instead, its "tokenization for 400 LLMs!"
[1] https://github.com/AgentOps-AI/tokencost/blob/e1d52dbaa3ada2...*
Earlier quoted context omitted.
Yes it does, and no it doesn't. It is exactly as bad of a situation as I laid out. It is a tiktoken wrapper that only does CL100K, doesn't bother with anything beyond that, even the message frame tokens, and claims to calculate cost for 400 LLMs.
> tiktoken.encoding_for_model(model) Calling this where model == 'gpt-4o' will encode with CL200k no? But yes, I do agree with you. I had time implementing non-tiktoken tokenizers for my project. I ended up manually adding tokenizer.json files into my repo.[1] The other options is downloading from HF, but the official repos where the model's tokenizer.json lives require agreeing to their terms to access. So it requir…
No, it will never use O200K, I don't know how to word where its located without sounding aggro, apologies: read below, i.e. the rest of the method.
They copied demo code for Tiktoken with an allowlist without gpt-4o in it, because the demo code is from before 4o.
The demo code has an allowlist, that does string matching, and if its not one of 5 models, none of which are gpt-4o, it says "eh, if it starts gpt-4, just use gpt-4-0613, and make a recursive call"
You can't really blame them, because all they did was copy demo code from OpenAI from before gpt-4o, but I hope you get a giggle out of the extreme clown car this situation is. It's a really bad paper-thin out-of-date tiktoken wrapper that can only do c100k and claims support for 400 LLMs.
Really bonkers.
I know you gotta read the whole method to get it, but, people really shouldn't have just been like "my word! its mean to say they don't get it!" -- it's horrible.
https://github.com/AgentOps-AI/tokencost/blob/e1d52dbaa3ada2...
Earlier quoted context omitted.
In many countries a taxi won't tell you how much the ride will cost. The first time I traveled to somewhere that negotiated the cost up front it blew my mind. Frequently, contracts will have room for additional charges if circumstances change even a little, or products will have a market rate (fish, equity, etc.). It might seem absurd but variable cost things are not uncommon.
In this case there's nothing that's variable, though, and the competition is able to pull it off precisely. Indeed, they themselves were able to do it before!
I am merely hypothesizing, it may not be nondeterministic but I'm not going to assume it's not.
Earlier quoted context omitted.
> Tangential critiques are preferable? Not at all. Reasoned arguments and adding information (the examples I gave were what I thought your main points were) to the discussion are preferable to (what seemed to be) character attacks. Your comment here, as an example, was mostly great. It provides the same level of usefulness to anyone reading it (highlighting that the computation is just C100K and that people will be m…
I really appreciate your engagement here and think it has great value on a personal level, but the length and claims tend to hide two very obvious, straightforward things that are hilariously bad to the point its unbelievable: 1. They only support GPT3.5 and GPT4.0. Note here: [1], and that gpt-4o would get swallowed into gpt-4-0613. 2. This will lead to massive, significant, embarrassingly large error in calculation…
> curious about the 30%+ error thing
I'm mildly curious. I have no doubt that small strings will often have high relative errors. I'd be surprised though if sum(estimated)/sum(actual) were very large if you copied in either a large piece of text or many small pieces of text, outside of specialized domains out of the normal scope of that tokenizer (e.g., throwing latex code into something trained just on wikipedia).
That's more for entropic reasons than anything else. The only way that's true is if (1) some of these tokenizers are much less naive than the normal LLM literature and actually approach entropic bounds, or (2) the baseline implementations are especially bad so that there's a lot of headroom for improvements.
What happens when you throw in something like a medium-sized plain-text wikipedia article (say, the first half as input and the second as output)?
> messaging -- tokenization for 400 LLMs
Alright, I'm sold. I'm still partial to Hanlon's razor for these sort of things, but that ought to be patched.
Earlier quoted context omitted.
In many countries a taxi won't tell you how much the ride will cost. The first time I traveled to somewhere that negotiated the cost up front it blew my mind. Frequently, contracts will have room for additional charges if circumstances change even a little, or products will have a market rate (fish, equity, etc.). It might seem absurd but variable cost things are not uncommon.
> In many countries a taxi won't tell you how much the ride will cost. I've only ever seen: fixed price based on destination (typically for fares originating from an airport), negotiated, or metered. A better analog analogy would be metered pricing, but where the cost per mile is a secret.
Earlier quoted context omitted.
Imagine a coffee shop refusing to have a price list until after the coffee's been made.
In many countries a taxi won't tell you how much the ride will cost. The first time I traveled to somewhere that negotiated the cost up front it blew my mind. Frequently, contracts will have room for additional charges if circumstances change even a little, or products will have a market rate (fish, equity, etc.). It might seem absurd but variable cost things are not uncommon.
Similarly, as LLMs become more and more commonplace, the pricing models will need to be more predictable. My LLM expenses are only around $100/month, but it's a bigger impediment to pushing projects to production when I can't tell the boss exactly how it'll be priced.
With all the options there seems like an opportunity for a single point API that can take a series of prompts, a budget and a quality hint to distribute batches for most bang for buck. Maybe a small triage AI to decide how effectively models handle certain prompts to preserve spending for the difficult tasks. Does anything like this exist yet?
I have yet to find a use case where quality can be traded off. Would love to hear what you had in mind.
Top story picker. Given a bunch of news stories, pick which one should be the lead story.
Data viz color picker. Given a list of categories for a chart, return a color for each one.
Windows Start menu. Given a list of installed programs and a query, select the five most likely programs that the user wants.