[flagged]
You don't like a repo, you don't use it. Stop shaming people for their open source repos.
Show HN: Token price calculator for 400+ LLMs
31–40 of 79 posts
Re: Show HN: Token price calculator for 400+ LLMs
#32Can we get conversions for kg of CO2 emitted, too?
Re: Show HN: Token price calculator for 400+ LLMs
#33Re: Show HN: Token price calculator for 400+ LLMs
#34[flagged]
This is unnecessarily harsh. Not every model has a publicly available tokenizer, and using a fallback like cl100k is usually a decent enough estimator from my experience.
Besides, there's a warning message for when you specify a model without a known tokenizer.
If you're upset with the implementation, you can always raise an issue or fix it yourself
Re: Show HN: Token price calculator for 400+ LLMs
#35Can we get conversions for kg of CO2 emitted, too?
It would be nice, but how do we get this information required for the conversion?
Re: Show HN: Token price calculator for 400+ LLMs
#36wow, this is really useful!! Just the price list alone has given me a lot of inspiration, thank you
Re: Show HN: Token price calculator for 400+ LLMs
#37[flagged]
This is unnecessarily harsh. Not every model has a publicly available tokenizer, and using a fallback like cl100k is usually a decent enough estimator from my experience. Besides, there's a warning message for when you specify a model without a known tokenizer. If you're upset with the implementation, you can always raise an issue or fix it yourself
> This is unnecessarily harsh.
Which part? All I can tease out from your comment are "the lies are impossible" (agreed!) and "close enough afaik". (it's not, the closest in the Big 5 has percent error of 32%, see end of comment. ex. GPT4o has a tokenizer with 2x the vocab so you'd expect ~1/2 the tokens)
> Not every model has a publicly available tokenizer,
Right. Ex. Claude 3s and Geminis. So why are Claude 3s and Geminis listed as supported models?
> using a fallback
CL100K isn't a fallback, its the only tokenizer.
> like cl100k is usually a decent enough estimator from my experience.
I'm very surprised to hear this, per stats demonstrating minimum error of 32%.
> If you're upset with the implementation, you can always raise an issue
I'm not "upset with the implementation", I'm sharing that the claims about being able to make financial calculations for 400 different LLMs is lying.
> or fix it yourself
How?
As you pointed out, its unfixable for at least some subset of the ones they're claiming, ex. Gemini and Claude 3s.
Let's pretend it was possible.
Why?
If someone puts out a library making wildly false claims, is th right thing to do to stay quiet and fix the library making false claims until its claims are true?
> usually a decent enough estimator
No, not for financial things certainly, which is the stated core purpose of the library.
As promised, data: I picked the simplest example from my unit tests because you won't believe the divergence on larger ones.
OpenAI (CL100K) - 18 in/1 out = 19.
Gemini 1.5 - 41 in/14 out = 55. (65% error)
Claude 3 - 21 in/4 out = 25. (24% error)
Llama 3 - 23 in/5 out = 28. (32% error)
Mistral - 10 in/3 out = 13. (46% error)
Re: Show HN: Token price calculator for 400+ LLMs
#38Earlier quoted context omitted.
You don't like a repo, you don't use it. Stop shaming people for their open source repos.
When did the software community get so bad at handling legitimate critique?
"You should really be ashamed of yourself for doing this" is an inappropriate response for basically anything but kicking puppies.
Re: Show HN: Token price calculator for 400+ LLMs
#39Earlier quoted context omitted.
Imagine a coffee shop refusing to have a price list until after the coffee's been made.
In many countries a taxi won't tell you how much the ride will cost. The first time I traveled to somewhere that negotiated the cost up front it blew my mind. Frequently, contracts will have room for additional charges if circumstances change even a little, or products will have a market rate (fish, equity, etc.). It might seem absurd but variable cost things are not uncommon.
In this case there's nothing that's variable, though, and the competition is able to pull it off precisely. Indeed, they themselves were able to do it before!
Re: Show HN: Token price calculator for 400+ LLMs
#40Earlier quoted context omitted.
When did the software community get so bad at handling legitimate critique?
"You should really be ashamed of yourself for doing this" is an inappropriate response for basically anything but kicking puppies.
That sounds somewhat specious, lying about what you support in your cost calculation library is a pretty big oof. It's hard to rank it versus kicking puppys, but, I don't think we have to stack-rank bad things to figure out if it's okay to call out unethical behavior.