AI API Prices are 90% Subsidized
tinyml.substack.com
AI API Prices are 90% Subsidized
1–10 of 14 posts
Re: AI API Prices are 90% Subsidized
#2When the AI hype train left the station I said "we don't understand how these things work at all and they're going to get much cheaper to run" and that turned out to be... true.
Already vendors of legacy models like ChatGPT-4 have to subsidize inference to keep up with new entrants based on a better foundation. It's likely that inference costs can be brought down by another factor of ten or so so of course you have to 90% subsidize these to get where the industry will be in 2-3 years.
Re: AI API Prices are 90% Subsidized
#3No lol. The quality is mostly bad. Basically u need to prompt in detail like writing a novel for llm to understand. At that price, we want real AI who can really have common sense, not just an autocompletion tool.
Stop adverting LLM as AI, instead sell it as a superior copy & paste engine.
What's worst about LLM, is the more you talk with it, the worse it became to the point of broken.
Re: AI API Prices are 90% Subsidized
#4Subsidized is probably not the correct word here, it's probably more like loss leader in the race of the land grab.
It's like the early days of the internet when everything was amazing and all the people who put money into this thing were "losing" their money.
It's going to be like this until monopolization and moat becomes defensible and then they will enshittify the crap of it and make their money back 10x, 100x etc.
Re: AI API Prices are 90% Subsidized
#5This ignores batching - token generation is much more efficient in batch - and I strongly suspect is itself written by AI, given the heavy use of bullets
Re: AI API Prices are 90% Subsidized
#6This calculation doesn't account for batches, it makes no sense.
Re: AI API Prices are 90% Subsidized
#7This ignores batching - token generation is much more efficient in batch - and I strongly suspect is itself written by AI, given the heavy use of bullets
is it common for adjacent tokens to use the same weights in a memory cache?
Re: AI API Prices are 90% Subsidized
#8This calculation doesn't account for batches, it makes no sense.
On average how much does batching bring costs down?
Re: AI API Prices are 90% Subsidized
#9Re: AI API Prices are 90% Subsidized
#10I’ve been playing around with Gemma E4B and have gotten really good results. That’s a model you can run on a phone. So although prices have been going up recently I suspect they will start to fall again soon.