It can be just Google trying to capitalize Gemini's increasing popularity. Until 2.5 Gemini was a total underdog. Less so since 2.5.
The End of Moore's Law for AI? Gemini Flash Offers a Warning
31–40 of 78 posts
Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning
#32> If you’re building batch tasks with LLMs and are looking to navigate this new cost landscape, feel free to reach out to see how Sutro can help. I don't have any reason to doubt the reasoning this article is doing or the conclusions it reaches, but it's important to recognize that this article is part of a sales pitch.
There’s absolutely nothing wrong with putting a small plug at the end of an article.
Of course not.
But the thrust of the article is that contrary to conventional wisdom, we shouldn't expect llm models to continue getting more efficient, and so its worthwhile to explore other options for cost savings in inference, such as batch processing.
The conclusion they reach is one which directly serves what they're selling.
I'll repeat; I'm not disputing anything in this article. I'm really not, I'm not even trying to be coy and make allusions without directly saying anything. If I thought this was bullshit I'm not afraid to semi-anonymously post a comment saying so.
But this is advertising, just like Backblaze's hard drive reliability blog posts are advertising.
Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning
#33> This is the first time a major provider has backtracked on the price of an established model Arguably that was Haiku 3.5 in October 2024. I think the same hypothesis could apply though, that you price your model expecting a certain average input size, and then adjust price up to accommodate the reality that people use that cheapest model when they want to throw as much as they can into the context.
I had the same thought about haiku 3.5. They claimed it was due to the model being more capable, which basically means that they raised the price because they could.
Then there is Poe with its pricing games. Prices at Poe have been going up over time since they were extremely aggressive to gain market share presumably under the assumption that there would be reduced pricing in the future and the reduced pricing for LLMs did not materialize.
Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning
#34Extremely doubtful that it boils down to quadratic scaling of attention. That whole issue is a leftover from the days of small bert models with very few parameters.
For large models, compute is very rarely dominated by attention. Take, for example, this FLOPs calculation from https://www.adamcasson.com/posts/transformer-flops
Compute per token = 2(P + L × W × D)
P: total parameters L: Number of Layers W: context size D: Embedding dimension
For Llama 8b, the window size starts dominating compute cost per token only at 61k tokens.
Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning
#35"In a move that at first went unnoticed, Google significantly increased the price of its popular Gemini 2.5 Flash model" It's not quite that simple. Gemini 2.5 Flash previously had two prices, depending on if you enabled "thinking" mode or not. The new 2.5 Flash has just a single price, which is a lot more if you were using the non-thinking mode and may be slightly less for thinking mode. Another way to think about t…
I really hate the thinking. I do my best to disable it but don't always remember. So often it just gets into a loop second guessing itself until it hits the token limit. It's rare it figures anything out while it's thinking too but maybe that's because I'm better at writing prompts.
Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning
#36>By embracing batch processing and leveraging the power of cost-effective open-source models, you can sidestep the price floor and continue to scale your AI initiatives in ways that are no longer feasible with traditional APIs. Context size is the real killer when you look at running open source alternatives on your own hardware. Has anything even come close to the 100k+ range yet?
Yes! Both Llama 3 and Gemma 3 have 128k context windows.
Llama 3 had a 8192 token context window. Llama 3.1 increased it to 131072.
Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning
#37>By embracing batch processing and leveraging the power of cost-effective open-source models, you can sidestep the price floor and continue to scale your AI initiatives in ways that are no longer feasible with traditional APIs. Context size is the real killer when you look at running open source alternatives on your own hardware. Has anything even come close to the 100k+ range yet?
Mistral Small 3.2 has a 131072 token context window.
Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning
#38> If you’re building batch tasks with LLMs and are looking to navigate this new cost landscape, feel free to reach out to see how Sutro can help. I don't have any reason to doubt the reasoning this article is doing or the conclusions it reaches, but it's important to recognize that this article is part of a sales pitch.
Yes, we're a startup! And LLM inference is a major component of what we do - more importantly, we're working on making these models accessible as analytical processing tools, so we have a strong focus on making them cost-effective at scale.
I see your prices page lists the average cost per million tokens. Is that because you are using the formula you describe, which depends on hardware time and throughput?
> API Price ≈ (Hourly Hardware Cost / Throughput in Tokens per Hour) + Margin
Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning
#39"In a move that at first went unnoticed, Google significantly increased the price of its popular Gemini 2.5 Flash model" It's not quite that simple. Gemini 2.5 Flash previously had two prices, depending on if you enabled "thinking" mode or not. The new 2.5 Flash has just a single price, which is a lot more if you were using the non-thinking mode and may be slightly less for thinking mode. Another way to think about t…
I really hate the thinking. I do my best to disable it but don't always remember. So often it just gets into a loop second guessing itself until it hits the token limit. It's rare it figures anything out while it's thinking too but maybe that's because I'm better at writing prompts.
I have the impression that the thinking helps even if the actual content of the thinking output is nonsense. It awards more cycles to the model to think about the problem.
Re: The End of Moore's Law for AI? Gemini Flash Offers a Warning
#40Earlier quoted context omitted.
I really hate the thinking. I do my best to disable it but don't always remember. So often it just gets into a loop second guessing itself until it hits the token limit. It's rare it figures anything out while it's thinking too but maybe that's because I'm better at writing prompts.
I have the impression that the thinking helps even if the actual content of the thinking output is nonsense. It awards more cycles to the model to think about the problem.
That would be strange. There's no hidden memory or data channel, the "thinking" output is all the model receives afterwards. If it's all nonsense, then nonsense is all it gets. I wouldn't be completely surprised if a context with a bunch of apparent nonsense still helps somehow, LLMs are weird, but it would be odd.