I’ve been tinkering with the last version for code gen. This update might finally put it on par with Claude for latency. Anyone tried benchmarking the new preview yet?
Improved Gemini 2.5 Flash and Flash-Lite
21–30 of 285 posts
Re: Improved Gemini 2.5 Flash and Flash-Lite
#22Gemini 2.5 Flash is an impressive model for its price. However, I don't understand why Gemini 2.0 Flash is still popular. From OpenRouter last week: * xAI: Grok Code Fast 1: 1.15T * Anthropic: Claude Sonnet 4: 586B * Google: Gemini 2.5 Flash: 325B * Sonoma Sky Alpha: 227B * Google: Gemini 2.0 Flash: 187B * DeepSeek: DeepSeek V3.1 (free): 180B * xAI: Grok 4 Fast (free): 158B * OpenAI: GPT-4.1 Mini: 157B * DeepSeek: De…
I know we have a lot of workloads at my company on older models no one has bothered to upgrade yet
Primarily classification or something else?
Re: Improved Gemini 2.5 Flash and Flash-Lite
#23I think a Model-specific SemVer needs to be created to be clearer as to what degree of change has taken place, in the age of model weights.
Something that distinguishes between a completely new pre-training process/architecture, and standard RLHF cycles/optimizations.
Re: Improved Gemini 2.5 Flash and Flash-Lite
#24I'm not even sure how to evaluate what a "better" LLM is, when I've tried running the exact same model (Qwen3) and prompt and gotten vastly different responses on Qwen Chat vs OpenRouter vs running the model locally.
There several reasons responses from the same model might vary:
- "temperature" - intentional random sampling from the most likely next tokens to improve "creativity" and help avoid repetition
- quantization - running models with lower numeric precision (saves on both memory and compute, without impacting accuracy too much)
- differences in/existence of a system prompt, especially when using something end-user-oriented like Qwen Chat
- not-quite-deterministic GPU acceleration
Benchmarks are usually run at temperature zero (always take the most likely next token), with the full-precision weights, and no additions to the benchmark prompt except necessary formatting and stuff like end-of-turn tokens. They also usually are multiple-choice or otherwise expect very short responses, which leaves less room for run-to-run variance.
Of course a benchmark still can't tell you everything - real-world performance can be very different.
Re: Improved Gemini 2.5 Flash and Flash-Lite
#25Earlier quoted context omitted.
Why is Grok so popular
I think it has been free in some editor plugins, which is probably a significant factor. I would rather use a model that is good than a model that is free, but different people have different priorities.
Non free has double usage than free. Free one uses your data for training.
Re: Improved Gemini 2.5 Flash and Flash-Lite
#26Am I the only one who is starting to feel the Gemini Flash models are better than Pro?
Flash is super fast, gets straight to the point.
Pro takes ages to even respond, then starts yapping endlessly, usually confuses itself in the process and ends up with a wrong answer.
Re: Improved Gemini 2.5 Flash and Flash-Lite
#27Google seems to be the main foundation model provider that's really focusing on the latency/TPS/cost dimensions. Anthropic/OpenAI are really making strides in model intelligence, but underneath some critical threshold of performance, the really long thinking times make workflows feel a lot worse in collaboration-style tools, vs a much snappier but slightly less intelligent model.
It's a delicate balance, because these Gemini models sometimes feel downright lobotomized compared to claude or gpt-5.
Re: Improved Gemini 2.5 Flash and Flash-Lite
#28Hopefully this isn't instead of the rumoured Gemini 3 pro this week.
Re: Improved Gemini 2.5 Flash and Flash-Lite
#29Gemini 2.5 Flash is an impressive model for its price. However, I don't understand why Gemini 2.0 Flash is still popular. From OpenRouter last week: * xAI: Grok Code Fast 1: 1.15T * Anthropic: Claude Sonnet 4: 586B * Google: Gemini 2.5 Flash: 325B * Sonoma Sky Alpha: 227B * Google: Gemini 2.0 Flash: 187B * DeepSeek: DeepSeek V3.1 (free): 180B * xAI: Grok 4 Fast (free): 158B * OpenAI: GPT-4.1 Mini: 157B * DeepSeek: De…
Price, 2.0 Flash is cheaper than 2.5 Flash but still very good model.
Re: Improved Gemini 2.5 Flash and Flash-Lite
#30Gemini 2.5 Flash has been the LLM I've used the most recently for a variety of domains, especially image inputs and structured outputs which beat both OpenAI and Anthropic in my opinion.