Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Flash: Frontier intelligence built for speed

blog.google

Post reply on HN
Print view View original

431–440 of 609 posts

Re: Gemini 3 Flash: Frontier intelligence built for speed

#431

Post by willis936 »

floundy wrote: post #414
willis936 wrote: post #405

Earlier quoted context omitted.

That's too bad. Apple's most interesting value proposition is running local inference with big privacy promises. They wouldn't need to be the highest performer to offer something a lot of people might want.

Apple’s most interesting value proposition was ignoring all this AI junk and letting users click “not interested” on Apple Intelligence and never see it again. From a business perspective it’s a smart move (inasmuch as “integrating AI” is the default which I fundamentally disagree with) since Apple won’t be left holding the bag on a bunch of AI datacenters when/if the AI bubble pops. I don’t want to lose trust in App…

I also agree with this. Microsoft successfully removed my entire household from ever owning one of their products again after this year. Apple and linux make up the entire delta.

As long as Apple doesn't take any crazy left turns with their privacy policy then it should be relatively harmless if they add in a google wrapper to iOS (and we won't need to take hard right turns with grapheneOS phones and framework laptops).

Re: Gemini 3 Flash: Frontier intelligence built for speed

#432

Post by akoboldfrying »

grog454 wrote: post #376
Turskarama wrote: post #315

Earlier quoted context omitted.

The problem with publicly disclosing these is that if lots of people adopt them they will become targeted to be in the model and will no longer be a good benchmark.

This thought process is pretty baffling to me, and this is at least the second time I've encountered it on HN. What's the value of a secret benchmark to anyone but the secret holder? Does your niche benchmark even influence which model you use for unrelated queries? If LLM authors care enough about your niche (they don't) and fake the response somehow, you will learn on the very next query that something is amiss. No…

I actually think "concealing the question" is not only a good idea, but a rather general and powerful idea that should be much more widely deployed (but often won't be, for what I consider "emotional reasons").

Example: You are probably already aware that almost any metric that you try to use to measure code quality can be easily gamed. One possible strategy is to choose a weighted mixture of metrics and conceal the weights. The weights can even change over time. Is it perfect? No. But it's at least correlated with code quality -- and it's not trivially gameable, which puts it above most individual public metrics.

Re: Gemini 3 Flash: Frontier intelligence built for speed

#433

Post by grog454 »

eru wrote: post #420
grog454 wrote: post #419

Earlier quoted context omitted.

Ok, but then your "post" isn't scientific by definition since it cannot be verified. "Post" is in quotes because I don't know what you're trying to but you're implying some sort of public discourse. For fun: https://chatgpt.com/s/t_694361c12cec819185e9850d0cf0c629

I didn't see anyone claiming any 'science'? Did I miss something?

I guess there's two things I'm still stuck on:

1. What is the purpose of the benchmark?

2. What is the purpose of publicly discussing a benchmark's results but keeping the methodology secret?

To me it's in the same spirit as claiming to have defeated alpha zero but refusing to share the game.

Re: Gemini 3 Flash: Frontier intelligence built for speed

#434

Post by eru »

baq wrote: post #300
Uehreka wrote: post #256

Earlier quoted context omitted.

I would expect open weights models to always lag behind; training is resource-intensive and it’s much easier to finance if you can make money directly from the result. So in a year we may have a ~700B open weights model that competes with Gemini 3, but by then we’ll have Gemini 4, and other things we can’t predict now.

If Gemini 3 flash is really confirmed close to Opus 4.5 at coding and a similarly capable model is open weights, I want to buy a box with an usb cable that has that thing loaded, because today that’s enough to run out of engineering work for a small team.

Open weights doesn't mean you can necessarily run it on a (small) box.

If Google released their weights today, it would technically be open weight; but I doubt you'd have an easy time running the whole Gemini system outside of Google's datacentres.

Re: Gemini 3 Flash: Frontier intelligence built for speed

#435

Post by DrewADesign »

prettyblocks wrote: post #313
lambda wrote: post #267

Earlier quoted context omitted.

I'm a significant genAI skeptic. I periodically ask them questions about topics that are subtle or tricky, and somewhat niche, that I know a lot about, and find that they frequently provide extremely bad answers. There have been improvements on some topics, but there's one benchmark question that I have that just about every model I've tried has completely gotten wrong. Tried it on LMArena recently, got a comparison…

I don't think tricky niche knowledge is the sweet spot for genai and it likely won't be for some time. Instead, it's a great replacement for rote tasks where a less than perfect performance is good enough. Transcription, ocr, boilerplate code generation, etc.

Yeah, but tests like that deliberately prod the boundaries of its capability rather than how well it does what it’s good at.

Re: Gemini 3 Flash: Frontier intelligence built for speed

#436

Post by eru »

lukan wrote: post #345
pests wrote: post #338

Earlier quoted context omitted.

How many people do you know? Do you talk to your local shop keeper? Or the clerk at the gas station? How are they using AI? I'm a pretty techy person with a lot of tech friends, and I know more people not using AI (on purpose, or lack of knowledge) then do.

Hm, quite some. Like I said, it depends what you count as AI. Just googling means you use AI nowdays.

Whether Googling something counts as AI has more to do with the shifting definition of AI over time, then with Googling itself.

Remember, really back in the day the A* search algorithm was part of AI.

If you had asked anyone in the 1970s whether a box that given a query pinpoints the right document that answers that question (aka Google search in the early 2000s), they'd definitely would have called it AI.

Re: Gemini 3 Flash: Frontier intelligence built for speed

#438

Post by wasabi991011 »

albumen wrote: post #287
scrollop wrote: post #225

Earlier quoted context omitted.

Also https://artificialanalysis.ai/evaluations/omniscience Prepare to be amazed

I’m amazed by how much Gemini 3 flash hallucinates; it performs poorly in that metric (along with lots of other models). In the Hallucination Rate vs. AA-Omniscience Index chart, it’s not in the most desirable quadrant; GPT-5.1 (high), opus 4.5 and 4.5 haiku are. Can someone explain how Gemini 3 pro/flash then do so well then in the overall Omniscience: Knowledge and Hallucination Benchmark?

Hallucination rate is hallucination/(hallucination+partial+ignored), while omniscience is correct-hallucination.

One hypothesis is that gemini 3 flash refuses to answer when unsuure less often than other models, but when sure is also more likely to be correct. This is consistent with it having the best accuracy score.

Re: Gemini 3 Flash: Frontier intelligence built for speed

#439

Post by ZuoCen_Liu »

samyok wrote: post #4

Don’t let the “flash” name fool you, this is an amazing model. I have been playing with it for the past few weeks, it’s genuinely my new favorite; it’s so fast and it has such a vast world knowledge that it’s more performant than Claude Opus 4.5 or GPT 5.2 extra high, for a fraction (basically order of magnitude less!!) of the inference time and price

What type of question is your one about testing AI inference time?

Re: Gemini 3 Flash: Frontier intelligence built for speed

#440

Post by grog454 »

akoboldfrying wrote: post #432
grog454 wrote: post #376

Earlier quoted context omitted.

This thought process is pretty baffling to me, and this is at least the second time I've encountered it on HN. What's the value of a secret benchmark to anyone but the secret holder? Does your niche benchmark even influence which model you use for unrelated queries? If LLM authors care enough about your niche (they don't) and fake the response somehow, you will learn on the very next query that something is amiss. No…

I actually think "concealing the question" is not only a good idea, but a rather general and powerful idea that should be much more widely deployed (but often won't be, for what I consider "emotional reasons"). Example: You are probably already aware that almost any metric that you try to use to measure code quality can be easily gamed. One possible strategy is to choose a weighted mixture of metrics and conceal the…

It's hard to have any certainty around concealment unless you are only testing local LLMs. As a matter of principle I assume the input and output of any query I run in a remote LLM is permanently public information (same with search queries).

Will someone (or some system) see my query and think "we ought to improve this"? I have no idea since I don't work on these systems. In some instances involving random sampling... probably yes!

This is the second reason I find the idea of publicly discussing secret benchmarks silly.

Post reply on HN