Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Pro Model Card [pdf]

storage.googleapis.com

Post reply on HN
Print view View original

241–250 of 359 posts

Re: Gemini 3 Pro Model Card [pdf]

#241

Post by lkbm »

SiempreViernes wrote: post #210
transcriptase wrote: post #42

There needs to be a sycophancy benchmark in these comparisons. More baseless praise and false agreement = lower score.

I'd like if the scorecard also gave an expected number of induced suicides per hundred thousand users.

https://llmdeathcount.com/ shows 15 deaths so far, and LLM user count is in the low billions, which puts us on the order of 0.0015 deaths per hundred thousand users.

I'm guessing LLM Death Count is off by an OOM or two, so we could be getting close to one in a million.

Re: Gemini 3 Pro Model Card [pdf]

#242

Post by PrairieFire »

amluto wrote: post #229
postalcoder wrote: post #79

Earlier quoted context omitted.

Google was never really late. Where people perceived Google to have dropped the ball was in its productization of AI. The Google's Bard branding stumble was so (hilariously) bad that it threw a lot of people off the scent. My hunch is that, aside from "safety" reasons, the Google Books lawsuit left some copyright wounds that Google did not want to reopen.

Google’s productization is still rather poor. If I want to use OpenAI’s models, I go to their website, look up the price and pay it. For Google’s, I need to figure out whether I want AI Studio or Google Cloud Code Assist or AI Ultra, etc, and if this is for commercial use where I need to prevent Google from training on my data, figuring out which options work is extra complicated. As of a couple weeks ago (the last t…

Not to mention no macOS app. This is probably unimportant to many in the hn audience, but more broadly it matters for your average knowledge worker.

Re: Gemini 3 Pro Model Card [pdf]

#243

Post by rvz »

patates wrote: post #21
rvz wrote: post #2

> The training dataset also includes: publicly available datasets that are readily downloadable; data obtained by crawlers; licensed data obtained via commercial licensing agreements; user data (i.e., data collected from users of Google products and services to train AI models, along with user interactions with the model) in accordance with Google’s relevant terms of service, privacy policy, service-specific policies…

It says "pursuant to user controls, where appropriate". We can now sleep peacefully with the knowledge that Google will give us the tools to disable this where it's not inappropriate.

So that's why Google is getting sued for Gemini being enabled by default in Gmail and analyzing emails and our data; completely going against whatever privacy policy they came up with. [0]

I don't expect them to follow their own privacy policies.

[0] https://www.yahoo.com/news/articles/google-sued-over-gemini-...

Re: Gemini 3 Pro Model Card [pdf]

#244

Post by radial_symmetry »

aliljet wrote: post #239

What's wild here is that among every single score they've absolutely killed, somehow, Anthropic and Claude Sonnet 4.5 have won a single victory in the fight: SWE Bench Verified and only by a singular point. I already enjoy Gemini 2.5 pro for planning and if Gemini 3 is priced similarly, I'll be incredibly happy to ditch the painfully pricey Claude max subscription. To be fair, I've already got an extremely sour taste…

SWE bench is weird because Claude has always underperformed on it relative to other models despite Claude Code blowing them away. The real test will be if Gemini CLI beats Claude Code, both using the agentic framework and tools they were trained on.

Re: Gemini 3 Pro Model Card [pdf]

#245

Post by roman_soldier »

scrlk wrote: post #23

Benchmarks from page 4 of the model card: | Benchmark | 3 Pro | 2.5 Pro | Sonnet 4.5 | GPT-5.1 | |-----------------------|-----------|---------|------------|-----------| | Humanity's Last Exam | 37.5% | 21.6% | 13.7% | 26.5% | | ARC-AGI-2 | 31.1% | 4.9% | 13.6% | 17.6% | | GPQA Diamond | 91.9% | 86.4% | 83.4% | 88.1% | | AIME 2025 | | | | | | (no tools) | 95.0% | 88.0% | 87.0% | 94.0% | | (code execution) | 100% | -…

Why is Grok 4.1 not in the benchmarks?

Re: Gemini 3 Pro Model Card [pdf]

#246

Post by xnx »

vharish wrote: post #128
mynti wrote: post #75

It is interesting that the Gemini 3 beats every other model on these benchmarks, mostly by a wide margin, but not on SWE Bench. Sonnet is still king here and all three look to be basically on the same level. Kind of wild to see them hit such a wall when it comes to agentic coding

From my personal experience using the CLI agentic coding tools, I think gemini-cli is fairly on par with the rest in terms of the planning/code that is generated. However, when I recently tried qwen-code, it gave me a better sense of reasoning and structure that geimini. Claude definitely has it's own advantages but is expensive(at least for some if not for all). My point is, although the model itself may have perfor…

Gemini CLI is moving really fast. Noticeable improvements in features and functionality every week.

Re: Gemini 3 Pro Model Card [pdf]

#247

Post by theptip »

Alifatisk wrote: post #71
scrlk wrote: post #23

Benchmarks from page 4 of the model card: | Benchmark | 3 Pro | 2.5 Pro | Sonnet 4.5 | GPT-5.1 | |-----------------------|-----------|---------|------------|-----------| | Humanity's Last Exam | 37.5% | 21.6% | 13.7% | 26.5% | | ARC-AGI-2 | 31.1% | 4.9% | 13.6% | 17.6% | | GPQA Diamond | 91.9% | 86.4% | 83.4% | 88.1% | | AIME 2025 | | | | | | (no tools) | 95.0% | 88.0% | 87.0% | 94.0% | | (code execution) | 100% | -…

These numbers are impressive, at least to say. It looks like Google has produced a beast that will raise the bar even higher. What's even more impressive is how Google came into this game late and went from producing a few flops to being the leader at this (actually, they already achieved the title with 2.5 Pro). What makes me even more curious is the following > Model dependencies: This model is not a modification o…

> So did they start from scratch with this one

Their major version number bumps are a new pre-trained model. Minor bumps are changes/improvements to post-training on the same foundation.

Re: Gemini 3 Pro Model Card [pdf]

#248

Post by __jl__ »

API pricing is up to $2/M for input and $12/M for output

For comparison: Gemini 2.5 Pro was $1.25/M for input and $10/M for output Gemini 1.5 Pro was $1.25/M for input and $5/M for output

Re: Gemini 3 Pro Model Card [pdf]

#249

Post by rvnx »

spoaceman7777 wrote: post #183
scrlk wrote: post #23

Benchmarks from page 4 of the model card: | Benchmark | 3 Pro | 2.5 Pro | Sonnet 4.5 | GPT-5.1 | |-----------------------|-----------|---------|------------|-----------| | Humanity's Last Exam | 37.5% | 21.6% | 13.7% | 26.5% | | ARC-AGI-2 | 31.1% | 4.9% | 13.6% | 17.6% | | GPQA Diamond | 91.9% | 86.4% | 83.4% | 88.1% | | AIME 2025 | | | | | | (no tools) | 95.0% | 88.0% | 87.0% | 94.0% | | (code execution) | 100% | -…

Wow. They must have had some major breakthrough. Those scores are truly insane. O_O Models have begun to fairly thoroughly saturate "knowledge" and such, but there are still considerable bumps there But the _big news_, and the demonstration of their achievement here, are the incredible scores they've racked up here for what's necessary for agentic AI to become widely deployable. t2-bench. Visual comprehension. Comput…

The problem is that we know in advance what is the benchmark, so Humanity's Last Exam for example, it's way easier to optimize your model when you have seen the questions before.

Re: Gemini 3 Pro Model Card [pdf]

#250

Post by CjHuber »

koakuma-chan wrote: post #231

> Gemini 3 Pro was trained using Google’s Tensor Processing Units (TPUs) NVDA is down 3.26%

If it’s because of that, then honestly it’s as insane as the deepseek thing where all the info was released weeks before but the markt got nervous only when they released an app. I mean info about Gemini 3 is out quite a while now and of course they trained it using TPUs, I didn’t even think that was in question.
Post reply on HN