Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Pro Model Card [pdf]

storage.googleapis.com

Post reply on HN
Print view View original

291–300 of 359 posts

Re: Gemini 3 Pro Model Card [pdf]

#291

Post by mountainriver »

HardCodedBias wrote: post #261
scrollop wrote: post #214

Earlier quoted context omitted.

Used an AI to populate some of 5.1 thinking's results. Benchmark | Gemini 3 Pro | Gemini 2.5 Pro | Claude Sonnet 4.5 | GPT-5.1 | GPT-5.1 Thinking ---------------------------|--------------|----------------|-------------------|---------|------------------ Humanity's Last Exam | 37.5% | 21.6% | 13.7% | 26.5% | 52% ARC-AGI-2 | 31.1% | 4.9% | 13.6% | 17.6% | 28% GPQA Diamond | 91.9% | 86.4% | 83.4% | 88.1% | 61% AIM 2025…

What? The 4.5 and 5.1 columns aren't thinking in Google's report? That's a scandal, IMO. Given that Gemini-3 seems to do "fine" against the thinking versions why didn't they post those results? I get that PMs like to make a splash but that's shockingly dishonest.

Every single time

Re: Gemini 3 Pro Model Card [pdf]

#292

Post by Miraste »

Workaccount2 wrote: post #218
mynti wrote: post #75

It is interesting that the Gemini 3 beats every other model on these benchmarks, mostly by a wide margin, but not on SWE Bench. Sonnet is still king here and all three look to be basically on the same level. Kind of wild to see them hit such a wall when it comes to agentic coding

I think Anthropic is reading the room, and just going to go hard on being "the" coding model. I suppose they feel that if they can win that, they can get an ROI without having to do full blown multimodality at the highest level. It's probably pretty liberating, because you can make a "spikey" intelligence with only one spike to really focus on.

It remains to be seen whether that works out for them, but it seems like a good bet to me. Coding is the most monetizatable use anyone has found for LLMs so far, and the most likely to persist past this initial hype bubble (if the Singularity doesn't work out :p).

Re: Gemini 3 Pro Model Card [pdf]

#293

Post by oppegard »

aoeusnth1 wrote: post #274
perardi wrote: post #268

Earlier quoted context omitted.

And a REALLY good macOS app. Like, kind of unreasonably good. You’d expect some perfunctory Electronic app that just barely wraps the website. But no, you get something that feels incredibly polished…more so than a lot of recent apps from Apple…and has powerful integrations into other apps, including text editors and terminals.

Which app are you referring to?

The ChatGPT app for Mac is native and very good.

Re: Gemini 3 Pro Model Card [pdf]

#295

Post by vitaflo »

xnx wrote: post #240
risyachka wrote: post #126

Earlier quoted context omitted.

> it's over for the other labs. Its not over and never will be for 2 decade old accounting software, it is definitely will not be over for other AI labs.

Can you explain what you mean by this? iPhone was the end of Blackberry. It seems reasonable that a smarter, cheaper, faster model would obsolete anything else. ChatGPT has some brand inertia, but not that much given it's barely 2 years old.

Ask yourself why Microsoft Teams won. These are business tools first and foremost.

Re: Gemini 3 Pro Model Card [pdf]

#296

Post by aerhardt »

Workaccount2 wrote: post #218
mynti wrote: post #75

It is interesting that the Gemini 3 beats every other model on these benchmarks, mostly by a wide margin, but not on SWE Bench. Sonnet is still king here and all three look to be basically on the same level. Kind of wild to see them hit such a wall when it comes to agentic coding

I think Anthropic is reading the room, and just going to go hard on being "the" coding model. I suppose they feel that if they can win that, they can get an ROI without having to do full blown multimodality at the highest level. It's probably pretty liberating, because you can make a "spikey" intelligence with only one spike to really focus on.

Codex has been good enough to me and it’s much cheaper.

I code non-trivial stuff with it like multi-threaded code and at least for my style of AI coding which is to do fairly small units of work with multiple revisions it is good enough for me to not to even consider the competition.

Just giving you a perspective on how the benchmarks might not be important at all for some people and how Claude may have a difficult time being the definitive coding model.

Re: Gemini 3 Pro Model Card [pdf]

#298

Post by [deleted] »

scrollop wrote: post #214
scrlk wrote: post #23

Benchmarks from page 4 of the model card: | Benchmark | 3 Pro | 2.5 Pro | Sonnet 4.5 | GPT-5.1 | |-----------------------|-----------|---------|------------|-----------| | Humanity's Last Exam | 37.5% | 21.6% | 13.7% | 26.5% | | ARC-AGI-2 | 31.1% | 4.9% | 13.6% | 17.6% | | GPQA Diamond | 91.9% | 86.4% | 83.4% | 88.1% | | AIME 2025 | | | | | | (no tools) | 95.0% | 88.0% | 87.0% | 94.0% | | (code execution) | 100% | -…

Used an AI to populate some of 5.1 thinking's results. Benchmark | Gemini 3 Pro | Gemini 2.5 Pro | Claude Sonnet 4.5 | GPT-5.1 | GPT-5.1 Thinking ---------------------------|--------------|----------------|-------------------|---------|------------------ Humanity's Last Exam | 37.5% | 21.6% | 13.7% | 26.5% | 52% ARC-AGI-2 | 31.1% | 4.9% | 13.6% | 17.6% | 28% GPQA Diamond | 91.9% | 86.4% | 83.4% | 88.1% | 61% AIM 2025…

[deleted]

Re: Gemini 3 Pro Model Card [pdf]

#300

Post by svnt »

FrequentLurker wrote: post #116
basch wrote: post #98

Earlier quoted context omitted.

At least at the moment, coming in late seems to matter little. Anyone with money can trivially catch up to a state of the art model from six months ago. And as others have said, late is really a function of spigot, guardrails, branding, and ux, as much as it is being a laggard under the hood.

> Anyone with money can trivially catch up to a state of the art model from six months ago. How come apple is struggling then?

Anyone with enough money and without an entrenched management hierarchy preventing the right people from being hired and enabled to run the project.
Post reply on HN