Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Pro Model Card [pdf]

storage.googleapis.com

Post reply on HN
Print view View original

161–170 of 359 posts

Re: Gemini 3 Pro Model Card [pdf]

#161

Post by siva7 »

alecco wrote: post #43
margorczynski wrote: post #11

If these numbers are true then OpenAI is probably done, Anthropic too. Still, it's hard to see an effective monetization method for this tech and it clearly is eating Google's main pie which is search.

For SWE it is the same ranking. But if Google's $20/mo plan is comparable to the $100-200 plans for OpenAI and Anthropic, yes they are done. But we'll have to wait a few weeks to see if the nerfed model post-release is still as good.

I have a few secret prompts to test complex reasoning capabilities of new models (in law and medicine). Gemini (2.5 pro) is by a wide margin behind Anthropic (sonnet 4.5 basic thinking) and Openai (pro model) on my own benchmark and I trust my own benchmark more than public leaderboards. So it's the other way around. Google is trying to catch up where the others are. It just doesn't seem so to some because Google undercuts prices and most people don't have own complex problems with a verified solution to test against (so they could see how bad Gemini is in reality)

Re: Gemini 3 Pro Model Card [pdf]

#162

Post by spwa4 »

decster wrote: post #45
itsmevictor wrote: post #26

Earlier quoted context omitted.

Noteworthily, although Gemini 3 Pro seems to have much benchmark scores than other models across the board (including compared to Claude), it's not the case for coding, where it appears to score essentially the same as the others. I wonder why that is. So far, IMHO, Claude Code remains significantly better than Gemini CLI. We'll see whether that changes with Gemini 3.

from my experience, the quality of gemini-cli isn't great, experiencing lot of stupied bug.

Google is currently constantly laying off people. Everyone who really exceeds has jumped ship, and the people who remain ... are not top of the class anymore.

Not that Google didn't use to have problems shipping useful things. But it's gotten a lot worse.

Re: Gemini 3 Pro Model Card [pdf]

#163

Post by Palmik »

HereBePandas wrote: post #122
Palmik wrote: post #100

Earlier quoted context omitted.

The reported results where GPT 5.1 beats Gemini 3 are on SWE Bench Verified, and GPT 5.1 Codex also beats Gemini 3 on Terminal Bench.

You're right on SWE Bench Verified, I missed that and I'll delete my comment. GPT 5.1 Codex beats Gemini 3 on Terminal Bench specifically on Codex CLI, but that's apples-to-oranges (hard to tell how much of that is a Codex-specific harness vs model). Look forward to seeing the apples-to-apples numbers soon, but I wouldn't be surprised if Gemini 3 wins given how close it comes in these benchmarks.

All evals on Terminal Bench require some harness. :) Or "Agent", as Terminal Bench calls it. Presumably the Gemini 3 are using Gemini CLI.

Re: Gemini 3 Pro Model Card [pdf]

#164

Post by tngranados »

embedding-shape wrote: post #38

Curiously, this website seems to be blocked in Spain for whatever reason, and the website's certificate is served by `allot.com/emailAddress=info@allot.com` which obviously fails... Anyone happen to know why? Is this website by any change sharing information on safe medical abortions or women's rights, something which has gotten websites blocked here before?

It works fine for me using Movistar

Re: Gemini 3 Pro Model Card [pdf]

#165

Post by Palmik »

HereBePandas wrote: post #112
Palmik wrote: post #96

Earlier quoted context omitted.

Also does not beat GPT-5.1 Codex on terminal bench (57.8% vs 54.2%): https://www.tbench.ai/ I did not bother verifying the other claims.

Not apples-to-apples. "Codex CLI (GPT-5.1-Codex)", which the site refers to, adds a specific agentic harness, whereas the Gemini 3 Pro seems to be on a standard eval harness. It would be interesting to see the apples-to-apples figure, i.e. with Google's best harness alongside Codex CLI.

All evals on Terminal Bench require some harness. :) Or "Agent", as Terminal Bench calls it. Presumably the Gemini 3 are using Gemini CLI.

What do you mean by "standard eval harness"?

Re: Gemini 3 Pro Model Card [pdf]

#166

Post by __jl__ »

nilayj wrote: post #144

Curious to see the API pricing. SOTA performance across tasks at a price cheaper than GPT 5 / Claude would make mostly everyone switch to Gemini.

Same here. They have been aggressively increasing prices with each iteration (maybe because they started so low). Still hope that is not the case this time. GPT 5.1 is priced pretty aggressively so maybe that is an incentive to keep the current gemini API prices.

Re: Gemini 3 Pro Model Card [pdf]

#167

Post by rfoo »

martinald wrote: post #104
catigula wrote: post #63

I know this is a little controversial but the lack of performance on SWE-bench is hugely disappointing I think economically. These models don’t have any viable path to profitability if they can’t take engineering jobs.

I thought that but it does do a lot better on other benchmarks. Perhaps SWE bench just doesn't capture a lot of the improvement? If the web design improvements people have been posting on twitter, I suspect this will be a huge boon for developers. SWE benchmark is really testing bugfixing/feature dev more. Anyway let's see. I'm still hyped!

SWE Bench doesn't even test bugfixing / feature dev properly after you achieve roughly 70% if you don't benchmaxx it .

Re: Gemini 3 Pro Model Card [pdf]

#168

Post by bootlooped »

adriand wrote: post #65
remus wrote: post #34

Earlier quoted context omitted.

Agreed, too early to write off others entirely. It'll be interesting to see who comes out the other side of the bubble with a working business.

Anthropic has a fairly significant lead when it comes to enterprise usage and for coding. This seems like a workable business model to me.

I feel this is a tenuous position though. I find it incredibly easy to switch to Gemini CLI when I want a second opinion, or when Claude is down.

Re: Gemini 3 Pro Model Card [pdf]

#169

Post by alyxya »

mynti wrote: post #75

It is interesting that the Gemini 3 beats every other model on these benchmarks, mostly by a wide margin, but not on SWE Bench. Sonnet is still king here and all three look to be basically on the same level. Kind of wild to see them hit such a wall when it comes to agentic coding

I think Google probably cares more about a strong generalist model rather than solely optimizing for coding.
Post reply on HN