Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3

blog.google

Post reply on HN
Print view View original

971–980 of 1001 posts

Re: Gemini 3

#971

Post by [deleted] »

thefourthchime wrote: post #103
prodigycorp wrote: post #22

I'm sure this is a very impressive model, but gemini-3-pro-preview is failing spectacularly at my fairly basic python benchmark. In fact, gemini-2.5-pro gets a lot closer (but is still wrong). For reference: gpt-5.1-thinking passes, gpt-5.1-instant fails, gpt-5-thinking fails, gpt-5-instant fails, sonnet-4.5 passes, opus-4.1 passes (lesser claude models fail). This is a reminder that benchmarks are meaningless – you…

I like to ask "Make a pacman game in a single html page". No model has ever gotten a decent game in one shot. My attempt with Gemini3 was no better than 2.5.

[deleted]

Re: Gemini 3

#972

Post by nsingh2 »

column wrote: post #929
tngranados wrote: post #920

Earlier quoted context omitted.

The point of benchmarking that is checking for hallucinations and overfitting. Does the model actually check the picture to count the legs or does it just see it's a dog and answer four because it knows dogs usually has four legs? It's a perfectly valid benchmark and very telling.

Very telling of what?

Telling of where the boundary of competence is for these models. And to show that these models aren't doing what most expect them to be doing, i.e. not counting legs, and maybe instead inferring information based on the overall image (dogs usually have 4 legs) to the detriment of find grained or out-of-distribution tasks.

Re: Gemini 3

#973

Post by blubber »

lairv wrote: post #293

Out of curiosity, I gave it the latest project euler problem published on 11/16/2025, very likely out of the training data Gemini thought for 5m10s before giving me a python snippet that produced the correct answer. The leaderboard says that the 3 fastest human to solve this problem took 14min, 20min and 1h14min respectively Even thought I expect this sort of problem to very much be in the distribution of what the mo…

Is this a problem for which the (human) solution is well documented an known and was learned during the training phase? Or is it a novel problem?

I personally think anthropomorphizing LLMs is a bad idea.

Re: Gemini 3

#974

Post by junon »

tart-lemonade wrote: post #769
junon wrote: post #435

Earlier quoted context omitted.

I thought this was a joke at first. It actually needs drive access to run someone else's prompt. Wild.

Not really, that's just basic access control. If you've used Colab or Cloud Shell (or even just Google Cloud in general, given the need to explicitly allow the usage of each service), it's not surprising at all.

Why does AI studio need access to my drive in order to run someone else's prompt? It's not a prompt for authentication with my Google account. I'm already signed in. It's prompting for what appears to be full read/write access to my drive account. No thanks.

Re: Gemini 3

#975

Post by LogicFailsMe »

nomilk wrote: post #837
thomasahle wrote: post #379

Earlier quoted context omitted.

I also used Gemini 3 Pro Preview. It finished it 271s = 4m31s. Sadly, the answer was wrong. It also returned 8 "sources", like stackexchange.com, youtube.com, mpmath.org, ncert.nic.in, and kangaroo.org.pk, even though I specifically told it not to use websearch. Still a useful tool though. It definitely gets the majority of the insights. Prompt: https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%...

Terrence Tao claims [0] contributions by the public are counter -productive since the energy required to check a contribution outweighs its benefit: > (for) most research projects, it would not help to have input from the general public. In fact, it would just be time-consuming, because error checking Since frontier LLMs make clumsy mistakes, they may fall into this category of 'error-prone' mathematician whose net c…

I would counter that any computationally correct code that accelerates any existing research code base is a net positive. I don't care how that is achieved as long as it doesn't sacrifice accuracy and precision.

We're not exactly swimming in power generation and efficient code uses less power.

Re: Gemini 3

#976

Post by nextworddev »

WhyOhWhyQ wrote: post #970

Why doesn't this spell the death of OpenAI? Maybe someone with a better business sense can explain, but here's what I'm seeing: OpenAI is going for the consumer-grade AI market, as opposed to a company like Anthropic making a specialized developer tool. Google can inject their AI tool in front of everybody in the world, and already have with Google AI search. All of these models are just going to reach parity eventua…

Yep. Except OpenAI is mainly burning LP money (saudis, softbank, pension funds)

Re: Gemini 3

#977

Post by thingsilearned »

I love that the recipe example is still being used as one of the main promising use cases for computers and now AGI. One day hopefully computers will solve that pressing problem...

Re: Gemini 3

#978

Post by Kiro »

hollowturtle wrote: post #864
Kiro wrote: post #832

Earlier quoted context omitted.

You can criticize the code but "wow looks like total shit" is such an embarrassing thing to say considering the context. Imagine going back a few years and show them a tool outputting this from text. No-one would believe it.

It simply is non impressive at all to me, we had an industry(games not web) that was the most innovativd and was able to do things, and in part still is, thousands of years ahead of the slop glorified here

Such an absurd comparison. It's like hearing the world's first audio recording, saying "wow sounds like total shit" and not be impressed because you can go listen to a full symphony orchestra live.

Re: Gemini 3

#979

Post by skylurk »

oblio wrote: post #800
skylurk wrote: post #550

Earlier quoted context omitted.

They've been training for months to draw that pelican, just for you to move the goalposts.

It's a pelican on a bike, not a goalpost. And bikes move. Well, pelicans move, too.

The chase is on!

Re: Gemini 3

#980

Post by ld4nt3 »

HarHarVeryFunny wrote: post #960
ld4nt3 wrote: post #824

Earlier quoted context omitted.

Its almost certain that it was, but the purpose of this puzzle benchmark is that it shouldn't really be possible just to be memorized by the amount of variations that can be created and other criteria detailed in it.

Sure, but the types of pattern in these problems do repeat, so I don't think it'd be too hard to RL train on these, whether public samples, or a privately generated more-of-the-same dataset, to improve performance a lot. Every company releasing new models leads with benchmark numbers, so it's hard to imagine they are not all putting a lot of effort into benchmark-maxxing.

Yes everyone is doing that on benchmarks but they are still somewhat useful and the likes arc agi even more, though we are not be able to quantize exactly how much better they are getting they are still necessary. For arc agi these are some big gains by which ever way the went about it, since everyone also has been trying to max it for the last 3 years but we do need to come up with better benchmarks/evals like arc tried.
Post reply on HN