Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Pro: the frontier of vision AI

blog.google

Post reply on HN
Print view View original

171–180 of 309 posts

Re: Gemini 3 Pro: the frontier of vision AI

#171

Post by themanmaran »

aziis98 wrote: post #105

> Pointing capability: Gemini 3 has the ability to point at specific locations in images by outputting pixel-precise coordinates. Sequences of 2D points can be strung together to perform complex tasks, such as estimating human poses or reflecting trajectories over time Does somebody know how to correctly prompt the model for these tasks or even better provide some docs? The pictures with the pretty markers are apprec…

Simon Wilson has some good blogs on this: https://simonwillison.net/2024/Aug/26/gemini-bounding-box-vi...

Re: Gemini 3 Pro: the frontier of vision AI

#172

Post by FeepingCreature »

cyanmagenta wrote: post #145
qnleigh wrote: post #142

Earlier quoted context omitted.

It's not obvious to me whether we should count these errors as failures of intelligence or failures of perception. There's at least a loose analogy to optical illusion, which can fool humans quite consistently. Now you might say that a human can usually figure out what's going on and correctly identify the illusion, but we have the luxury of moving our eyes around the image and taking it in over time, while the model…

I am having trouble understanding the distinction you’re trying to make here. The computer has the same pixel information that humans do and can spend its time analyzing it in any way it wants. My four-year-old can count the legs of the dog (and then say “that’s silly!”), whereas LLMs have an existential crisis because five-legged-dogs aren’t sufficiently represented in the training data. I guess you can call that pe…

Your kid, it should be noted, has a massively bigger brain than the LLM. I think the surprising thing here maybe isn't that the vision models don't work well in corner cases but that they work at all.

Also my bet would be that video capable models are better at this.

Re: Gemini 3 Pro: the frontier of vision AI

#173

Post by FeepingCreature »

danielvaughn wrote: post #57
Workaccount2 wrote: post #48

Well It is the first model to get partial-credit on an LLM image test I have. Which is counting the legs of a dog. Specifically, a dog with 5 legs. This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. In fact GPT5 wrote an edge detection script to see where "golden dog feet" met "bright green grass" to prove to me that there were only 4 legs. The script found 5, and GPT-5 the…

I don’t know much about AI, but I have this image test that everything has failed at. You basically just present an image of a maze and ask the LLM to draw a line through the most optimal path. Here’s how Nano Banana fared: https://x.com/danielvaughn/status/1971640520176029704?s=46

I kinda want to know what happens if you make it continue the line by one step 20 times in a row. A human can draw this gradually, the image model has to draw it in one shot all at once.

Re: Gemini 3 Pro: the frontier of vision AI

#174

Post by inquirerGeneral »

aziis98 wrote: post #105

> Pointing capability: Gemini 3 has the ability to point at specific locations in images by outputting pixel-precise coordinates. Sequences of 2D points can be strung together to perform complex tasks, such as estimating human poses or reflecting trajectories over time Does somebody know how to correctly prompt the model for these tasks or even better provide some docs? The pictures with the pretty markers are apprec…

[dead]

Re: Gemini 3 Pro: the frontier of vision AI

#175

Post by energy123 »

runarberg wrote: post #104
Workaccount2 wrote: post #48

Well It is the first model to get partial-credit on an LLM image test I have. Which is counting the legs of a dog. Specifically, a dog with 5 legs. This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. In fact GPT5 wrote an edge detection script to see where "golden dog feet" met "bright green grass" to prove to me that there were only 4 legs. The script found 5, and GPT-5 the…

This is exactly why I believe LLMs are a technological dead end. Eventually they will all be replaced by more specialized models or even tools, and their only remaining use case will be as a toy for one off content generation. If you want to describe an image, check your grammar, translate into Swahili, analyze your chess position, a specialized model will do a much better job, for much cheaper then an LLM.

I think we are too quick to discount the possibility that this flaw is slightly intentional, in the sense that the optimization has a tight budget to work with (equivalent of ~3000 tokens) so why would it waste capacity on this when it could improve capabilities around reading small text in obscured images? Sort of like humans have all these rules of thumbs that backfire in all these ways but that's the energy efficient way to do things.

Re: Gemini 3 Pro: the frontier of vision AI

#176

Post by seanmcdirmid »

Workaccount2 wrote: post #48

Well It is the first model to get partial-credit on an LLM image test I have. Which is counting the legs of a dog. Specifically, a dog with 5 legs. This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. In fact GPT5 wrote an edge detection script to see where "golden dog feet" met "bright green grass" to prove to me that there were only 4 legs. The script found 5, and GPT-5 the…

> This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs.

Most human beings, if they see a dog that has 5 legs, will quickly think they are hallucinating and the dog really only has 4 legs, unless the fifth leg is really really obvious. It is weird how humans are biased like that:

1. You can look directly at something and not see it because your attention is focused elsewhere (on the expected four legs).

2. Our pre-existing knowledge (dogs have four legs) influences how we interpret visual information from the bottom-up.

3. Our brain actively filters out "unimportant" details that don't align with our expectations or the main "figure" of the dog.

Attention should fix this however, like if you ask the AI to count the number of legs the dog has specifically, it shouldn't go nuts.

A straight up "dumber" computer algorithm that isn't trained extensively on real and realistic image data is going to get this right more often than a transformer that was.

Re: Gemini 3 Pro: the frontier of vision AI

#178

Post by sumedh »

rohanlikesai wrote: post #58
simonw wrote: post #2

In case the article author sees this, the "HTML transcription" link is broken - it goes to https://aistudio-preprod.corp.google.com/prompts/1GUEWbLIlpX... which is a Google-employee-only URL.

hey, it's Rohan (the author of the article) - appreciate you catching this, we just fixed this!

You should try using AI to check such things :)

Re: Gemini 3 Pro: the frontier of vision AI

#179

Post by irjustin »

seanmcdirmid wrote: post #176
Workaccount2 wrote: post #48

Well It is the first model to get partial-credit on an LLM image test I have. Which is counting the legs of a dog. Specifically, a dog with 5 legs. This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. In fact GPT5 wrote an edge detection script to see where "golden dog feet" met "bright green grass" to prove to me that there were only 4 legs. The script found 5, and GPT-5 the…

> This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. Most human beings, if they see a dog that has 5 legs, will quickly think they are hallucinating and the dog really only has 4 legs, unless the fifth leg is really really obvious. It is weird how humans are biased like that: 1. You can look directly at something and not see it because your attention is focused elsewhere (o…

> It is weird how humans are biased like that.

We're all just pattern matching machines and we humans are very good at it.

So much so that we have the sayings - you can't teach an old dog... and a specialist in their field only sees hammer => nails.

Evolution anyone?

Post reply on HN