Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Pro: the frontier of vision AI

blog.google

Post reply on HN
Print view View original

91–100 of 309 posts

Re: Gemini 3 Pro: the frontier of vision AI

#91

Post by biophysboy »

pwagland wrote: post #74
danielvaughn wrote: post #57

Earlier quoted context omitted.

I don’t know much about AI, but I have this image test that everything has failed at. You basically just present an image of a maze and ask the LLM to draw a line through the most optimal path. Here’s how Nano Banana fared: https://x.com/danielvaughn/status/1971640520176029704?s=46

I tried this with Claude: ``` > [Image #1] Create a unicode "ascii-art" version of this image, with the optimal path through the maze highlighted in a solid colour. I'll create an ASCII art version of this maze with the solution path highlighted! ┌─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┐ ●●│ │ │ │ │ │ │ │ │ │ │ │ │ │ ├─┤●└─┴─┐ ├───┐ │ ╔═╗ ├─────┤ ╔═══╝ │ ╔═╝ ╔═╝ │ │ ╔═╝ ├─┤ │ │●●●●●└─┤ │ │ ║ │ │ │ ║…

Honestly, even though it failed, I'm kind of impressed that the trajectory mostly stays in the lines. If you remove all but two openings, does it work? The drawing you show has more than two openings, some of which are inaccessible from the inside of the maze.

Re: Gemini 3 Pro: the frontier of vision AI

#92

Post by knollimar »

webdood90 wrote: post #90
reducesuffering wrote: post #81

Earlier quoted context omitted.

"AI could never replace the creativity of a human" "Ok, I guess it could wipe out the economic demand for digital art, but it could never do all the autonomous tasks of a project manager" "Ok, I guess it could automate most of that away but there will always be a need for a human engineer to steer it and deal with the nuances of code" "Ok, well it could never automate blue collar work, how is it gonna wrench a pipe i…

> blue collar work I don't think it's fair to qualify this as blue collar work

It is definitely not. Entry pay is 60k and the senior guys I know make about 200k in HCoL areas. A few wear white dress shirts every day.

Re: Gemini 3 Pro: the frontier of vision AI

#93

Post by vunderba »

grog454 wrote: post #84
Workaccount2 wrote: post #48

Well It is the first model to get partial-credit on an LLM image test I have. Which is counting the legs of a dog. Specifically, a dog with 5 legs. This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. In fact GPT5 wrote an edge detection script to see where "golden dog feet" met "bright green grass" to prove to me that there were only 4 legs. The script found 5, and GPT-5 the…

What image are you using? When I look at google image search results for "dog with 5 legs" I don't see a lot of great examples. The first unequivocal "dog with 5 legs" was an illustration. Here was my conversation with Chat GPT. > How many legs does this dog have? "The dog in the image has four legs." > look closer. " looking closely, the drawing is a bit tricky because of the shading, but the dog actually has five v…

Sounds like they used GenAI to make them. The "Editor" models (Seedream, Nano-Banana) can easily integrate a fifth limb to create the "dog with awkward walking animation".

https://imgur.com/a/wXQskhL

Re: Gemini 3 Pro: the frontier of vision AI

#94

Post by irthomasthomas »

vunderba wrote: post #82
rottencupcakes wrote: post #79

Earlier quoted context omitted.

Super interesting. I replicated this. I passed the AIs this image and asked them how many fingers were on the hands: https://media.post.rvohealth.io/wp-content/uploads/sites/3/2... Claude said there were 3 hands and 16 fingers. GPT said there are 10 fingers. Grok impressively said "There are 9 fingers visible on these two hands (the left hand is missing the tip of its ring finger)." Gemini smashed it and said 12.

I just re-ran that image through Gemini 3.0 Pro via AI Studio and it reported: I've moved on to the right hand, meticulously tagging each finger. After completing the initial count of five digits, I noticed a sixth! There appears to be an extra digit on the far right. This is an unexpected finding, and I have counted it as well. That makes a total of eleven fingers in the image. This right HERE is the issue. It's not…

Thanks for that. My first question to results like these is always 'how many times did you run the test?'. N=1 tells us nothing. N=2 tells us something.

Re: Gemini 3 Pro: the frontier of vision AI

#95

Post by ichik »

Frankly, it's insane how laughably bad under scrutiny their own examples are. It both distorted the data and made the chart less readable (labels placement, segments separation, missing labels, worse contrast). And it combined them into one, so you you'll have harder time comparing them compared to the original image! Isn't it amazing that it added a toggle? Post author seems to think it deserves an exclamation point even.

Re: Gemini 3 Pro: the frontier of vision AI

#96

Post by majormajor »

AIorNot wrote: post #80
Rover222 wrote: post #64

Earlier quoted context omitted.

I just tried to get Gemini to produce an image of a dog with 5 legs to test this out, and it really struggled with that. It either made a normal dog, or turned the tail into a weird appendage. Then I asked both Gemini and Grok to count the legs, both kept saying 4. Gemini just refused to consider it was actually wrong. Grok seemed to have an existential crisis when I told it it was wrong, becoming convinced that I ha…

Its not that they aren’t intelligent its that they have been RL’d like crazy to not do that Its rather like as humans we are RL’d like crazy to be grossed out if we view a picture of a handsome man and beautiful woman kissing (after we are told they are brother and sister) - Ie we all have trained biases - that we are told to follow and trained on - human art is about subverting those expectations

Why should I assume that a failure that looks like a model just doing fairly simple pattern matching "this is dog, dogs don't have 5 legs, anything else is irrelevant" vs more sophisticated feature counting of a concrete instance of an entity is RL vs just a prediction failure due to training data not containing a 5-legged dog and an inability to go outside-of-distribution?

RL has been used extensively in other areas - such as coding - to improve model behavior on out-of-distribution stuff, so I'm somewhat skeptical of handwaving away a critique of a model's sophistication by saying here it's RL's fault that it isn't doing well out-of-distribution.

If we don't start from a position of anthropomorphizing the model into a "reasoning" entity (and instead have our prior be "it is a black box that has been extensively trained to try to mimic logical reasoning") then the result seems to be "here is a case where it can't mimic reasoning well", which seems like a very realistic conclusion.

Re: Gemini 3 Pro: the frontier of vision AI

#97

Post by knollimar »

webdood90 wrote: post #90
reducesuffering wrote: post #81

Earlier quoted context omitted.

"AI could never replace the creativity of a human" "Ok, I guess it could wipe out the economic demand for digital art, but it could never do all the autonomous tasks of a project manager" "Ok, I guess it could automate most of that away but there will always be a need for a human engineer to steer it and deal with the nuances of code" "Ok, well it could never automate blue collar work, how is it gonna wrench a pipe i…

> blue collar work I don't think it's fair to qualify this as blue collar work

I'm double replying to you since the replies are disparate subthreads. This is the necessary step so the robots who can turn wrenches know how to turn them. Those are near useless without perfect automated models.

Anything like this willl have trouble getting adopted since you'd need these to work with imperfect humans, which becomes way harder. You could bankroll a whole team of subcontractors (e.g. all trades) using that, but you would have one big liability.

The upper end of the complexity is similar to EDA in difficulty, imo. Complete with "use other layers for routing" problems.

I feel safer here than in programming. The senior guys won't be automated out any time soon, but I worry for Indian drafting firms without trade knowledge; the handholding I give them might go to an LLM soon.

Re: Gemini 3 Pro: the frontier of vision AI

#98

Post by Rover222 »

irthomasthomas wrote: post #87
Rover222 wrote: post #64

Earlier quoted context omitted.

I just tried to get Gemini to produce an image of a dog with 5 legs to test this out, and it really struggled with that. It either made a normal dog, or turned the tail into a weird appendage. Then I asked both Gemini and Grok to count the legs, both kept saying 4. Gemini just refused to consider it was actually wrong. Grok seemed to have an existential crisis when I told it it was wrong, becoming convinced that I ha…

Isn't this proof that LLMs still don't really generalize beyond their training data?

Kind of feels that way

Re: Gemini 3 Pro: the frontier of vision AI

#99

Post by Rover222 »

dwringer wrote: post #78
Rover222 wrote: post #64

Earlier quoted context omitted.

I just tried to get Gemini to produce an image of a dog with 5 legs to test this out, and it really struggled with that. It either made a normal dog, or turned the tail into a weird appendage. Then I asked both Gemini and Grok to count the legs, both kept saying 4. Gemini just refused to consider it was actually wrong. Grok seemed to have an existential crisis when I told it it was wrong, becoming convinced that I ha…

I had no trouble getting it to generate an image of a five-legged dog first try, but I really was surprised at how badly it failed in telling me the number of legs when I asked it in a new context, showing it that image. It wrote a long defense of its reasoning and when pressed, made up demonstrably false excuses of why it might be getting the wrong answer while still maintaining the wrong answer.

Yeah it gave me the 5-legged dog on the 4th or 5th try.

Re: Gemini 3 Pro: the frontier of vision AI

#100

Post by moonu »

vunderba wrote: post #77
Workaccount2 wrote: post #48

Well It is the first model to get partial-credit on an LLM image test I have. Which is counting the legs of a dog. Specifically, a dog with 5 legs. This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. In fact GPT5 wrote an edge detection script to see where "golden dog feet" met "bright green grass" to prove to me that there were only 4 legs. The script found 5, and GPT-5 the…

Anything that needs to overcome concepts which are disproportionately represented in the training data is going to give these models a hard time. Try generating: - A spider missing one leg - A 9-pointed star - A 5-leaf clover - A man with six fingers on his left hand and four fingers on his right You'll be lucky to get a 25% success rate. The last one is particularly ironic given how much work went into FIXING the ol…

https://gemini.google.com/share/8cef4b408a0a

Surprisingly, it got all of them right

Post reply on HN