Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Pro: the frontier of vision AI

blog.google

Post reply on HN
Print view View original

111–120 of 309 posts

Re: Gemini 3 Pro: the frontier of vision AI

#111

Post by CamperBob2 »

irthomasthomas wrote: post #87
Rover222 wrote: post #64

Earlier quoted context omitted.

I just tried to get Gemini to produce an image of a dog with 5 legs to test this out, and it really struggled with that. It either made a normal dog, or turned the tail into a weird appendage. Then I asked both Gemini and Grok to count the legs, both kept saying 4. Gemini just refused to consider it was actually wrong. Grok seemed to have an existential crisis when I told it it was wrong, becoming convinced that I ha…

Isn't this proof that LLMs still don't really generalize beyond their training data?

They do, but we call it "hallucination" when that happens.

Re: Gemini 3 Pro: the frontier of vision AI

#112

Post by jvolkman »

Workaccount2 wrote: post #48

Well It is the first model to get partial-credit on an LLM image test I have. Which is counting the legs of a dog. Specifically, a dog with 5 legs. This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. In fact GPT5 wrote an edge detection script to see where "golden dog feet" met "bright green grass" to prove to me that there were only 4 legs. The script found 5, and GPT-5 the…

Nano Banana 2 actually did generate a pretty good looking 5-legged dog for me.

https://gemini.google.com/share/e7a8b902ff67

But it didn't actually think it had done so, aligning with your observations. The last bits of its thinking were pretty sad (for it):

Analyzing Missing Anomaly

I'm focusing on the discrepancy - the persistent absence of the fifth leg in the generated images. Despite multiple attempts, the output consistently depicts a four-legged dog, contrary to the explicit requirement. This ongoing failure necessitates a shift in approach.

Acknowledge Leg Omission

I've made a final check, and unfortunately, the image still lacks the anomaly. The consistent absence of the fifth leg necessitates admitting my limitation. I'll provide the best image I could generate, clearly stating the missing feature.

Re: Gemini 3 Pro: the frontier of vision AI

#113

Post by CamperBob2 »

runarberg wrote: post #107
Benjammer wrote: post #83

Earlier quoted context omitted.

It always feels to me like these types of tests are being somewhat intentionally ignorant of how LLM cognition differs from human cognition. To me, they don't really "prove" or "show" anything other than simply - LLMs thinking works different than human thinking. I'm always curious if these tests have comprehensive prompts that inform the model about what's going on properly, or if they're designed to "trick" the LLM…

This is the first time I hear the term LLM cognition and I am horrified. LLMs don‘t have cognition. LLMs are a statistical inference machines which predict a given output given some input. There are no mental processes, no sensory information, and certainly no knowledge involved, only statistical reasoning, inference, interpolation, and prediction. Comparing the human mind to an LLM model is like comparing a rubber t…

You'll need to explain the IMO results, then.

Re: Gemini 3 Pro: the frontier of vision AI

#114

Post by semi-extrinsic »

reducesuffering wrote: post #81
knollimar wrote: post #33

I do some electrical drafting work for construction and throw basic tasks at LLMs. I gave it a shitty harness and it almost 1 shotted laying out outlets in a room based on a shitty pdf. I think if I gave it better control it could do a huge portion of my coworkers jobs very soon

"AI could never replace the creativity of a human" "Ok, I guess it could wipe out the economic demand for digital art, but it could never do all the autonomous tasks of a project manager" "Ok, I guess it could automate most of that away but there will always be a need for a human engineer to steer it and deal with the nuances of code" "Ok, well it could never automate blue collar work, how is it gonna wrench a pipe i…

> Remember when the Turing test was a thing? No one seems to remember it was considered serious in 2020

To be clear, it's only ever been a pop science belief that the Turing test was proposed as a literal benchmark. E.g. Chomsky in 1995 wrote:

  The question “Can machines think?” is not a question of fact but one of language, and Turing himself observed that the question is 'too meaningless to deserve discussion'.

Re: Gemini 3 Pro: the frontier of vision AI

#115

Post by Fraterkes »

reducesuffering wrote: post #81
knollimar wrote: post #33

I do some electrical drafting work for construction and throw basic tasks at LLMs. I gave it a shitty harness and it almost 1 shotted laying out outlets in a room based on a shitty pdf. I think if I gave it better control it could do a huge portion of my coworkers jobs very soon

"AI could never replace the creativity of a human" "Ok, I guess it could wipe out the economic demand for digital art, but it could never do all the autonomous tasks of a project manager" "Ok, I guess it could automate most of that away but there will always be a need for a human engineer to steer it and deal with the nuances of code" "Ok, well it could never automate blue collar work, how is it gonna wrench a pipe i…

The turing test is still a thing. No llm could pass for a person for more than a couple minutes of chatting. That’s a world of difference compared to a decade ago, but I would emphatically not call that “passing the turing test”

Also, none of the other things you mentioned have actually happened. Don’t really know why I bother responding to this stuff

Re: Gemini 3 Pro: the frontier of vision AI

#116

Post by Benjammer »

biophysboy wrote: post #88
Benjammer wrote: post #83

Earlier quoted context omitted.

It always feels to me like these types of tests are being somewhat intentionally ignorant of how LLM cognition differs from human cognition. To me, they don't really "prove" or "show" anything other than simply - LLMs thinking works different than human thinking. I'm always curious if these tests have comprehensive prompts that inform the model about what's going on properly, or if they're designed to "trick" the LLM…

I thought adversarial testing like this was a routine part of software engineering. He's checking to see how flexible it is. Maybe prompting would help, but it would be cool if it was more flexible.

So the idea is what? What's the successful outcome look like for this test, in your mind? What should good software do? Respond and say there are 5 legs? Or question what kind of dog this even is? Or get confused by a nonsensical picture that doesn't quite match the prompt in a confusing way? Should it understand the concept of a dog and be able to tell you that this isn't a real dog?

Re: Gemini 3 Pro: the frontier of vision AI

#117

Post by averageRoyalty »

joaomacp wrote: post #109
Workaccount2 wrote: post #48

Well It is the first model to get partial-credit on an LLM image test I have. Which is counting the legs of a dog. Specifically, a dog with 5 legs. This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. In fact GPT5 wrote an edge detection script to see where "golden dog feet" met "bright green grass" to prove to me that there were only 4 legs. The script found 5, and GPT-5 the…

And just like that, you no longer have a good benchmark. Scrapers / AI developers will read this comment, and add 5-legged dogs to LLM's training data.

That's okay. Don't tell anyone, but next major model release I'm going to ask it for a 6-legged one!

Re: Gemini 3 Pro: the frontier of vision AI

#118

Post by bee_rider »

Workaccount2 wrote: post #48

Well It is the first model to get partial-credit on an LLM image test I have. Which is counting the legs of a dog. Specifically, a dog with 5 legs. This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. In fact GPT5 wrote an edge detection script to see where "golden dog feet" met "bright green grass" to prove to me that there were only 4 legs. The script found 5, and GPT-5 the…

Naive question, but what is Gemini?

I wonder if a lot of these models are large language models that have had image recognition and generation tools bolted on? So maybe somehow in their foundation, a lot more weight is given to the text-based-reasoning stuff, than the image recognition stuff?

Re: Gemini 3 Pro: the frontier of vision AI

#119

Post by JamesSwift »

danielvaughn wrote: post #57
Workaccount2 wrote: post #48

Well It is the first model to get partial-credit on an LLM image test I have. Which is counting the legs of a dog. Specifically, a dog with 5 legs. This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. In fact GPT5 wrote an edge detection script to see where "golden dog feet" met "bright green grass" to prove to me that there were only 4 legs. The script found 5, and GPT-5 the…

I don’t know much about AI, but I have this image test that everything has failed at. You basically just present an image of a maze and ask the LLM to draw a line through the most optimal path. Here’s how Nano Banana fared: https://x.com/danielvaughn/status/1971640520176029704?s=46

I just oneshot it with claude code (opus 4.5) using this prompt. It took about 5 mins and included detecting that it was cheating at first (drew a line around the boundary of the maze instead), so it added guardrails for that:

```

Create a devenv project that does the following:

  - Read the image at maze.jpg
  - Write a script that solves the maze  in the most optimal way between the mouse and the cheese
  - Generate a new image which is of the original maze, but with a red line that represents the calculated path
Use whatever lib/framework is most appropriate

```

  Output: https://gist.github.com/J-Swift/ceb1db348f46ba167948f734ff0fc604  
  Solution: https://imgur.com/a/bkJloPT

Re: Gemini 3 Pro: the frontier of vision AI

#120

Post by Benjammer »

runarberg wrote: post #107
Benjammer wrote: post #83

Earlier quoted context omitted.

It always feels to me like these types of tests are being somewhat intentionally ignorant of how LLM cognition differs from human cognition. To me, they don't really "prove" or "show" anything other than simply - LLMs thinking works different than human thinking. I'm always curious if these tests have comprehensive prompts that inform the model about what's going on properly, or if they're designed to "trick" the LLM…

This is the first time I hear the term LLM cognition and I am horrified. LLMs don‘t have cognition. LLMs are a statistical inference machines which predict a given output given some input. There are no mental processes, no sensory information, and certainly no knowledge involved, only statistical reasoning, inference, interpolation, and prediction. Comparing the human mind to an LLM model is like comparing a rubber t…

>They belong in different categories

Categories of _what_, exactly? What word would you use to describe this "kind" of which LLMs and humans are two very different "categories"? I simply chose the word "cognition". I think you're getting hung up on semantics here a bit more than is reasonable.

Post reply on HN