Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3

blog.google

Post reply on HN
Print view View original

911–920 of 1001 posts

Re: Gemini 3

#911

Post by mpalmer »

MaxL93 wrote: post #362
siva7 wrote: post #237

I have my own private benchmarks for reasoning capabilities on complex problems and i test them against SOTA models regularly (professional cases from law and medicine). Anthropic (Sonnet 4.5 Extended Thinking) and OpenAI (Pro Models) get halfway decent results on many cases while Gemini Pro 2.5 struggled (it was overconfident in its initial assumptions). So i ran these benchmarks against Gemini 3 Pro and i'm not imp…

> It seems very US centric in its thinking I'm not surprised. I'm French and one thing I've consistently seen with Gemini is that it loves to use Title Case (Everything is Capitalized Except the Prepositions) even in French or other languages where there is no such thing. A 100% american thing getting applied to other languages by the sheer power of statistical correlation (and probably being overtrained on USA-centr…

> Title Case (Everything is Capitalized Except the Prepositions)

If this is an American thing I'm happy to disown/denounce it; it's my least favorite pattern in Gemini output.

Re: Gemini 3

#912

Post by empath75 »

hollowturtle wrote: post #778
SXX wrote: post #443

Earlier quoted context omitted.

After few more attempts longer animation with a story from my gamedev inspired mind: https://codepen.io/Runway/pen/zxqzPyQ PS: but yeah thats attempt #20 or something.

Wow looks like total shit and eventually very hard to take on and actually improve it, given the convoluted code it generated, YET people are impressed. What world are we living in...

When people figure out how to make a computer do something that it couldn't do before, that is interesting and impressive. It doesn't need to be useful.

Re: Gemini 3

#913

Post by agentcoops »

prodigycorp wrote: post #421
dekhn wrote: post #149

Earlier quoted context omitted.

Using a single custom benchmark as a metric seems pretty unreliable to me. Even at the risk of teaching future AI the answer to your benchmark, I think you should share it here so we can evaluate it. It's entirely possible you are coming to a wrong conclusion.

after taking a walk for a bit i decided you’re right. I came to the wrong conclusion. Gemini 3 is incredibly powerful in some other stuff I’ve run. This probably means my test is a little too niche. The fact that it didn’t pass one of my tests doesn’t speak to the broader intelligence of the model per se. While i still believe in the importance of a personalized suite of benchmarks, my python one needs to be down wei…

I definitely agree on the importance of personalized benchmarks for really feeling when, where and how much progress is occurring. The standard benchmarks are important, but it’s hard to really feel what a 5% improvement in X exam means beyond hype. I have a few projects across domains that I’ve been working on since ChatGPT 3 launched and I quickly give them a try on each new model release. Despite popular opinion, I could really tell a huge difference between GPT 4 and 5 , but nothing compared to the current delta between 5.1 and Gemini 3 Pro…

TLDR; I don’t think personal benchmarks should replace the official ones of course, but I think the former are invaluable for building your intuition about the rate of AI progress beyond hype.

Re: Gemini 3

#914

Post by GuB-42 »

Workaccount2 wrote: post #100

It still failed my image identification test ([a photoshopped picture of a dog with 5 legs]...please count the legs) that so far every other model has failed agonizingly, even failing when I tell them they are failing, and they tend to fight back at me. Gemini 3 however, while still failing, at least recognized the 5th leg, but thought the dog was...well endowed. The 5th leg however is clearly a leg, despite being wh…

> Gemini 3 however, while still failing, at least recognized the 5th leg, but thought the dog was...well endowed.

I see that AI is reaching the level of a middle school boy...

Re: Gemini 3

#915

Post by arnorhs »

zarzavat wrote: post #779
davidpolberger wrote: post #692

This is wild. I gave it some legacy XML describing a formula-driven calculator app, and it produced a working web app in under a minute: https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%... I spent years building a compiler that takes our custom XML format and generates an app for Android or Java Swing. Gemini pulled off the same feat in under a minute, with no explanation of the format. The XML is fairly…

This is exactly the kind of task that LLMs are good at. They are good at transforming one format to another. They are good at boilerplate. They are bad at deciding requirements by themselves. They are bad at original research, for example developing a new algorithm.

+/-

> They are bad at deciding requirements by themselves.

What do you mean by requirements here? In my experience the frontier models today are pretty good at figuring out requirements, even when you don't explicitly state them.

> They are bad at original research

Sure, I don't have any experience with that, so I'll trust you on that.

> for example developing a new algorithm.

This is just not correct. I used to think so, but I was trying to come up with a pretty complicated pattern matching, multi-dimensional algorithm (I can't go into the details) - it was something that I could figure out on my own, and was half way through it, but decided to write up a description of it and feed it to gemini 2.5 pro a couple of months ago, and I was stunned.

It came up with a really clever approach and something I had previously been convinced the models weren't very good at it.

In hindsight, since they are getting so good at math in general, there's probably some overlap, but you should revisit your views on this.

--

Your 'bad at' list is missing a few things though:

- Calculations (they can come up with how to calculate or write a program to calculate from given data, but they are not good at calculating in their responses)

- Even though the frontier models are multi-modal, they are still bad at visualizing html/css - or interpreting what it would look like

- Same goes for visualizing/figuring out visual errors in graphics programming such as games programming or 3d modeling (z-index issues, orientation etc)

Re: Gemini 3

#916

Post by gregw2 »

NaomiLehman wrote: post #873
mpalmer wrote: post #716

Earlier quoted context omitted.

Gemini still calls me "the architect" in half of the prompts. It's very cringe. Can't say I've ever seen this in my own chats. Maybe it's something about your writing style?

it absolutely does. and human employees don't call me "the architect." that's the point.

I wonder if under the covers it uses your word choices to infer your Myers-Briggs personality type and you are INTJ so it calls you "The Architect"?? Crazy thought but conceivable...

Re: Gemini 3

#918

Post by tim333 »

calmoo wrote: post #889
panarky wrote: post #697

Earlier quoted context omitted.

[flagged]

Your comment is AI generated

I'll give you the style is like an LLM but the thoughts seem a bit unlike one. I mean the MathArena Apex results indicating a new discovery rather than more data is definitely a hypothesis.

Also panarky denies it.

Re: Gemini 3

#919

Post by abixb »

Okay, Gemini 3.0 Pro has officially surpassed Claude 4.5 (and GPT-5.1) as the top ranked model based on my private evals (multimodal reasoning w/ images/audio files and solving complex Caesar/transposition ciphers, etc.).

Claude 4.5 solved it as well (the Caesar/transposition ciphers), but Gemini 3.0 Pro's method and approach was a lot more elegant. Just my $0.02.

Re: Gemini 3

#920

Post by tngranados »

column wrote: post #863
Workaccount2 wrote: post #100

It still failed my image identification test ([a photoshopped picture of a dog with 5 legs]...please count the legs) that so far every other model has failed agonizingly, even failing when I tell them they are failing, and they tend to fight back at me. Gemini 3 however, while still failing, at least recognized the 5th leg, but thought the dog was...well endowed. The 5th leg however is clearly a leg, despite being wh…

"[a photoshopped picture of a dog with 5 legs]...please count the legs" Meanwhile you could benchmark for something actually useful. If you're about to say "But that means it won't work for my use case of identifying a person on a live feed" or whatever, then why don't you test that? I really don't understand the kick people get of successfully tricking LLMs on non productive task with no real world application. Just…

The point of benchmarking that is checking for hallucinations and overfitting. Does the model actually check the picture to count the legs or does it just see it's a dog and answer four because it knows dogs usually has four legs?

It's a perfectly valid benchmark and very telling.

Post reply on HN