Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3

blog.google

Post reply on HN
Print view View original

541–550 of 1001 posts

Re: Gemini 3

#541

Post by brabel »

ApolloFortyNine wrote: post #525
ThrowawayR2 wrote: post #151

Earlier quoted context omitted.

They've poisoned the internet with their monopoly on advertising, the air pollution of the online world, which is an transgression that far outweighs any good they might have done. Much of the negative social effects of being online come from the need to drive more screen time, more engagement, more clicks, and more ad impressions firehosed into the faces of users for sweet, sweet, advertiser money. When Google final…

People love getting their content for free and that's what Google does. Even 25 years ago people wouldn't even believe Youtube exists. Anyone can upload whatever they want, however often they want, Youtube will be responsible for promoting it, they'll provide to however many billions users want to view it, and they'll pay you 55% of the revenue it makes?

Yep, it's hard to believe it exists for free and with not a lot of ads when you have a good ad blocker... though the content creator's ads are inescapable, which I think is ok since they're making a little money in exchange for what, your little inconvenience for 1 minute or so - if you're not skipping the ad, which you aren't, right??) - after which you can watch some really good content. The history channels on YT are amazing, maybe world changing - they get people to learn history and actually enjoy it. Same with some match channels like 3brown1blue which are just outstanding, and many more.

Re: Gemini 3

#542

Post by libraryofbabel »

simonw wrote: post #487

Here are my notes and pelican benchmark, including a new, harder benchmark because the old one was getting too easy: https://simonwillison.net/2025/Nov/18/gemini-3/

I was interested (and slightly disappointed) to read that the knowledge cutoff for Gemini 3 is the same as for Gemini 2.5: January 2025. I wonder why they didn't train it on more recent data.

Is it possible they use the same base pre-trained model and just fine-tuned and RL-ed it better (which, of course, is where all the secret sauce training magic is these days anyhow)? That would be odd, especially for a major version bump, but it's sort of what having the same training cutoff points to?

Re: Gemini 3

#544

Post by apetresc »

stalfie wrote: post #334
dwringer wrote: post #236

Well, I tried a variation of a prompt I was messing with in Flash 2.5 the other day in a thread about AI-coded analog clock faces. Gemini Pro 3 Preview gave me a result far beyond what I saw with Flash 2.5, and got it right in a single shot.[0] I can't say I'm not impressed, even though it's a pretty constrained example. > Please generate an analog clock widget, synchronized to actual system time, with hands that upd…

The subtle "wiggle" animation that the second hand makes after moving doesn't fire when it hits 12. Literally unwatchable.

In its defence, the code actually specifically calls that edge case out and justifies it:

    // Calculate rotations
    // We use a cumulative calculation logic mentally, but here simple degrees work because of the transition reset trick or specific animation style.
    // To prevent the "spin back" glitch at 360->0, we can use a simple tick without transition for the wrap-around,
    // but for simplicity in this specific React rendering, we will stick to standard 0-360 degrees.
    // A robust way to handle the spin-back on the second hand is to accumulate degrees, but standard clock widgets often reset.

Re: Gemini 3

#545

Post by stephc_int13 »

AstroBen wrote: post #536
stephc_int13 wrote: post #305

Earlier quoted context omitted.

What I would do if I was in the position of a large company in this space is to arrange an internal team to create an ARC replica, covering very similar puzzles and use that as part of the training. Ultimately, most benchmarks can be gamed and their real utility is thus short-lived. But I think this is also fair to use any means to beat it.

Is "good at benchmarks instead of real world tasks" really something to optimize for? What does this achieve? Surely people would be initially impressed, try it out, be underwhelmed and then move on. That's not great for Google

Benchmarks are intended as proxy for real usage, and they are often useful to incrementally improve a system, especially when the end-goal is not well-defined.

The trick is to not put more value in the score than what it is.

Re: Gemini 3

#546

Post by chermi »

prodigycorp wrote: post #421
dekhn wrote: post #149

Earlier quoted context omitted.

Using a single custom benchmark as a metric seems pretty unreliable to me. Even at the risk of teaching future AI the answer to your benchmark, I think you should share it here so we can evaluate it. It's entirely possible you are coming to a wrong conclusion.

after taking a walk for a bit i decided you’re right. I came to the wrong conclusion. Gemini 3 is incredibly powerful in some other stuff I’ve run. This probably means my test is a little too niche. The fact that it didn’t pass one of my tests doesn’t speak to the broader intelligence of the model per se. While i still believe in the importance of a personalized suite of benchmarks, my python one needs to be down wei…

Walks are magical. But also this reads partially like you got sent to a reeducation camp lol.

Re: Gemini 3

#548

Post by vlmrun-admin »

https://www.youtube.com/watch?v=cUbGVH1r_1U

Everyone is talking about the release of Gemini 3. The benchmark scores are incredible. But as we know in the AI world, paper stats don't always translate to production performance on all tasks.

We decided to put Gemini 3 through its paces on some standard Vision Language Model (VLM) tasks – specifically simple image detection and processing.

The result? It struggled where I didn't expect it to.

Surprisingly, VLM Run's Orion (https://chat.vlm.run/) significantly outperformed Gemini 3 on these specific visual tasks. While the industry chases the "biggest" model, it’s a good reminder that specialized agents like Orion are often punching way above their weight class in practical applications.

Has anyone else noticed a gap between Gemini 3's benchmarks and its VLM capabilities?

Re: Gemini 3

#549

Post by stephc_int13 »

tylervigen wrote: post #313
stephc_int13 wrote: post #305

Earlier quoted context omitted.

What I would do if I was in the position of a large company in this space is to arrange an internal team to create an ARC replica, covering very similar puzzles and use that as part of the training. Ultimately, most benchmarks can be gamed and their real utility is thus short-lived. But I think this is also fair to use any means to beat it.

I agree that for any given test, you could build a specific pipeline to optimize for that test. I supposed that's why it is helpful to have many tests. However, many people have worked hard to optimize tools specifically for ARC over many years, and it's proven to be a particularly hard test to optimize for. This is why I find it so interesting that LLMs can do it well at all, regardless of whether tests like it are…

The real strength of current neural nets/transformers relies on huge datasets.

ARC do not provide this kind of dataset, only a small public one and a private one where they do the benchmarks.

Building your own large private ARC set does not seem too difficult if you have enough resources.

Re: Gemini 3

#550

Post by skylurk »

simonw wrote: post #487

Here are my notes and pelican benchmark, including a new, harder benchmark because the old one was getting too easy: https://simonwillison.net/2025/Nov/18/gemini-3/

They've been training for months to draw that pelican, just for you to move the goalposts.
Post reply on HN