Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3

blog.google

Post reply on HN
Print view View original

681–690 of 1001 posts

Re: Gemini 3

#681

Post by kenjackson »

rbjorklin wrote: post #519
lairv wrote: post #293

Out of curiosity, I gave it the latest project euler problem published on 11/16/2025, very likely out of the training data Gemini thought for 5m10s before giving me a python snippet that produced the correct answer. The leaderboard says that the 3 fastest human to solve this problem took 14min, 20min and 1h14min respectively Even thought I expect this sort of problem to very much be in the distribution of what the mo…

Your post made me curious to try a problem I have been coming back to ever since ChatGPT was first released: https://open.kattis.com/problems/low I have had no success using LLM's to solve this particular problem until trying Gemini 3 just now despite solutions to it existing in the training data. This has been my personal litmus test for testing out LLM programming capabilities and a model finally passed.

ChatGPT solves this problem now as well with 5.1. Time for a new litmus test.

Re: Gemini 3

#682

Post by udkl »

SXX wrote: post #366

Static Pelican is boring. First attempt: Generate SVG animation of following: 1 - There is High fantasy mage tower with a top window a dome 2 - Green goblin come in front of tower with a torch 3 - Grumpy old mage with beard appear in a tower window in high purple hat 4 - Mage sends fireball that burns goblin and all screen is covered in fire. Camera view must be from behind of goblin back so we basically look at towe…

This reminded me of https://youtube.com/playlist?list=PLSq76P-lbX8VQmtv7gcAPkqlj...

Re: Gemini 3

#683

Post by energy123 »

riku_iki wrote: post #649
stephc_int13 wrote: post #305

Earlier quoted context omitted.

What I would do if I was in the position of a large company in this space is to arrange an internal team to create an ARC replica, covering very similar puzzles and use that as part of the training. Ultimately, most benchmarks can be gamed and their real utility is thus short-lived. But I think this is also fair to use any means to beat it.

> internal team to create an ARC replica, covering very similar puzzles they can target benchmark directly, not just replica. If google or OAI are bad actors, they already have benchmark data from previous runs.

The 'private' set is just a pinkie promise not to store logs or not to use the logs when the evaluator uses the API to run the test, so yeah. It's trivially exploitable.

Not only do you have the financial self-interest to do it (helps with capital raising to be #1), but you are worried that your competitors are doing it, so you may as well cheat to make things fair. Easy to do and easy to justify.

Maybe a way to make the benchmark more robust to this adversarial environment is to introduce noise and random red herrings into the question, and run the test 20 times and average the correctness. So even if you assume they're training on it, you have some semblance of a test still happening. You'd probably end up with a better benchmark anyway which better reflects real-world usage, where there's a lot of junk in the context window.

Re: Gemini 3

#684

Post by kldg »

dwringer wrote: post #236

Well, I tried a variation of a prompt I was messing with in Flash 2.5 the other day in a thread about AI-coded analog clock faces. Gemini Pro 3 Preview gave me a result far beyond what I saw with Flash 2.5, and got it right in a single shot.[0] I can't say I'm not impressed, even though it's a pretty constrained example. > Please generate an analog clock widget, synchronized to actual system time, with hands that upd…

in defense of 2.5 (Pro, at least), it was able to generate for me a metric UNIX clock as a webpage which I was amused by. it uses kiloseconds/megaseconds/etc. there are 86.4ks/day. The "seconds" hand goes around 1000 seconds, which ticks over the "hour" hand. Instead of saying 4am, you'd say it's 14.

as a calendar or "date" system, we start at UNIX time's creation, so it's currently 1.76 gigaseconds AUNIX. You might use megaseconds as the "week" and gigaseconds more like an era, e.g. Queen Elizabeth III's reign, persisting through the entire fourth gigasecond and into the fifth. The clock also displays teraseconds, though this is just a little purple speck atm. of course, this can work off-Earth where you would simply use 88.775ks as the "day"; the "dates" a Martian and Earthling share with each other would be interchangeable.

I can't seem to get anyone interested in this very serious venture, though... I guess I'll have to wait until the 50th or so iteration of Figure, whenever it becomes useful, to be able to build a 20-foot-tall physical metric UNIX clock in my front yard.

Re: Gemini 3

#685

Post by rhetocj23 »

sebzim4500 wrote: post #383
bigstrat2003 wrote: post #317

Earlier quoted context omitted.

No LLM has ever been as good as people said it was. That doesn't mean this one won't be, but it does make it an unlikely bet based on past trends.

With the exception of GPT-5, which was a significant advance yet because it was slightly less sycophantic than gpt-4o the internet decided it was terrible for the first few days.

[dead]

Re: Gemini 3

#686

Post by tonyhart7 »

aussieguy1234 wrote: post #652
tonyhart7 wrote: post #645

Earlier quoted context omitted.

after outsource developer job, we can outsource all of manager job and leaving CEO with AI agentic code as its servant

Not sure what you mean here, but the only real jobs at risk from AI right now are middle/upper management. Not a single engineer has ever been laid off because of AI. Any company claiming this is the case is trying to cover up bad decisions. "Were automating with AI" sounds better to investors than "We over hired and now need to downsize" or "We made some bad market bets, now need to free up cash flow"

"Not a single engineer has ever been laid off because of AI."

are you insane??? big tech literally make one of the most biggest layoff for the past few months

Re: Gemini 3

#687

Post by hamasho »

I just googled latest LLM models and this page appears at the top. It looks like Gemini Pro 3 can score 102% in high school math tests.

Re: Gemini 3

#688

Post by bgwalter »

lairv wrote: post #293

Out of curiosity, I gave it the latest project euler problem published on 11/16/2025, very likely out of the training data Gemini thought for 5m10s before giving me a python snippet that produced the correct answer. The leaderboard says that the 3 fastest human to solve this problem took 14min, 20min and 1h14min respectively Even thought I expect this sort of problem to very much be in the distribution of what the mo…

Does it matter if it is out of the training data? The models integrate web search quite well.

What if they have an internal corpus of new and curated knowledge that is constantly updated by humans and accessed in a similar manner? It could be active even if web search is turned off.

They would surely add the latest Euler problems with solutions in order to show off in benchmarks.

Re: Gemini 3

#689

Post by riku_iki »

energy123 wrote: post #683
riku_iki wrote: post #649

Earlier quoted context omitted.

> internal team to create an ARC replica, covering very similar puzzles they can target benchmark directly, not just replica. If google or OAI are bad actors, they already have benchmark data from previous runs.

The 'private' set is just a pinkie promise not to store logs or not to use the logs when the evaluator uses the API to run the test, so yeah. It's trivially exploitable. Not only do you have the financial self-interest to do it (helps with capital raising to be #1), but you are worried that your competitors are doing it, so you may as well cheat to make things fair. Easy to do and easy to justify. Maybe a way to make…

they have two sets:

- semi-private, which they use to test proprietary models and which could be leaked

-private: used to test downloadable open source models.

ARG-AGI prize itself is for open source models.

Re: Gemini 3

#690

Post by jpkw »

Hoping someone here may know the answer to this, but do any of the benchmarks that exist currently account for false answers in any meaningful way, other than it would in a typical test (ie, if I give any answer at all it is better than saying "I don't know" as the answer I give at least has a chance of being correct(which in the real world is bad))? I want an LLM that tells me when it doesn't know something. If it gives me an accurate response 90% of the time and an inaccurate one 10% of the time, it is less useful than one that gives me an accurate answer 10% of the time and tells me "I don't know" the other 90%.
Post reply on HN