Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 3 Deep Think

blog.google

Post reply on HN
Print view View original

671–680 of 722 posts

Re: Gemini 3 Deep Think

#671

Post by grantcas »

modeless wrote: post #101
nubg wrote: post #26

Earlier quoted context omitted.

Weren't we barely scraping 1-10% on this with state of the art models a year ago and it was considered that this is the final boss, ie solve this and its almost AGI-like? I ask because I cannot distinguish all the benchmarks by heart.

François Chollet, creator of ARC-AGI, has consistently said that solving the benchmark does not mean we have AGI. It has always been meant as a stepping stone to encourage progress in the correct direction rather than as an indicator of reaching the destination. That's why he is working on ARC-AGI-3 (to be released in a few weeks) and ARC-AGI-4. His definition of reaching AGI, as I understand it, is when it becomes i…

[dead]

Re: Gemini 3 Deep Think

#672

Post by ky3 »

blinding-streak wrote: post #543
sinuhe69 wrote: post #38

I'm pretty certain that DeepMind (and all other labs) will try their frontier (and even private) models on First Proof [1]. And I wonder how Gemini Deep Think will fare. My guess is that it will get half the way on some problems. But we will have to take an absence as a failure, because nobody wants to publish a negative result, even though it's so important for scientific research. [1] https://1stproof.org/

As a non-mathematician, reading these problems feels like reading a completely foreign language. https://arxiv.org/html/2602.05192v1

LLM to the rescue. Feed in a problem and ask it to explain it to a layperson. Also feed in sentences that remain obscure and ask to unpack.

Re: Gemini 3 Deep Think

#673

Post by S1M0N38-hn »

brokensegue wrote: post #670
S1M0N38-hn wrote: post #383

Earlier quoted context omitted.

Hi, BalatroBench creator here. Yeah, Google models perform well (I guess the long context + world knowledge capabilities). Opus 4.6 looks good on preliminary results (on par with Gemini 3 Pro). I'll add more models and report soon. Tbh, I didn't expect LLMs to start winning runs. I guess I have to move to harder stakes (e.g. red stake).

Did you consider doing it as a computer use task? Probably I find those more compelling It's what I did for my game benchmark https://d.erenrich.net/paperclip-bench/index.html

not really. I've downloaded balatro. I saw that it was moddable. I wrote a mod API to interact programmatically. I was just curious if, from text only game state representation, a LLM would be able to make some decent play. the benchmark was a late pivoting.

Re: Gemini 3 Deep Think

#674

Post by ky3 »

LightBug1 wrote: post #666

But it can't parse my mathematically really basic personal financial spreadsheet ... I learned a lot about Gemini last night. Namely that I have lead it like a reluctant bull to understand what I want it to do (beyond normal conversations, etc). Don't get me wrong, ChatGPT didn't do any better. It's an important spreadsheet so I'm triple checking on several LLM's and, of course, comparing results with my own in depth…

You can ask the LLM to write a prompt for you. Example: "Explore prompts that would have circumvented all the previous misunderstanding."

Re: Gemini 3 Deep Think

#675

Post by S1M0N38-hn »

raincole wrote: post #437
S1M0N38-hn wrote: post #383

Earlier quoted context omitted.

Hi, BalatroBench creator here. Yeah, Google models perform well (I guess the long context + world knowledge capabilities). Opus 4.6 looks good on preliminary results (on par with Gemini 3 Pro). I'll add more models and report soon. Tbh, I didn't expect LLMs to start winning runs. I guess I have to move to harder stakes (e.g. red stake).

Thank you for the site! I've got a few suggestions: 1. I think winrate is more telling than the average round number. 2. Some runs are bugged (like Gemini's run 9) and should be excluded from the result. Selling Invisible Joker is always bugged, rendering all the runs with the seed EEEEEE invalid. 3. Instead of giving them "strategy" like "flush is the easiest hand..." it's fairer to clarify some mechanisms that conf…

Im pretty open to feedback and contribution (also regarding the default strategy). So feel free to open Issues on GH. However I'd like to collect a bunch of them (including bugs) before re-running the whole benchmark (balatrobench v2).

Re: Gemini 3 Deep Think

#676

Post by olyjohn »

CamperBob2 wrote: post #439
imiric wrote: post #374

Earlier quoted context omitted.

> What evidence of intelligence would satisfy you? That is a loaded question. It presumes that we can agree on what intelligence is, and that we can measure it in a reliable way. It is akin to asking an atheist the same about God. The burden of proof is on the claimer. The reality is that we can argue about that until we're blue in the face, and get nowhere. In this case it would be more productive to talk about the…

(Shrug) Unless and until you provide us with your own definition of intelligence, I'd say the marketing people are as entitled to their opinion as you are.

I would say that marketing people have a motivation to make exaggerated claims, while the rest of us are trying to just come up with a definition that makes sense and helps us understand the world.

I'll give you some examples. "Unlimited" now has limits on it. "Lifetime" means only for so many years. "Fully autonomous" now means with the help of humans on occasion. These are all definitions that have been distorted by marketers, which IMO is deceptive and immoral.

Re: Gemini 3 Deep Think

#677

Post by vessenes »

modeless wrote: post #653
vessenes wrote: post #649

Earlier quoted context omitted.

Please let’s hold M Chollet to account, at least a little. He launched ARC claiming transformer architectures could never do it and that he thought solving it would be AGI. And he was smug about it. ARC 2 had a very similar launch. Both have been crushed in far less time without significantly different architectures than he predicted. It’s a hard test! And novel, and worth continuing to iterate on. But it was not lau…

Here is what the original paper for ARC-AGI-1 said in 2019: > Our definition, formal framework, and evaluation guidelines, which do not capture all facets of intelligence, were developed to be actionable, explanatory, and quantifiable, rather than being descriptive, exhaustive, or consensual. They are not meant to invalidate other perspectives on intelligence, rather, they are meant to serve as a useful objective fun…

https://www.dwarkesh.com/p/francois-chollet (June 2024, about ARC-AGI-1. Note the AGI right in the name)

> I’m pretty skeptical that we’re going to see an LLM do 80% in a year. That said, if we do see it, you would also have to look at how this was achieved. If you just train the model on millions or billions of puzzles similar to ARC, you’re relying on the ability to have some overlap between the tasks that you train on and the tasks that you’re going to see at test time. You’re still using memorization.

> Maybe it can work. Hopefully, ARC is going to be good enough that it’s going to be resistant to this sort of brute force attempt but you never know. Maybe it could happen. I’m not saying it’s not going to happen. ARC is not a perfect benchmark. Maybe it has flaws. Maybe it could be hacked in that way.

e.g. If ARC is solved not through memorization, then it does what it says on the tin.

[Dwarkesh suggests that larger models get more generalization capabilities and will therefore continue to become more intelligent]

> If you were right, LLMs would do really well on ARC puzzles because ARC puzzles are not complex. Each one of them requires very little knowledge. Each one of them is very low on complexity. You don't need to think very hard about it. They're actually extremely obvious for human

> Even children can do them but LLMs cannot. Even LLMs that have 100,000x more knowledge than you do still cannot.

If you listen to the podcast, he was super confident, and super wrong. Which, like I said, NBD. I'm glad we have the ARC series of tests. But they have "AGI" right in the name of the test.

Re: Gemini 3 Deep Think

#678

Post by goodmythical »

lm28469 wrote: post #225
brokencode wrote: post #212

Earlier quoted context omitted.

Who said they’re godlike today? And yes, you are probably using them wrong if you don’t find them useful or don’t see the rapid improvement.

Let's come back in 12 months and discuss your singularity then. Meanwhile I spent like $30 on a few models as a test yesterday, none of them could tell me why my goroutine system was failing, even though it was painfully obvious (I purposefully added one too many wg.Done), gemini, codex, minimax 2.5, they all shat the bed on a very obvious problem but I am to believe they're 98% conscious and better at logic and math…

I think you're being awfully generous to the average human.

Consider that a nonzero percent of otherwise competent adults can't write in their native language.

Consider that some tens of percentage of people wouldn't have the foggiest idea of how to calculate a square root let alone a cube.

Consider that well less than half of the population has ever seen code let alone produced functioning code.

The average adult is strikingly incapable of things that the average commenter here would consider basic skills.

Re: Gemini 3 Deep Think

#679

Post by peheje »

modeless wrote: post #101
nubg wrote: post #26

Earlier quoted context omitted.

Weren't we barely scraping 1-10% on this with state of the art models a year ago and it was considered that this is the final boss, ie solve this and its almost AGI-like? I ask because I cannot distinguish all the benchmarks by heart.

François Chollet, creator of ARC-AGI, has consistently said that solving the benchmark does not mean we have AGI. It has always been meant as a stepping stone to encourage progress in the correct direction rather than as an indicator of reaching the destination. That's why he is working on ARC-AGI-3 (to be released in a few weeks) and ARC-AGI-4. His definition of reaching AGI, as I understand it, is when it becomes i…

Hello Gemini, please fix:

Biological Aging: Find the cellular "reset switch" so humans can live indefinitely in peak physical health.

Global Hunger: Engineer a food system where nutritious meals are a universal right and never a scarcity.

Cancer: Develop a precision "search and destroy" therapy that eliminates every malignant cell without side effects.

War: Solve the systemic triggers of conflict to transition humanity into an era of permanent global peace.

Chronic Pain: Map the nervous system to shut off persistent physical suffering for every person on Earth.

Infectious Disease: Create a universal shield that detects and neutralizes any pathogen before it can spread.

Clean Energy: Perfect nuclear fusion to provide the world with limitless, carbon-free power forever.

Mental Health: Unlock the brain's biology to fully cure depression, anxiety, and all neurological disorders.

Clean Water: Scale low-energy desalination so that safe, fresh water is available in every corner of the globe.

Ecological Collapse: Restore the Earth’s biodiversity and stabilize the climate to ensure a thriving, permanent biosphere.

Re: Gemini 3 Deep Think

#680

Post by glenstein »

mapontosevenths wrote: post #616
glenstein wrote: post #444

Earlier quoted context omitted.

Agreed, it's a truly wild take. While I fully support the humility of not knowing, at a minimum I think we can say determinations of consciousness have some relation to specific structure and function that drive the outputs, and the actual process of deliberating on whether there's consciousness would be a discussion that's very deep in the weeds about architecture and processes. What's fascinating is that evolution…

> at a minimum I think we can say determinations of consciousness have some relation to specific structure and function that drive the outputs Every time anyone has tried that it excludes one or more classes of human life, and sometimes led to atrocities. Let's just skip it this time.

Having trouble parsing this one. Is it meant to be a WWII reference? If anything I would say consciousness research has expanded our understanding of living beings understood to be conscious.

And I don't think it's fair or appropriate to treat study of the subject matter of consciousness like it's equivalent to 20th century authoritarian regimes signing off on executions. There's a lot of steps in the middle before you get from one to the other that distinguish them to the extent necessary and I would hope that exercise shouldn't be necessary every time consciousness research gets discussed.

Post reply on HN