Live data from Hacker News

Why I don't think AGI is imminent

dlants.me

61–70 of 313 posts

Re: Why I don't think AGI is imminent

#61
> The transformer architectures powering current LLMs are strictly feed-forward.

This is true in a specific contextual sense (each token that an LLM produces is from a feed-forward pass). But untrue for more than a year with reasoning models, who feed their produced tokens back as inputs, and whose tuning effectively rewards it for doing this skillfully.

Heck, it was untrue before that as well, any time an LLM responded with more than one token.

> A [March] 2025 survey by the Association for the Advancement of Artificial Intelligence (AAAI), surveying 475 AI researchers, found that 76% believe scaling up current AI approaches to achieve AGI is "unlikely" or "very unlikely" to succeed.

I dunno. This survey publication was from nearly a year ago, so the survey itself is probably more than a year old. That puts us at Sonnet 3.7. The gap between that and present day is tremendous.

I am not skilled enough to say this tactfully, but: expert opinions can be the slowest to update on the news that their specific domain may have, in hindsight, have been the wrong horse. It's the quote about it being difficult to believe something that your income requires to be false, but instead of income it can be your whole legacy or self concept. Way worse.

> My take is that research taste is going to rely heavily on the short-duration cognitive primitives that the ARC highlights but the METR metric does not capture.

I don't have an opinion on this, but I'd like to hear more about this take.

Re: Why I don't think AGI is imminent

#62
post #11

How will we know if its AGI/Not AGI? (I don't think a simple app is gonna cut it here haha) What is the benchmark now that the Turing test has been blown out of the water?

Until recently, philosophy of artificial intelligence seemed to be mostly about arguments why the Turing test was not a useful benchmark for intelligence. Pretty much everyone who had ever thought about the problem seriously had come to the same conclusion.

The fundamental issue was the assumption that general intelligence is an objective property that can be determined experimentally. It's better to consider intelligence an abstraction that may help us to understand the behavior of a system.

A system where a fixed LLM provides answers to prompts is little more than a Chinese room. If we give the system agency to interact with external systems on its own initiative, we get qualitatively different behavior. The same happens if we add memory that lets the system scale beyond the fixed context window. Now we definitely have some aspects of general intelligence, but something still seems to be missing.

Current AIs are essentially symbolic reasoning systems that rely on a fixed model to provide intuition. But the system never learns. It can't update its intuition based on its experiences.

Maybe the ability to learn in a useful way is the final obstacle on the way towards AGI. Or maybe once again, once we start thinking we are close to solving intelligence, we realize that there is more to intelligence than what we had thought so far.

Re: Why I don't think AGI is imminent

#63
Here's a thought. Lets all arbitrarily agree AGI is here. I can't even be bothered discussing what the definition of AGI is. It's just here, accept it. Or vice versa.

Now what....? Whats happening right now that should make me care that AGI is here (or not). Whats the magic thing thats happening with AGI that wasn't happening before?

Right, so, not much has changed from 1-2 years ago that I can tell. The job markets a bit shit if you're in software...is that what we get for billions of dollars spent?

Re: Why I don't think AGI is imminent

#64
post #4

Our brains evolved to hunt prey, find mates, and avoid becoming hunted ourselves. Those three tasks were the main factors for the vast majority of evolutionary history. We didn't evolve our brains to do math, write code, write letters in the right registers to government institutions, or get an intuition on how to fold proteins. For us, these are hard tasks. That's why you get AI competing at IMO level but unable to…

> Our brains evolved to hunt prey, find mates, and avoid becoming hunted ourselves. Those three tasks were the main factors for the vast majority of evolutionary history.

That seems like a massive oversimplification of the things our brains evolved to do.

Re: Why I don't think AGI is imminent

#65
post #49
post #22

Earlier quoted context omitted.

AGI is when it can do all intellectual work that can be done by humans. It can improve its own intelligence and create a feedback loop because it is as smart as the humans who created it.

No, that is ASI. No human can do all intellectual work themselves. You have millions of different human models based on roughly the same architecture to do that. When you have a single model that can do all you require, you are looking at something that can run billions of copies of itself and cause an intelligence explosion or an apocalypse.

"Artificial general intelligence (AGI) is a type of artificial intelligence that matches or surpasses human capabilities across virtually all cognitive tasks."

Re: Why I don't think AGI is imminent

#66
post #34

I don't really understand the argument that AGI cannot be achieved just by scaling current methods. I too believe that (for any sane level of scaling anyway), but this-year's LLMs are not using entirely last-year's methods. And they, in turn, are using methods that weren't used the year before. It seems like a prediction like "Bob won't become a formula one driver in a minivan". It's true, but not very interesting. I…

Then you don't understand Machine Learning in any real way. Literally the 3rd or 4th thing you learn about ML is that for any given problem, there is an ideal model size. Just making the model bigger doesn't work because of something called the curse of dimensionality. This is something we have discovered about every single problem and type of learning algorithm used in ML. For LLMs, we probably moved past the ideal model size about 18 months ago. From the POV of something who actually learned ML in school (from the person who coined the term), I see no real reason to think that AGI will happen based upon the current techniques. Maybe someday. Probably not anytime soon.

PS The first thing you learn about ML is to compare your models to random to make sure the model didn't degenerate during training.

Re: Why I don't think AGI is imminent

#67
post #13

I'm certainly not holding my breath. In a handful of prompts I got the paid version of ChatGPT to say it's possible for dogs to lay eggs under the right circumstances.

Do you believe you could not find humans who would do this?

His objection might be that those humans aren't actually intelligent.

Re: Why I don't think AGI is imminent

#68
post #11

How will we know if its AGI/Not AGI? (I don't think a simple app is gonna cut it here haha) What is the benchmark now that the Turing test has been blown out of the water?

Supranormal GDP growth is my bar. When its actually able to get around bottlenecks and produce value on a societal level

An agent need not have wants, so why would it try to increase its efficiency to obtain things?

Re: Why I don't think AGI is imminent

#69
post #6

I think it's really poor argument that AGI won't happen because model doesn't understand physical world. That can be trained the same way everything else is. I think the biggest issue we currently have is with proper memory. But even that is because it's not feasible to post-train an individual model on its experiences at scale. It's not a fundamental architectural limitation.

You need to be able to at least control things that interact with the world to learn from it.

Re: Why I don't think AGI is imminent

#70
I think it is.

I just struck me - would be fun to re-read The Age of Spiritual Machines (Kurzweil, 1999.) I was so into it 26-27 years ago. The amount of ridicule this man has suffered on HN is immense.

Post reply on HN