Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The changing goalposts of AGI and timelines

mlumiste.com

Post reply on HN
Print view View original

261–270 of 411 posts

Re: The changing goalposts of AGI and timelines

#261

Post by stavros »

0xbadcafebee wrote: post #22

AGI isn't going to happen within the next 30 years so this is moot. The actual researchers have said so many times. It's only the business people and laypeople whooping about AGI always being imminent. You cannot get real, actual AGI (the same ability to perform tasks as a human) without a continuous cycle of learning and deep memory, which LLMs cannot do. The best LLM "memory" is a search engine and document summari…

> tell one "don't delete files in X/", and after a while, it will delete all the files in "X/", whereas a human would likely remember it's not supposed to delete some files, and go check first.

Have you seriously never had someone to go do something you told them not to do?

> It also does fun stuff like follow arbitrary instructions from an attacker found in random documents, which most humans also wouldn't do.

I guess my coworker didn't actually fall for that "hey this is your CEO, please change my password" WhatsApp message then, phew.

I've seen people move the goalposts on what it means for AI to be intelligent, but this is the first time I've seen someone move the goalposts on what it means for humans to be intelligent.

Re: The changing goalposts of AGI and timelines

#262

Post by heavyset_go »

sulam wrote: post #237
famouswaffles wrote: post #208

Earlier quoted context omitted.

People just overstate their understanding and knowledge, the usual human stuff. The same user has a comment in this thread that contains: 'If you actually know what models are doing under the hood to product output that...' Any one that tells you they know 'what models are dong under the hood' simply has no idea what they're talking about, and it's amazing how common this is.

Fair, I should define what I mean by under the hood. By “under the hood” I mean that models are still just being fed a stream of text (or other tokens in the case of video and audio models), being asked to predict the next token, and then doing that again. There is no technique that anyone has discovered that is different than that, at least not that is in production. If you think there is, and people are just keepin…

> Btw, don’t take this reductionist approach as being synonymous with thinking these models aren’t incredibly useful and transformative for multiple industries. They’re a very big deal. But OpenAI shouldn’t give up because Opus 4.whatever is doing better on a bunch of benchmarks that are either saturated or in the training data, or have been RLHF’d to hell and back. This is not AGI.

It's sad that you have to add this postscript lest you be accused of being ignorant or anti-AI because you acknowledge that LLMs are not AGI.

Re: The changing goalposts of AGI and timelines

#263

Post by torginus »

takwatanabe wrote: post #252
0xbadcafebee wrote: post #22

AGI isn't going to happen within the next 30 years so this is moot. The actual researchers have said so many times. It's only the business people and laypeople whooping about AGI always being imminent. You cannot get real, actual AGI (the same ability to perform tasks as a human) without a continuous cycle of learning and deep memory, which LLMs cannot do. The best LLM "memory" is a search engine and document summari…

The post-it note analogy is good, but as a psychiatrist, I'd frame it differently: LLMs are essentially patients with anterograde amnesia. They can reason brilliantly within a single conversation — just like an amnesic patient can hold an intelligent discussion — but the moment the session ends, everything is gone. No learning happened. No memory formed. What's worse, even within a session, they degrade. Research sho…

I want to believe I'm reading an insightful comment from an actual human deeply familiar with both human congnition and how LLMs work, but this post is chock full of LLMisms

Re: The changing goalposts of AGI and timelines

#266

Post by torginus »

fwipsy wrote: post #89
0xbadcafebee wrote: post #22

AGI isn't going to happen within the next 30 years so this is moot. The actual researchers have said so many times. It's only the business people and laypeople whooping about AGI always being imminent. You cannot get real, actual AGI (the same ability to perform tasks as a human) without a continuous cycle of learning and deep memory, which LLMs cannot do. The best LLM "memory" is a search engine and document summari…

Given how many "fundamental" limitations of AI have been resolved within the past few years, I'm skeptical. Even if you're right, I am not sure that the limitations you identified matter all that much in practice. I think very few human engineers are working on problems which are so novel and unique that AIs cannot grasp them without additional reinforcement learning. > it will delete all the files in "X/" How many "…

> How many "I deleted the prod database" stories have you seen?

If you've used the latest models extensively, you must've noticed times when AI 'runs out of common sense' and keeps trying stupid stuff.

I'm somewhat convinced that the amazing (and improving!) coding ability of these LLMs comes from it being RLHFd on the conversations its having with programmers, with each successfully resolved bug, implemented feature ending up in training data.

Thus we are involuntarily building the world's biggest stackoverflow.

Which for the record is incredibly useful, and may even put most programmers out of a job (who I think at that point should feel a bit stupid for letting this happen), but its not necessarily AGI.

Re: The changing goalposts of AGI and timelines

#267

Post by takwatanabe »

torginus wrote: post #263
takwatanabe wrote: post #252

Earlier quoted context omitted.

The post-it note analogy is good, but as a psychiatrist, I'd frame it differently: LLMs are essentially patients with anterograde amnesia. They can reason brilliantly within a single conversation — just like an amnesic patient can hold an intelligent discussion — but the moment the session ends, everything is gone. No learning happened. No memory formed. What's worse, even within a session, they degrade. Research sho…

I want to believe I'm reading an insightful comment from an actual human deeply familiar with both human congnition and how LLMs work, but this post is chock full of LLMisms

Yeah, fair enough. I leaned on Claude to clean up my English. I normally write in Japanese. The clinical stuff is mine though, I run a psych clinic in Japan (link in profile). Should've just written it messier.

Re: The changing goalposts of AGI and timelines

#269

Post by lejalv »

remarkEon wrote: post #66
BoxFour wrote: post #55

Earlier quoted context omitted.

I don't think it's about lethal autonomy specifically as much as it's just about government autonomy period. They don’t think private companies should have any veto power over how the government uses some technology they're provided. On its face that’s not a crazy stance: Governments are meant to represent the public, while private companies obviously aren't. I think it’s somewhat understandable why the government mi…

Agree, and I think the labeling of them (Anthropic) a supply chain risk was handled poorly and will likely be reverted over time. That being said, I would be nervous if I was in the Pentagon and depended on Anthropic tooling for something, even if that something was unrelated to kinetic operations. How do they audit that Anthropic can't alter model outputs for contexts they (the ethics board or whatever it's called,…

> If you sell a weapon to the department that is in charge of killing people and breaking things, you don't get a say in who gets killed or how. It's never worked like that.

I can't agree that this is the right comparison. What is being sold here is not just another missile or tank type, it is the very agency and responsibility over life and death. It's potentially the firing of thousands of missiles.

Re: The changing goalposts of AGI and timelines

#270

Post by BoxFour »

mlinhares wrote: post #176
BoxFour wrote: post #87

Earlier quoted context omitted.

> It’s not obvious that the government should have to power to overwrite this The government shouldn’t be able to set the terms of its contracts with private companies and walk away if those terms aren’t acceptable? That seems like a stretch. The constitution is a wildly different premise from government contracting with private companies.

There was no contract, the government wanted to have a contract where they'd be able to use the tool to violate privacy rights of its citizens and issue kill orders without a human present and the company said no. The government shouldn't be able to coerce a business to do whatever it wants.

> There was no contract, the government wanted to have a contract where they'd be able to use the tool to violate privacy rights of its citizens and issue kill orders without a human present and the company said no.

So the contract process worked. The seller wanted certain clauses, the buyer rejected them, and the deal didn’t happen.

Setting aside the supply chain risk designation, which I already said was an extreme overreaction, this is basically how it’s supposed to work.

> The government shouldn't be able to coerce a business to do whatever it wants.

Governments coerce businesses all the time to do what the government wants. Taxes are the obvious example, but there are many others like OFAC sanctions lists or even just regular old business regulations.

It mostly works because we rely on governments to use that power wisely, and to use it in a way that represents the wishes of the populace. Clearly that assumption is being tested with the current administration and especially in this particular situation, but the government coerces businesses to do what they want all the time and we often see it as a good thing.

Post reply on HN