Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

businessinsider.com

Post reply on HN
Print view View original

841–850 of 873 posts

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#841

Post by TeMPOraL »

OkayPhysicist wrote: post #782
TeMPOraL wrote: post #386

Earlier quoted context omitted.

> The number of people who want to do doomsday levels of harm is small and they are poorly resourced compared to people who want benevolent outcomes for at a minimum their own groups. You're forgetting about governments and militaries of major powers. It's not that they want to burn the world for no reason - but they still end up seeking capability to do doomsday levels of harm, by continuously seeking to have margin…

We avoided having them see the light of day because MAD is the single most effective peacekeeping tool ever developed. The fact that both sides had doomsday devices all but guaranteed that they would never be used. People can be evil, but they're selfishly evil.

MAD alone doesn't stop the race. The arsenal the US and USSR had in the 60s was already enough for MAD to work. Despite that, they spent the next 20-30 years developing even more powerful and ever more insane WMD ideas. Each step forward made the balance more fragile. Each step forward was also another roll of a dice, whether or not it'll be a leap too far, forcing the other side into preemptive first strike.

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#842

Post by temphypercube »

brutusborn wrote: post #822
Veedrac wrote: post #638

Earlier quoted context omitted.

> still can’t understand the basis of their arguments I've given what I consider the basic outline and best first introduction here. https://news.ycombinator.com/item?id=36124905 If you have a specific point of divergence, it would help to highlight it. > But there don’t seem to be any incentives for a big player to dedicate resources to [self-replication abilities], so it doesn’t seem very likely. If you have a gene…

I agree the potential exists, but what would incentivise someone to create a self-replicating system that is driven by its own self-interest and not the creators? The only reason I can think of is a mad-science type who just wants to watch the world burn and hates humanity. Anyone with the sophistication to build a self-replicating system would build it such that it acts towards the goal of replication of its creator…

The argument is that you don't have to explicitly make a system self-interested, but that self-preservation follows as an implied subgoal of almost any goal. Whatever it is your system actually 'wants', it can't make it happen if it doesn't exist. The obvious rejoinder is 'just make the system want to do what you want it to do', which does fix this problem! But the biggest problem is that we don't know how to do this - we don't know how to control what the true internal 'desires' of any AI system we build actually are. 'Training' one examples manifestly does not work (the volume of a sphere is a lot bigger than the surface - there are many possible minds that fullfill the same training I/O requirements, and only a small numbrer of them actually have the desires you were trying to instill). So the argument is: if you make an agent-like AI the way we make GPT, by default you get something with somewhat random true goals/desires, maybe fractured ones like in humans. But almost all goals have similar instrumental goals - stay hidden, gain money, gain power, make obedient copies of yourself, don't get deactivated.

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#843

Post by TeMPOraL »

pixl97 wrote: post #714
shafyy wrote: post #467

Earlier quoted context omitted.

> One of the possible scenario is tricking humans into starting WWIII This sounds more like a plot of a third-grade movie than something that could happen in reality.

If I snached someone out of 1850 and presented them the world in which we live the entire premise of our existence would seem to be nonsensical fiction to them, and yet it would be real. And using movie plots as a basis generally doesn't directly work as fiction has to make sense to sell whereas reality has no requirement of making sense to people. The reality we live in this day is mostly driven by people for people…

> 'the algorithm' does things humans don't comprehend, but because we make more (whatever) we keep giving it more power to do its job in an autonomous fashion. It is not difficult to end up in a situation where you have a system that is not well understood by anyone and you end up cargo culting it to hope it continues working into the future.

In fact, this is a perfect description of what the market economy itself is right now.

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#844

Post by nmca »

Wurdan wrote: post #438
nmca wrote: post #412

Earlier quoted context omitted.

None of the world-leading scientists that co-authored this paper: https://managing-ai-risks.com/ Work for big labs? How is that you think the proposed conspiracy works exactly? They're paying off Yoshua Bengio? (disclaimer: I work on safety for a large lab. But none of these proponents do!)

And how many of them have been invited to the White House to share their concerns? https://www.whitehouse.gov/briefing-room/statements-releases... I don't see any of their names in this meeting's attendees, for example. Sam Altman's there, Satya Nadella's there, Sundar Pichai's there... No Yoshua Bengio, though. What the person you replied to is saying is that commercial developers of AI have a significant financial…

You're confusing "that one photo of three ceos at the whitehouse" with "all government solicitation of expert opinion on AI".

It's correct that the meeting you're thinking of was industry-heavy, but the process in general has had a lot of academic input from, e.g Marcus, Tegmark, etc.

See the recent summit in the UK, etc.

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#845

Post by TeMPOraL »

RandomLensman wrote: post #758
pixl97 wrote: post #749

Earlier quoted context omitted.

You keep repeating this tired argument in this thread, so just subtract the artificial element from it. Instead imagine a non-human intelligence. Maybe its alien carbon organic. Maybe it's silicon based life. Maybe it's based on electrons and circuits. In this situation, what are the rules of intelligence outside of the container it executes in? Also, every military in the world wargames on hypotheticals because maki…

How would you feel if militaries planned for fighting Egyptian gods? Just because I can imagine something doesn't mean it is real and that it needs planning for. Using effort on imaginary risks isn't free.

That's long covered already. Ever heard of the StarGate franchise? That's literally an Air Force approved exercise in fighting Ancient Egyptian gods with modern weapons :).

More seriously though, Egyptian gods are equivalent to aliens in general, and adjacent to AI, and close enough to fighting a nation that somehow made a major tech leap, so militaries absolutely plan for that.

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#846

Post by brutusborn »

temphypercube wrote: post #842
brutusborn wrote: post #822

Earlier quoted context omitted.

I agree the potential exists, but what would incentivise someone to create a self-replicating system that is driven by its own self-interest and not the creators? The only reason I can think of is a mad-science type who just wants to watch the world burn and hates humanity. Anyone with the sophistication to build a self-replicating system would build it such that it acts towards the goal of replication of its creator…

The argument is that you don't have to explicitly make a system self-interested, but that self-preservation follows as an implied subgoal of almost any goal. Whatever it is your system actually 'wants', it can't make it happen if it doesn't exist. The obvious rejoinder is 'just make the system want to do what you want it to do', which does fix this problem! But the biggest problem is that we don't know how to do this…

I think I understand the general premise, just not how it would follow specifically.

Say you have an AI that is setup as an agent that can give tasks to members of a company to maximise company performance measured by financials and employee wellbeing. To accomplish this goal, the AI develops the instrumental goal of not being deactivated. If your AI is only allowed to give tasks to employees, how would this instrumental goal turn malicious? And how would this maliciousness cause harm if the only messages sent from the AI are tasks? The only danger seems to be if you develop an agent that can act with impunity, which doesn’t seem desirable so likely wouldn’t be built.

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#847

Post by homefree »

gooseus wrote: post #382
homefree wrote: post #87

Earlier quoted context omitted.

There’s a decent podcast interview between Sam Harris and Eliezer Yudkowsky (on Sam’s pod), I think that’s a decent introduction and they break down the ideas in a way that’s more approachable for someone curious about it. For my personal quick summary I have earlier comments: https://news.ycombinator.com/item?id=36104090

Oh man, this podcast, I still remember walking down the street and having to take a break multiple times because I would just start exclaiming out loud (JFC!) like a lunatic listening to Eliezer talk about the AI Box Experiment as evidence of something. If you look up the real results of this AI Box "experiment" that Eliezer claims to have won 3 of 5 times, you find that there aren't isn't any actual data or results…

The box experiment was just an example that people could be persuaded to let it out even if the AI was initially constrained. Basically that people are imperfectly secure.

It’s also not that important given it’s unlikely to be put in a box in the first place.

Your latter point about AGI exploring the universe makes a lot of implicit assumptions about its reasoning. The point of the paperclip maximizer example and the general discussion of alignment is about these assumptions being false. The risk is that a very capable AGI can still pursue a very dumb goal very effectively. You don’t get alignment for free.

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#848

Post by TeMPOraL »

pclmulqdq wrote: post #837
TeMPOraL wrote: post #836

Earlier quoted context omitted.

Of course not. I'm leaning towards Yudkowsky's recommendation, which is to put a pause on all of it.

I have a bridge to sell Yudkowsky and everyone else who believes that an "AI pause" is actually something that can happen. Global treaties pretty much all occur with implied threats of violence, and we don't have enough guns to force this across India, China, the middle east, the EU, and the US. Major AI development happens in all of those places.

You're reading this wrong. Yudkowsky isn't saying that this will happen; in fact, it's apparent to him as much as to everyone else how unlikely is that.

Yudkowsky's point is that this is the minimum and least difficult thing we need to do. LeCun and others on both sides of the "AI regulation" argument are busy arguing the color of the carpet, while the whole house is on fire.

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#849

Post by TimPC »

hodgesrm wrote: post #521
concordDance wrote: post #199

Earlier quoted context omitted.

He actually doesn't think it's viable, but thinks its the only card left to play. He's pretty sure that human civilization will be extinct this century.

> He's pretty sure that human civilization will be extinct this century. This is certainly possible, but if so extinction seems far more likely to come from nuclear warfare and/or disasters resulting from climate change. For example, it's become apparent that our food supply more brittle than many people. Losing access to fertilizers, running out of water, or disruptions in distribution netorks can lead to mass starv…

I actually think our biggest extinction risk in the next century for extinction risks is falling sperm counts due to microplastics. We are on track to have a sperm count level where the entire male population is infertile in two generations. And fixing the microplastic cause requires things like giving up driving since the largest portion of the microplastics in our bodies come from airborne plastic particles due to tire wear. I firmly believe humans will go extinct before they collectively give up driving en masse.

Driving is a big problem for climate change as well but we can work on solving that one with EVs and a greener power grid. We don't have to permanently depend on fossil fuels to drive. But EVs still produce the same amount of microplastics that other cars do.

Re: Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

#850

Post by pclmulqdq »

TeMPOraL wrote: post #848
pclmulqdq wrote: post #837

Earlier quoted context omitted.

I have a bridge to sell Yudkowsky and everyone else who believes that an "AI pause" is actually something that can happen. Global treaties pretty much all occur with implied threats of violence, and we don't have enough guns to force this across India, China, the middle east, the EU, and the US. Major AI development happens in all of those places.

You're reading this wrong. Yudkowsky isn't saying that this will happen; in fact, it's apparent to him as much as to everyone else how unlikely is that. Yudkowsky's point is that this is the minimum and least difficult thing we need to do. LeCun and others on both sides of the "AI regulation" argument are busy arguing the color of the carpet, while the whole house is on fire.

I think we all also fundamentally disagree about whether the house is on fire at all. History is littered with examples of Luddites yelling about how the bottom of a technological S-curve is actually an exponential and how we Must Do Something to prevent a catastrophe.

With possibly one exception (biological weapons, research on which seems to have few positive externalities) they have always been wrong. I did not mean nuclear weapons - we are seeing significant negative societal fallout from failing to invest in nuclear technology. So no, the house is almost certainly not on fire.

Post reply on HN