Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sam Altman's Second Coming Sparks New Fears of the AI Apocalypse

wired.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Sam Altman's Second Coming Sparks New Fears of the AI Apocalypse

#11

Post by ithkuil »

barrysteve wrote: post #5

LLMs in theory can absorb a stream of integers into a pattern of weighted nodes. Every program written is vulnerable of being generalized into an LLM pattern. The outputs of AI and the predictive abilities, are not scary. The input capacity of AI will eventually eat every programmer's lunch and their programs too. The existential risk is mapping out computer programming as an entire field and locking it away.

I'm not an AI doomer myself but I want to do my best to understand (and steelman) the argument of those who are concerned about existential risks.

If you really think that the job of programmers is at risk because the current wave of AI can write any program, that also means that AI can now write a new generation of AI systems that is potentially much more powerful than the previous generation and presumably this iteration process can proceed N times until the end result is sufficiently incomprehensible for humans.

Eventually such systems will work very well and be very useful and we'll rely on it more and more.

At that point the people worried about existential risk will point out that when a vastly superior intelligence is tasked to love and protect us, a lower intelligence, it may do things for 'our own good' that we may find appealing.

I'm just about to neuter my cat today

Re: Sam Altman's Second Coming Sparks New Fears of the AI Apocalypse

#12

Post by barrysteve »

ithkuil wrote: post #11
barrysteve wrote: post #5

LLMs in theory can absorb a stream of integers into a pattern of weighted nodes. Every program written is vulnerable of being generalized into an LLM pattern. The outputs of AI and the predictive abilities, are not scary. The input capacity of AI will eventually eat every programmer's lunch and their programs too. The existential risk is mapping out computer programming as an entire field and locking it away.

I'm not an AI doomer myself but I want to do my best to understand (and steelman) the argument of those who are concerned about existential risks. If you really think that the job of programmers is at risk because the current wave of AI can write any program, that also means that AI can now write a new generation of AI systems that is potentially much more powerful than the previous generation and presumably this ite…

I said nothing about writing any program. I explicitly said I don't care what AIs can output.

It can read and devour every program.

Imagine a database where all possible programs are stored.

Any github project, any private repository of source code is reduced to an LLM pattern and stored away.

We can now generalize the meaning of integers and capture every single program that has been written and that you could write.

I'm not going to rewrite my comment a third time. The idea is plainly explained. LLMs can map out and store all possible programs.

Think about the structural implications of this fact. If you make money from programming, you won't like where it leads.

What AIs generate is irrelevant and non-theeatening.

What AIs are capable of storing, is a threat to programming. A "complete map".

I'm about to set my mongoose on the snakes today.

Re: Sam Altman's Second Coming Sparks New Fears of the AI Apocalypse

#13

Post by proc0 »

reducesuffering wrote: post #6
proc0 wrote: post #3

Right now, at worse, AI will have a similar impact as social media. It's definitely worth thinking about the potential harm it could cause to the different industries, but thinking it's an existential risk is still premature. We don't have any cognitive architecture that will reach self-awareness and become an unstoppable bad actor. This entire ordeal more or less proved that people are a little too paranoid.

All the top brass domain experts of these systems, like 95% besides Yann LeCunn, believe it's an existential risk. Nothing has "proved that people are a little too paranoid." Why people listen to HN prognosticators over Hinton, Bengio, Sutskever, Altman, Amodei, Hassabis, etc. is beyond me.

I'm just commenting, and have some basic understanding. It's good to start thinking about it but I seriously doubt GPT or anything that can be built with it can have a direct existential risk, meaning it would somehow free itself and go rogue. Otherwise I don't know how it would be an existential risk (outside of massive impact in society, but that's technically a different problem). My argument hinges on not being able to accidentally stumble upon AGI without a theory first. At least not an AGI that thinks and has agency like a human, with selfish goals.

Re: Sam Altman's Second Coming Sparks New Fears of the AI Apocalypse

#14

Post by proc0 »

TerrifiedMouse wrote: post #7
proc0 wrote: post #3

Right now, at worse, AI will have a similar impact as social media. It's definitely worth thinking about the potential harm it could cause to the different industries, but thinking it's an existential risk is still premature. We don't have any cognitive architecture that will reach self-awareness and become an unstoppable bad actor. This entire ordeal more or less proved that people are a little too paranoid.

> but thinking it's an existential risk is still premature Even ignoring what the technology can do, it feels me with dread that we basically have IMHO some of the worst possible people leading the charge - people who are greedy, slimy, and ruthless; who only care about money. Such a bad timeline we are in.

Well at least Demis Hassabis seems cool.

Re: Sam Altman's Second Coming Sparks New Fears of the AI Apocalypse

#15

Post by ithkuil »

barrysteve wrote: post #12
ithkuil wrote: post #11

Earlier quoted context omitted.

I'm not an AI doomer myself but I want to do my best to understand (and steelman) the argument of those who are concerned about existential risks. If you really think that the job of programmers is at risk because the current wave of AI can write any program, that also means that AI can now write a new generation of AI systems that is potentially much more powerful than the previous generation and presumably this ite…

I said nothing about writing any program. I explicitly said I don't care what AIs can output. It can read and devour every program. Imagine a database where all possible programs are stored. Any github project, any private repository of source code is reduced to an LLM pattern and stored away. We can now generalize the meaning of integers and capture every single program that has been written and that you could write…

Sorry I don't understand. What's the problem if you're just storing all programs? I can store all possible programs today on a hard drive today.

You can easily "store" all programs written in the history of humankind and all those who are yet to be written, in the same way you can "store" all books ever written by humans and those yet to be written (see "The Library of Babel" by Jorge Luis Borges)

The threat to the working programmer is having something that "understands" the programs and tells which ones are "good" (correct, useful, etc), from those who don't work (perhaps because they are a little wrong or perhaps because they are just a random jumble of instructions).

That's what I understand when I read the word "store" in the context of what you've written, but I'm pretty sure I'm misunderstanding something here.

Are you referring to the fact that the incentives behind the current wave of AI is to get them trained by copying all source code publicly available and that that is a problem because these AI models now absorbed the code without honouring the intellectual rights of the authors (and not honouring the license the authors have chosen for their creations)?

BTW, I agree that that is a problem. But it is a problem because of the outputs: the models _output_ snippets of code that is distilled from patterns (and sometimes outright copies) of the code (and other text) it was trained on. The process of extracting that information from the model *is* the process of generating an output.

Re: Sam Altman's Second Coming Sparks New Fears of the AI Apocalypse

#16

Post by gumballindie »

barrysteve wrote: post #5

LLMs in theory can absorb a stream of integers into a pattern of weighted nodes. Every program written is vulnerable of being generalized into an LLM pattern. The outputs of AI and the predictive abilities, are not scary. The input capacity of AI will eventually eat every programmer's lunch and their programs too. The existential risk is mapping out computer programming as an entire field and locking it away.

Explain why y’all obsessed with spreading fear and why y’all culting a dubious dude such as altman?

Re: Sam Altman's Second Coming Sparks New Fears of the AI Apocalypse

#17

Post by Nasrudith »

reducesuffering wrote: post #6
proc0 wrote: post #3

Right now, at worse, AI will have a similar impact as social media. It's definitely worth thinking about the potential harm it could cause to the different industries, but thinking it's an existential risk is still premature. We don't have any cognitive architecture that will reach self-awareness and become an unstoppable bad actor. This entire ordeal more or less proved that people are a little too paranoid.

All the top brass domain experts of these systems, like 95% besides Yann LeCunn, believe it's an existential risk. Nothing has "proved that people are a little too paranoid." Why people listen to HN prognosticators over Hinton, Bengio, Sutskever, Altman, Amodei, Hassabis, etc. is beyond me.

Existential risk is a doomsday prediction. And we all know the track record of doomsday predictions. 99% of the time it shows a simple lack of a sense of proportion that cannot tell the difference between merely "bad" or "very bad" and "annihilation". A personal bankruptcy is bad but it isn't an existential risk to a person.

Re: Sam Altman's Second Coming Sparks New Fears of the AI Apocalypse

#18

Post by reducesuffering »

Nasrudith wrote: post #17
reducesuffering wrote: post #6

Earlier quoted context omitted.

All the top brass domain experts of these systems, like 95% besides Yann LeCunn, believe it's an existential risk. Nothing has "proved that people are a little too paranoid." Why people listen to HN prognosticators over Hinton, Bengio, Sutskever, Altman, Amodei, Hassabis, etc. is beyond me.

Existential risk is a doomsday prediction. And we all know the track record of doomsday predictions. 99% of the time it shows a simple lack of a sense of proportion that cannot tell the difference between merely "bad" or "very bad" and "annihilation". A personal bankruptcy is bad but it isn't an existential risk to a person.

99% throughout history. During that time, humanity has gained an accelerating command over our environment; Industrial Revolution to the Information Era. We are increasingly able to create unimaginably powerful things, yet coinciding with the fundamental that it is easier to destroy than create.

Nuclear weapons should dispel anyone of the notion that certain “doomsday” don’t have teeth. We are quite capable of obliterating ourselves out of existence by our unsophisticated control of technical might.

Re: Sam Altman's Second Coming Sparks New Fears of the AI Apocalypse

#19

Post by Obscurity4340 »

ilaksh wrote: post #4
proc0 wrote: post #3

Right now, at worse, AI will have a similar impact as social media. It's definitely worth thinking about the potential harm it could cause to the different industries, but thinking it's an existential risk is still premature. We don't have any cognitive architecture that will reach self-awareness and become an unstoppable bad actor. This entire ordeal more or less proved that people are a little too paranoid.

We don't have a dangerous AI yet, but because we are talking about an existential risk, yes we do want to start thinking about it now. AI does not have to be alive or truly conscious or anything to be dangerous. It just needs to be very effective at problem solving and have someone take the guardrails off and tell it to act in a self-interested way. Especially without humans in the loop. It seems likely that the syst…

Maybe I am just daft but how is this not a simple privelege escalation issue? Just don't give it admin, what's the problem?

Re: Sam Altman's Second Coming Sparks New Fears of the AI Apocalypse

#20

Post by BoiledCabbage »

Obscurity4340 wrote: post #19
ilaksh wrote: post #4

Earlier quoted context omitted.

We don't have a dangerous AI yet, but because we are talking about an existential risk, yes we do want to start thinking about it now. AI does not have to be alive or truly conscious or anything to be dangerous. It just needs to be very effective at problem solving and have someone take the guardrails off and tell it to act in a self-interested way. Especially without humans in the loop. It seems likely that the syst…

Maybe I am just daft but how is this not a simple privelege escalation issue? Just don't give it admin, what's the problem?

Because in software we've defined an environment where every sw method of gaining privilege/escalation is hardened. As a result, usually the most reliable way to get escalation is via social engineering.

LLMs exist in an environment whose vulnerabke surface area is social engineering. How do you lock down a system against all possible social engineering?

Oh yah and the "system" isn't any computer system somewhere, it's the entire world itself.

So one of the problems the safetiests are trying to solve is How do you protect a messy system, the size/complexity of the entire world from social engineering? The answer is clearly not using traditional approaches which consistently fail.

And that's just one of the problems they're trying to solve.

Post reply on HN