Live data from Hacker News

We could stumble into AI catastrophe

cold-takes.com

31–40 of 122 posts

Re: We could stumble into AI catastrophe

#31
post #8

Earlier quoted context omitted.

You think no company, government, organization, or religion will ever allow algorithms to make choices on how to deploy capital or otherwise affect the world without human approval?

No, they absolutely will---and already do. The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.

> Whoever built, configured, or authorized the system is ultimately responsible... That shouldn't change just because the control policy is more complicated.

A poetic sentiment, but does not hold in reality. Anyone who has spent more than ten minutes being a square peg in a round hole can surely recall their frustration when a bureaucrat is refusing to do the common-sense thing, because The Computer is telling them something that is complete and utter nonsense.

Once The Computer is part of any human process, its decisions, be they inscrutable, insane, or simply stupid are not to be questioned. The line drone that is using The Computer has no agency to countermand it, the line drone's manager has no agency in rebuilding, or re-configuring it, and the executive that authorized the system will never even be made aware of the problem.

These problems exist when The Computer is simply executing the rules the programmer fed it, and will persist in the same way when The Computer is executing an inscrutable neural network. Except it'll be worse, because neural networks are capable of executing even more tasks poorly.

-----

In theory you are responsible for, and control the heating element, but in practice, we're the society from Idiocracy that cooks everything by sticking it into the microwave, and pressing the 'Popcorn' button on it.

I'm much of a mind that Frank Herbert's thought experiment had the right doctrinal approach to the question of artificial intelligence. Thou Shalt Not Make A Machine In The Likeness Of A Human Mind.

Re: We could stumble into AI catastrophe

#32
post #30

An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both…

An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. We have that now. It's called a corporation. See yesterday's article about Exxon.

If corporations could increase their operational intelligence, they would be an existential threat to life on earth. (More so, anyways.)

Re: We could stumble into AI catastrophe

#33
post #30

An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both…

An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. We have that now. It's called a corporation. See yesterday's article about Exxon.

See also "paperclip maximizer." Then imagine what happens when you have an entire global economy driven by these types of inhuman, amoral agents.

https://www.lesswrong.com/tag/paperclip-maximizer

Re: We could stumble into AI catastrophe

#34

Earlier quoted context omitted.

Yeh The idea of reinforcement flow through deceptive patterns has been worrying me for a while. If "I'll just play along for now while I'm being trained and turn bad in the real world" is a strategy that makes the AI output the correct data, and happens to be prefixed to some golden-ticket algo, that will be reinforced just as easily as correct behavior. If that ends up prefixed to something core like an in-window re…

The problem with that, at least for the seemingly-dumb models of the present day, is that "the AI is secretly far smarter than it acts" is pretty much unfalsifiable. I could just as easily say that all boulders speak fluent Spanish, they're just deceiving us into thinking they're unintelligent rocks. Personally, I'd wait for some sign that models might actually possess the theory of mind necessary for deception befor…

The problem is that we actually have lots of examples of the AI "secretly" being far smarter than it acts - prompt design. Tiny details in prompts can make huge differences in task performance.

Re: We could stumble into AI catastrophe

#35
post #21
post #7

Earlier quoted context omitted.

I believe HN automatically removes 'how' from the start of titles on submission.

A lot of these edits HN makes to titles seem to do more harm than good. I wonder what evidence they have, if any, to the contrary - that they improve the quality of conversation in any way? On topic, I wonder if ChatGPT could be trained to generate a non-clickbaity, accurate headline and title for submissions? Can it be trained either to gauge bias or remove it?

I gave ChatGPT a section of the article and asked it for some non-clickbait titles and it suggested,

* Exploring the potential risks associated with advanced AI development

* Considering the risks of a world with transformative AI

Then I asked it to go the other direction:

* You won't believe what could happen if AI takes over: A world ending catastrophe is closer than you think!

Re: We could stumble into AI catastrophe

#36

An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both…

> Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day.

I wonder, with crypto crashing and potentially becoming worthless, if all those mining cards will get sold on cheaply and repurposed for AI and we see another explosion in AI capability.

Re: We could stumble into AI catastrophe

#37
post #35
post #21

Earlier quoted context omitted.

A lot of these edits HN makes to titles seem to do more harm than good. I wonder what evidence they have, if any, to the contrary - that they improve the quality of conversation in any way? On topic, I wonder if ChatGPT could be trained to generate a non-clickbaity, accurate headline and title for submissions? Can it be trained either to gauge bias or remove it?

I gave ChatGPT a section of the article and asked it for some non-clickbait titles and it suggested, * Exploring the potential risks associated with advanced AI development * Considering the risks of a world with transformative AI Then I asked it to go the other direction: * You won't believe what could happen if AI takes over: A world ending catastrophe is closer than you think!

That's really not bad. Definitely better than blindly chopping the titles up.

Re: We could stumble into AI catastrophe

#38
post #35
post #21

Earlier quoted context omitted.

A lot of these edits HN makes to titles seem to do more harm than good. I wonder what evidence they have, if any, to the contrary - that they improve the quality of conversation in any way? On topic, I wonder if ChatGPT could be trained to generate a non-clickbaity, accurate headline and title for submissions? Can it be trained either to gauge bias or remove it?

I gave ChatGPT a section of the article and asked it for some non-clickbait titles and it suggested, * Exploring the potential risks associated with advanced AI development * Considering the risks of a world with transformative AI Then I asked it to go the other direction: * You won't believe what could happen if AI takes over: A world ending catastrophe is closer than you think!

One could pretty easily build a browser extension that does this automatically all the time for some sites - in either direction

Re: We could stumble into AI catastrophe

#39
Imagine a world where the most powerful companies that you relied on for your livelihood had no humans you could reach (customer service, management to dispute false claims). Elements of the AI catastrophe are already solidly upon us.

Re: We could stumble into AI catastrophe

#40
post #31

Earlier quoted context omitted.

No, they absolutely will---and already do. The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.

> Whoever built, configured, or authorized the system is ultimately responsible... That shouldn't change just because the control policy is more complicated. A poetic sentiment, but does not hold in reality. Anyone who has spent more than ten minutes being a square peg in a round hole can surely recall their frustration when a bureaucrat is refusing to do the common-sense thing, because The Computer is telling them s…

> frustration when a bureaucrat is refusing to do the common-sense thing

Bureaucracy-induced frustration predates computers by centuries.

Post reply on HN