Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We could stumble into AI catastrophe

cold-takes.com

Post reply on HN
Print view View original

11–20 of 122 posts

Re: We could stumble into AI catastrophe

#11

Post by sturza »

FeepingCreature wrote: post #3
dankle wrote: post #2

[flagged]

Yeh The idea of reinforcement flow through deceptive patterns has been worrying me for a while. If "I'll just play along for now while I'm being trained and turn bad in the real world" is a strategy that makes the AI output the correct data, and happens to be prefixed to some golden-ticket algo, that will be reinforced just as easily as correct behavior. If that ends up prefixed to something core like an in-window re…

For this to be true, we'd had to assume sentience + malevolence. The shortest explanation for why an AI give us correct data currently is a range of confidence. We instructed it to pick the one with the highest number. We instructed it to give us a similar next data etc. We reinforced it, yes, but we'd have to be naive to believe that taking a small part of what makes something "intelligent" and training the shit out of it, then it actually turns intelligent. chatGPT is "just" a large language model, it does not have senses, it cannot move, it cannot live and die. It can only output text. The next gen will output text (maybe) exponentially better. That does not make it "intelligent". But, if we'd combine it with Boston Dynamics, Tesla Autopilot, internet, quantum computing, nano tech and other specifically trained parts (movement, sensing etc) into one, and release it into the world to see what it does, then your argument could (maybe)have a higher confidence of the outcome you fear.

Re: We could stumble into AI catastrophe

#13

Post by mattkrause »

mcculley wrote: post #8
DerekBickerton wrote: post #5

AI will always need humans in the loop to intervene. Remember the old IBM motto: 'A computer isn't accountable so a computer should never make a management decision'.

You think no company, government, organization, or religion will ever allow algorithms to make choices on how to deploy capital or otherwise affect the world without human approval?

No, they absolutely will---and already do.

The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.

Re: We could stumble into AI catastrophe

#14

Post by TedDoesntTalk »

krapp wrote: post #10
DerekBickerton wrote: post #5

AI will always need humans in the loop to intervene. Remember the old IBM motto: 'A computer isn't accountable so a computer should never make a management decision'.

But they won't. Once AIs become advanced enough to be useful, humans will be removed from the loop wherever it becomes profitable to do so. Even before then, it will be attempted just because the potential reward is worth the risk. That's the entire goal of automation through AI, both in terms of centralizing the capture of value and distributing risk. Who do you blame when a fully autonomous AI corporation commits t…

Isn’t this how Google and Facebook bans already work? Maybe not using AI, but it’s automated and there’s no accountability or appeal or due process.

Re: We could stumble into AI catastrophe

#16

Post by dwohnitmok »

sturza wrote: post #11
FeepingCreature wrote: post #3

Earlier quoted context omitted.

Yeh The idea of reinforcement flow through deceptive patterns has been worrying me for a while. If "I'll just play along for now while I'm being trained and turn bad in the real world" is a strategy that makes the AI output the correct data, and happens to be prefixed to some golden-ticket algo, that will be reinforced just as easily as correct behavior. If that ends up prefixed to something core like an in-window re…

For this to be true, we'd had to assume sentience + malevolence. The shortest explanation for why an AI give us correct data currently is a range of confidence. We instructed it to pick the one with the highest number. We instructed it to give us a similar next data etc. We reinforced it, yes, but we'd have to be naive to believe that taking a small part of what makes something "intelligent" and training the shit out…

> For this to be true, we'd had to assume sentience + malevolence.

No we don't. A "dumb," "parrot-like" AI is perfectly capable of large amounts of damage if it just happens to be really good at doing something that, in the limit, isn't very good for humans and is good at replication.

See for example Meta's Diplomacy-playing bots. They were perfectly capable of natural-language deception at the expense of other humans without any notion f sentience or malevolence.

In particular,

> It can only output text.

As we've seen with both private hobbyists and large companies, there is a race to hook up text to computers with real world access. Outputting text isn't much of a limit if that text ends up as instructions to command a real world system.

Re: We could stumble into AI catastrophe

#17

Post by moolcool »

mcculley wrote: post #8
DerekBickerton wrote: post #5

AI will always need humans in the loop to intervene. Remember the old IBM motto: 'A computer isn't accountable so a computer should never make a management decision'.

You think no company, government, organization, or religion will ever allow algorithms to make choices on how to deploy capital or otherwise affect the world without human approval?

Algorithms deploy capital all the time. Some massive chunk of the stock market is just algorithmic trading, as is pricing on plane tickets, and on marketplaces like ebay and amazon.

Re: We could stumble into AI catastrophe

#18

Post by mcculley »

moolcool wrote: post #17
mcculley wrote: post #8

Earlier quoted context omitted.

You think no company, government, organization, or religion will ever allow algorithms to make choices on how to deploy capital or otherwise affect the world without human approval?

Algorithms deploy capital all the time. Some massive chunk of the stock market is just algorithmic trading, as is pricing on plane tickets, and on marketplaces like ebay and amazon.

I agree. This is why I am surprised by the assertion and asking the person who made it.

Re: We could stumble into AI catastrophe

#19

Post by mcculley »

mattkrause wrote: post #13
mcculley wrote: post #8

Earlier quoted context omitted.

You think no company, government, organization, or religion will ever allow algorithms to make choices on how to deploy capital or otherwise affect the world without human approval?

No, they absolutely will---and already do. The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.

I agree. This is why I am surprised by the assertion and asking the person who made it.

Re: We could stumble into AI catastrophe

#20

Post by mcculley »

mattkrause wrote: post #13
mcculley wrote: post #8

Earlier quoted context omitted.

You think no company, government, organization, or religion will ever allow algorithms to make choices on how to deploy capital or otherwise affect the world without human approval?

No, they absolutely will---and already do. The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.

We already have a mechanism for absolving responsibility: the corporation. It allows shareholders to profit and not lose more than invested. When they invest in tobacco companies, for example, the system is working as designed.
Post reply on HN