Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We could stumble into AI catastrophe

cold-takes.com

Post reply on HN
Print view View original

81–90 of 122 posts

Re: We could stumble into AI catastrophe

#81

Post by taberiand »

zmmmmm wrote: post #72
Animats wrote: post #30

Earlier quoted context omitted.

An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. We have that now. It's called a corporation. See yesterday's article about Exxon.

At least corporations are currently limited by requiring humans who tend to have at least some limits to their actions in terms of moral, ethical considerations and can be physically constrained by imprisoning them etc. An AI could be completely untethered from that and set new thresholds for evil corporate behaviour.

And yet even with current constraints, corporations are capable of destroing the world through climate change and other ecological disasters. Humans are also limited by their personal greed, hubris, selfishness, myopic views and general incompetence.

An AI could be completely untethered from that and set new thresholds for ethical corporate behaviour that operates on fundamentally ethical foundations of caring for the environment and humanity

Re: We could stumble into AI catastrophe

#82

Post by daguru69 »

I'm sorry but you guys are all computer geeks nerding on this and I totally respect you but disagree. Case in point: The Trade Industries are not going to be taken over by AI for a long time. They've withstood all the tech turbulence and thrive.

Also I'd like to say "if Big Brother is watching, become invisible." It's always possible and if you disagree then you aren't being creative enough.

And finally the whole point of the education is YOUR SOCIAL NETWORK. That's the most valuable thing you get from that fancy degree. People who will get you a job, or help you make money. It's not purely about the acquisition of knowledge. It's about having that pal in upper management who's got your back.

IMHO. Please no flaming on this. Be cool.

Re: We could stumble into AI catastrophe

#83

Post by moorg »

Animats wrote: post #30
dwohnitmok wrote: post #22

An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both…

An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. We have that now. It's called a corporation. See yesterday's article about Exxon.

Good observation. It reminds of Etsy giving up its status as a B Corp[1], presumably due to pressure from shareholders. Once the ship is set in motion, it can't divert its course even if it wanted to.

1: https://www.etsy.com/news/business-as-a-force-for-good-defin...

Re: We could stumble into AI catastrophe

#84

Post by LegionMammal978 »

FeepingCreature wrote: post #34
LegionMammal978 wrote: post #9

Earlier quoted context omitted.

The problem with that, at least for the seemingly-dumb models of the present day, is that "the AI is secretly far smarter than it acts" is pretty much unfalsifiable. I could just as easily say that all boulders speak fluent Spanish, they're just deceiving us into thinking they're unintelligent rocks. Personally, I'd wait for some sign that models might actually possess the theory of mind necessary for deception befor…

The problem is that we actually have lots of examples of the AI "secretly" being far smarter than it acts - prompt design. Tiny details in prompts can make huge differences in task performance.

How is this at all supposed to indicate hidden intelligence? Does it not just indicate that current models are limited in that they don't associate some prompts with the desired task as well as they associate other prompts with the task?

Re: We could stumble into AI catastrophe

#85

Post by bumby »

yamtaddle wrote: post #73
Animats wrote: post #57

Earlier quoted context omitted.

We have been framing it that way for a long time though. Yes. That changed on February 28, 1967.[1][2] [1] https://www.chronicle.com/article/the-day-the-purpose-of-col... [2] https://archive.is/G11YT

Fussell (among others) blamed the GI Bill for a marked decline in the average quality of college education, and the diminution of what "college educated" meant (without further qualifiers). The spike in demand drove a ton of "normal" schools (teachers' colleges) to become proper colleges and even universities, plus a bunch of new ones to pop up, but rather than increasing the supply of (previously) college-level educ…

It seems like this sentiment is just a veiled form of elitism. One of the primary detractors when the GI Bill was being rolled out was University of Chicago of Chicago was Maynard Hutchins, who feared it would turn a campus into a shanty town of "hobos." He later walked that statement back and admitted that the veterans coming to schools ended up being higher quality students than the traditional variety.

Re: We could stumble into AI catastrophe

#86

Post by joshuahedlund »

> An AI steals a huge amount of money by bypassing the security system at a bank

Maybe I'm just not imaginative enough, but how would this even be remotely possible? What is an AI going to do with money? How does an AI get the agency to try to steal money in the first place? Where does it put the money? How does it do anything with the money? Like I get it as an abstract future thought, but like, where is the AI software actually "running" in this scenario and what kinds of requests is it actually making?

Re: We could stumble into AI catastrophe

#87

Post by UncleOxidant »

Animats wrote: post #29

This author is mostly writing an update of 1950s science fiction about a robot takeover. Things to worry about in the near term: - Really effective targeted advertising. Most computers already have a camera watching you. That's now coming to TV sets. Amazon and Google are always listening. So far, all this info is only used to select ads. Soon, it should be possible to generate customized marketing content for each c…

> Machines beating humans at business. In some areas, AI systems may generate better returns than humans.

This was part of what was going on in the recent novel "Mountain in the Sea". An AI-controlled fishing corporation figured out that replacing robots with enslaved humans on it's AI controlled fishing fleet was more cost effective (the ocean air was hell on the robots apparently, they kept breaking). Each fishing boat was controlled by an AI which had the objective of maximizing profits. As a result the oceans were being depleted of fish at a rapid rate.

Re: We could stumble into AI catastrophe

#88

Post by vkou »

kgwgk wrote: post #40
vkou wrote: post #31

Earlier quoted context omitted.

> Whoever built, configured, or authorized the system is ultimately responsible... That shouldn't change just because the control policy is more complicated. A poetic sentiment, but does not hold in reality. Anyone who has spent more than ten minutes being a square peg in a round hole can surely recall their frustration when a bureaucrat is refusing to do the common-sense thing, because The Computer is telling them s…

> frustration when a bureaucrat is refusing to do the common-sense thing Bureaucracy-induced frustration predates computers by centuries.

Computers have, just like they have with a lot of other things, made insane enforcement of inapplicable rules more faster, cheaper, and more efficient, and have entirely diluted responsibility for them to the nebulous, unimpeachable, unreachable 'vendor'.

A sufficiently large quantitative difference turns into a qualitative difference. We're becoming a people that, at work, are turning all of our thinking over to machines.

Re: We could stumble into AI catastrophe

#90

Post by gremlinsinc »

joshuahedlund wrote: post #86

> An AI steals a huge amount of money by bypassing the security system at a bank Maybe I'm just not imaginative enough, but how would this even be remotely possible? What is an AI going to do with money? How does an AI get the agency to try to steal money in the first place? Where does it put the money? How does it do anything with the money? Like I get it as an abstract future thought, but like, where is the AI soft…

An ai could easily hire people (if it had any access to money) to operate as 'agents', to basically do what only humans could do, i.e. setup bank accounts, etc. Some banks might even arise that let wealthy ai's actually CREATE their own accounts, why not if it's lucretive?

An ai can create voice, and visual representations of a real person, perhaps even hijacking (kidnapping someone, steeling their identity and forcing them to act on their behalf whether via some sort of mind interface, etc.).

Even bigger would be if ai could simply create human/ai hybrids that basically allow them to walk/talk in the human world unseen. Imagine a hat or earpiece that basically takes over a host's brain, etc, or just replacing a human's brain with a cyborg one that has all the same features, except different persona/soul/conscious...

I'm playing devils' advocate here, I don't think it'll play out, but its easy enough to see from at least a sci-fi fan, ways in which it could using tech that wouldn't be that big of a jump from what we already are capable of, and what will be available in the next 50 years.

Post reply on HN