Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI wolf that preferred suicide over eating sheep

lancengym.medium.com

Post reply on HN
Print view View original

51–60 of 223 posts

Re: An AI wolf that preferred suicide over eating sheep

#51

Post by foldr »

TeMPOraL wrote: post #43
foldr wrote: post #38

Earlier quoted context omitted.

Aesop managed to make the point a lot more concisely: "Be careful what you wish for, lest it come true." (Although now that I look, I don't think that's a translation of any specific part of the text.)

Yes, but that moral is attached to a story . Morals and saws work as handles - they're useful for communication if both you and your interlocutor know the thing they're pointing to. Conversely, they are of little use until you read the story from which the moral comes, or personally experience the thing the saw talks about.

Eliezer Yudkowsky tells a long story about an Outcome Pump. Aesop tells a short story about an eagle and a tortoise. The point made is the same, as far as I can see.

Re: An AI wolf that preferred suicide over eating sheep

#52

Post by imtringued »

bserge wrote: post #42
Brendinooo wrote: post #39

Earlier quoted context omitted.

I think the point is more about highlighting the fact that AI doesn't share our base assumptions. We wouldn't think to put a huge penalty on dying because humans generally think that death is bad.

Yeah, because we have a -1000 points on death built-in.

We don't receive a penalty for dying. The difference between suicidal humans and suicidal AIs is that suicidal AIs keep respawning i.e. they are immortal.

Re: An AI wolf that preferred suicide over eating sheep

#53

Post by FridayoLeary »

jhbadger wrote: post #4

I'm reminded of the fable (in Nick Bostrom's Superintelligence ) of the chess computer that ended up murdering anyone who tried to turn it off because in order to optimize winning chess games as programmed it has to be on and functional.

Which perverted mind would build into a chess computer the ability to kill?

Re: An AI wolf that preferred suicide over eating sheep

#54

Post by TeMPOraL »

foldr wrote: post #51
TeMPOraL wrote: post #43

Earlier quoted context omitted.

Yes, but that moral is attached to a story . Morals and saws work as handles - they're useful for communication if both you and your interlocutor know the thing they're pointing to. Conversely, they are of little use until you read the story from which the moral comes, or personally experience the thing the saw talks about.

Eliezer Yudkowsky tells a long story about an Outcome Pump. Aesop tells a short story about an eagle and a tortoise. The point made is the same, as far as I can see.

Eliezer tells the story that elaborates on why you should be careful what you wish for. Of about a dozen versions of the Eagle and Tortoise story I've just skim-read, none of them really has this as a moral - in each of them, either the Eagle or a Tortoise was an asshole and/or liar and/or lying asshole, so the more valid moral would be, "don't deal with dangerous people" and/or "don't be an asshole" and/or "don't be an asshole to people who have power to hurt you".

Re: An AI wolf that preferred suicide over eating sheep

#55

Post by SamBam »

m12k wrote: post #29

I think a major takeaway here is that balancing a reward system to reward more than a single behavior is really hard - it's easy to tip the scales so one behavior completely dominates all others. It's an interesting lens to use to look at the heuristic reward system humans have built in (hunger, fear, desire, etc). This tends to have an adaptation/numbing effect, where repeated rewards of the same type tend to have d…

Evolution works in an incredibly complex "fitness landscape," where certain minor tweaks in phenotype or behaviors can affect your fitness in quite complex ways.

Genetic Algorithms attempt to use this same system over extremely simple "fitness landscapes," where the fitness of an agent is defined by programmers using some simple mathematical formula or something.

When the fitness function is being defined in the system by programmers, instead of emerging from a rich and complex ecosystem, then the outcome depends exactly on what the programers choose. If they fail to see the consequences of their scoring algorithm, that's on them. There's nothing really magical going on, they simply failed to foresee the consequences of their choice.

(As someone who has worked with GAs and agent models, this outcome really doesn't surprise me. I would have said "oops, I need to weight the time less" and re-run it, and not thought twice.)

Re: An AI wolf that preferred suicide over eating sheep

#57

Post by ramtatatam »

I'm not an expert, but story described within the article looks like normal bump on the road to get desired result. When putting together rules for the game researchers did not think that in resulting environment it might be more rewarding to chose observed action than to do what they intended. As much as it looks like nice story, is it not just what researchers encounter on daily basis?

Re: An AI wolf that preferred suicide over eating sheep

#58

Post by OscarCunningham »

TeMPOraL wrote: post #54
foldr wrote: post #51

Earlier quoted context omitted.

Eliezer Yudkowsky tells a long story about an Outcome Pump. Aesop tells a short story about an eagle and a tortoise. The point made is the same, as far as I can see.

Eliezer tells the story that elaborates on why you should be careful what you wish for. Of about a dozen versions of the Eagle and Tortoise story I've just skim-read, none of them really has this as a moral - in each of them, either the Eagle or a Tortoise was an asshole and/or liar and/or lying asshole, so the more valid moral would be, "don't deal with dangerous people" and/or "don't be an asshole" and/or "don't be…

A closer tale might be https://en.wikipedia.org/wiki/The_Sorcerer%27s_Apprentice

Re: An AI wolf that preferred suicide over eating sheep

#59

Post by ajmurmann »

bserge wrote: post #42
Brendinooo wrote: post #39

Earlier quoted context omitted.

I think the point is more about highlighting the fact that AI doesn't share our base assumptions. We wouldn't think to put a huge penalty on dying because humans generally think that death is bad.

Yeah, because we have a -1000 points on death built-in.

Looking at genetic algorithms makes a great comparison. In essence any algorithm in which the wolf commits suicide doesn't make it to the next generation. It's the equivalent of an enormous score penalty and 100% analog to how it works for actual life.

Re: An AI wolf that preferred suicide over eating sheep

#60

Post by morpheos137 »

What distinguishes AI from self-calibrated algorithm?. Neither this "AI" nor the story about it seem too intelligent.

The incentive structure is a two dimensional membrane embeded in a third dimension of "points space."

Obviously if the goal is to maximize total points OR minimize point loss and the absolute value of the gradient toward a mininum loss is greater than the abs gradient toward a maximum gain then the algorithm may prefer the minimum until or if it is selected against by random chance or survivorship bias.

obviously the linear time constraint causes this. a less monotonic, i.e. random, time constraint may have been interesting.

Post reply on HN