31–40 of 223 posts
Similar story of unexpected AI outcomes... As part of my PhD research, I created a simplified Pac-Man style game where the agent would simply try to stay alive as long as possible whilst being chased by the 3 ghosts. The agent was un-motivated and understood nothing about the goal, but was optimising for maximising its observable control over the world (avoiding death is a natural outcome of this). I spent sometime t…
Isn't this just a cock up with incentives? If they'd put a -100 score on dying it would have sorted itself out pretty quick.
Isn't this just a cock up with incentives? If they'd put a -100 score on dying it would have sorted itself out pretty quick.
If you add your penalty, and a deficit of nearby sheep, you'd expect a trifurcation of strategy: hoarders that consume the nearby sheep immediately, explorers that bet on sheep further afield, and suicides from those that have evaluated the -100 penalty to still be optimal.
https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...
From https://deepmindsafetyresearch.medium.com/specification-gami...
Isn't this just a cock up with incentives? If they'd put a -100 score on dying it would have sorted itself out pretty quick.
The issue with AI safety and unanticipated AI outcomes in general is that it’s always just a cock-up with incentives. It’s easy to sort out in narrowly specified areas, but an extremely hard problem as the tasks become more general.
Here is the full video also linked at the bottom. It also shows the one that trained longer that the wolves start successfully hunting the sheep after more training examples.
Reminds me of the old essay by 'Eliezer: "The Hidden Complexity of Wishes". https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden... In it, there is a thought experiment of having an "Outcome Pump", a device that makes your wishes come true without violating laws of physics (not counting the unspecified internals of the device), by essentially running an optimization algorithm on possible futures. As the essay…
Isn't this just a cock up with incentives? If they'd put a -100 score on dying it would have sorted itself out pretty quick.
I think a major takeaway here is that balancing a reward system to reward more than a single behavior is really hard - it's easy to tip the scales so one behavior completely dominates all others. It's an interesting lens to use to look at the heuristic reward system humans have built in (hunger, fear, desire, etc). This tends to have an adaptation/numbing effect, where repeated rewards of the same type tend to have d…