This reads as click-bait, here's the original blog post and research paper by DeepMind:
"Understanding Agent Cooperation" https://news.ycombinator.com/edit?id=13635218
Understanding Agent Cooperation
21–30 of 60 posts
Re: Understanding Agent Cooperation
#22Earlier quoted context omitted.
I don't think "in-rules" and "aggressive" are mutually exclusive. It's fair to call blitzing the QB an aggressive move in American football.
You're technically correct, but the football analogy switches context so that the meaning of aggressive is no longer bad. I think the point bencollier49 is trying to make is that we simply gave software a specific set of rules to train it. It doesn't know how we perceive the actions it is performing. The game could be described as two people eating poisonous apples in order to prevent the other person from dying. In…
> we simply gave software a specific set of rules to train it. It doesn't know how we perceive the actions it is performing.
I think the author is making the same point from another angle.
The AI learns what we would consider aggressive moves when conditions favor those moves.
Re: Understanding Agent Cooperation
#23I'm rather worried about the wording used, and AI being created in that context. Do we really not realize what we're doing? AI is not magic, it's not free from fundamental math, it's not free from corruption. It's just going to multiply it that much more.
Any AI that has been programmed to highly value winning is not going to be very cooperative. For it to be cooperative, especially in situations that simulate survival, it needs to have higher ideals than winning, just like humans. It needs to be able to see and be aware of the big picture. You don't need to look at AI for that, you can just look at the world.
Development of AI's of this nature will just lead to a super-powered Moloch. Cooperative ethics is a highly advanced concept, it's not going to show up on its own from mere game theory without a lot of time.
Re: Understanding Agent Cooperation
#24The AI can minimize loss / maximize fitness by either moving to look for additional resources, or fire a laser. Turns out that when resources are scarce, the optimal move is to knock the opponent away. I think this tells us more about the problem space than the AI itself; it's just optimizing for the specific problem.
I think you're right, had the AI (like the second game) had the incentive to maximize the well being of the cooperating actors, rather than itself, the outcome would be different.
But if advanced AI is being developed in a capitalist economy by independent actors, it seems most likely the incentives will be anything other than optimizing the output for the individual outcome.
If that AI finds a way to "hurt" the other actor, there could be some major boat load of unintended consequences.
Re: Understanding Agent Cooperation
#25I'm rather worried about the wording used, and AI being created in that context. Do we really not realize what we're doing? AI is not magic, it's not free from fundamental math, it's not free from corruption. It's just going to multiply it that much more. Any AI that has been programmed to highly value winning is not going to be very cooperative. For it to be cooperative, especially in situations that simulate surviv…
Cooperative ethics arise immediately in the Prisoner's Dilemma merely by adding an unknown number of iterations to the game. The most efficient strategy is a version of tit-for-tat.
Re: Understanding Agent Cooperation
#26I'm rather worried about the wording used, and AI being created in that context. Do we really not realize what we're doing? AI is not magic, it's not free from fundamental math, it's not free from corruption. It's just going to multiply it that much more. Any AI that has been programmed to highly value winning is not going to be very cooperative. For it to be cooperative, especially in situations that simulate surviv…
Cooperative ethics arise immediately in the Prisoner's Dilemma merely by adding an unknown number of iterations to the game. The most efficient strategy is a version of tit-for-tat.
I'm assuming you're referring to something like this: https://egtheory.wordpress.com/2015/03/02/ipd/
I think we shouldn't confuse efficient strategies with the chosen strategies. What causes Moloch is the inability to see the big picture, to see outside of the self in the collective (maybe Buddhism has a point).
An efficient strategy may very well be something we'd prefer, such as tit-for-tat. But is that the strategy we choose? Looking at the long history of evolution, I'd say no.
Re: Understanding Agent Cooperation
#27Earlier quoted context omitted.
Cooperative ethics arise immediately in the Prisoner's Dilemma merely by adding an unknown number of iterations to the game. The most efficient strategy is a version of tit-for-tat.
I'm assuming you're referring to something like this: https://egtheory.wordpress.com/2015/03/02/ipd/ I think we shouldn't confuse efficient strategies with the chosen strategies. What causes Moloch is the inability to see the big picture, to see outside of the self in the collective (maybe Buddhism has a point). An efficient strategy may very well be something we'd prefer, such as tit-for-tat. But is that the strateg…
> Looking at the long history of evolution, I'd say no.
This entire lecture series on Human Behavioral Biology is worth watching from the beginning, but I've linked to a moment where Sapolsky describes tit-for-tat strategies arising in animals. First example: Vampire Bats.
[]: https://www.youtube.com/watch?v=Y0Oa4Lp5fLE&feature=youtu.be...
Re: Understanding Agent Cooperation
#28Why is this done on such a small level? I would have thought that with systems now in place that evolutionary game theory could be done in simulations on such a much larger scale (say 7bn agents +) ... if anything AI systems should be able to determine if certain strategies work (like items like blocking resources - such as a case of geopolitical theory) so see what cooperations occur at that level. Still amazing work but it should be applied to a larger scale for real meaning. More eager to see how RL applied to RTS games will explore and develop strategies more than anything.
Re: Understanding Agent Cooperation
#29Not entirely spawned by this article, but the whole genre and some other comments on HN by other users: I wonder if part of the "mystery" of cooperation in these simulations is that these people keep investigating the question of cooperation using simulations too simplistic to model any form of trade. A fundamental of economics 101 is that valuations for things differ for different agents. Trade ceases to exist in a world where everybody values everything exactly the same, because the only trade that makes any sense is to two trade two things of equal value, and even then, since the outcome is a wash and neither side obtains any value from it, why bother? I'm not sure the simulation hasn't been simplified to the point that the phenomena we're trying to use the simulation to explain are not capable of manifesting within the simulation.
I'm not saying that Trade Is The Answer. I would be somewhat surprised if it doesn't form some of the solution eventually, but that's not the argument I'm making today. The argument I'm making is that if the simulation can't simulate trade at all, that's a sign that it may have been too simplified to be useful. There are probably other things you could say that about; "communication" being another one. The only mechanism for communication being the result of iteration is questionable too, for instance. Obviously in the real world, most cooperation doesn't involve human speech, but a lot of ecology can be seen to involve communication, if for no other reason than you can't have the very popular strategy of "deception" if you don't have "communication" with which to deceive.
Which may also explain the in-my-opinion overpopular and excessively studied "Prisoner's Dilemma", since it has the convenient characteristic of explicitly writing communication out of it. I fear its popularity may blind us to the fact that it wasn't ever really meant to be the focus of study of social science, but more a simplified word problem for game theory. Studying a word problem over and over and over may be like trying to understand the real world of train transportation systems by repeatedly studying "A train leaves from Albuquerque headed towards Boston at 1pm on Tuesday and a train leaves from Boston headed towards Albuquerque at 3pm on Wednesday, when do they pass each other?" over and over again.
(Or to put it really simply in machine learning terms, what's the point of trying to study cooperation in systems whose bias does not encompass cooperation behaviors in the first place?)
Re: Understanding Agent Cooperation
#30Is it just me, or is this article extremely light on content? The core of it seems to be
> sequential social dilemmas, and us[ing] artificial agents trained by deep multi-agent reinforcement learning to study [them]