Emergent Tool Use from Multi-Agent Interaction • Hacker News Board

Re: Emergent Tool Use from Multi-Agent Interaction

#31

Post by visarga » Tue, Sep 17, 2019, 7:35 PM UTC

brianpgordon wrote: post #11

This is incredible. The various emergent behaviors are fascinating. I remember being amazed a decade ago by the primitive graphics in artificial life simulators like Polyworld: https://en.wikipedia.org/wiki/Polyworld https://www.youtube.com/watch?v=_m97_kL4ox0&t=9m43s It seems that OpenAI has a great little game simulated for their agents to play in. The next step to make this even cooler would be to use physical, ro…

> The next step to make this even cooler would be to use physical, robotic agents learning to overcome challenges in real meatspace!

That's one of the main challenges - how to learn safely and with fewer than millions of trials, so it can be feasible to do in the real world.

Re: Emergent Tool Use from Multi-Agent Interaction

#32

Post by boardwaalk » Tue, Sep 17, 2019, 7:56 PM UTC

visarga wrote: post #29

tlb wrote: post #20

The animations are nice, compared to a default visualization with dots and lines moving around. Was this done just for the public release, or was it worth it to researchers to have an eye-pleasing visualization while doing the experiments?

The environment was actually an important part of the project. It does physics simulation. Having such a 'realistic' environment allowed the agents to discover all sorts of cheats (they appear at the end of the article).

They're talking about the visualization, not the physics. The agents aren't getting visual input. That would make things much, much slower.

Re: Emergent Tool Use from Multi-Agent Interaction

#33

Post by vertoc » Tue, Sep 17, 2019, 8:12 PM UTC

boardwaalk wrote: post #32

visarga wrote: post #29

Earlier quoted context omitted.

The environment was actually an important part of the project. It does physics simulation. Having such a 'realistic' environment allowed the agents to discover all sorts of cheats (they appear at the end of the article).

They're talking about the visualization, not the physics. The agents aren't getting visual input. That would make things much, much slower.

Right but to fully understand what's going on you need to also visualize the physics in a 3D world - just dots and lines and squares wouldn't fully show what's going on. This may be close to the simplest visualization that made sense

Re: Emergent Tool Use from Multi-Agent Interaction

#34

Post by PhasmaFelis » Tue, Sep 17, 2019, 8:27 PM UTC

minimaxir wrote: post #10

corey_moncure wrote: post #7

One plausible, perhaps optimal strategy in the second arena is for the hiders to build a shelter around the seekers and lock them in place, circumventing the whole cat and mouse over ramps and ramp surfing (which the seekers would never be able to access). I wonder why this strategy is not arrived at.

There are multiple seekers, and the seekers may not be placed close together.

At one point in the video, it looked like a hider moving a object past a frozen seeker jostled the seeker with it. I wonder if it's possible to use the objects to push seekers together, then "jail" them.

Re: Emergent Tool Use from Multi-Agent Interaction

#35

Post by Falling3 » Tue, Sep 17, 2019, 8:32 PM UTC

ismail wrote: post #15

bryanrasmussen wrote: post #12

Earlier quoted context omitted.

hmm, yes in the story I'm envisioning the AIs don't wipe out humanity because they have achieved sentience, but just because it turns out killing all humans is an optimizing component of solving some other problem.

Asimov 3 rules as the final policy when making decisions should sort this problem out. This assumes that the rules cannot be changed by the AI.

I've always been very incredulous that there would be any possibility of taking something sufficiently complex to be considered an AGI and hard-coding anything like the 3 rules into it.

Re: Emergent Tool Use from Multi-Agent Interaction

#36

Post by Inufu » Tue, Sep 17, 2019, 8:42 PM UTC

Nice visualizations and explanation!

You might want to make it clearer that the agents don't actually receive any visual observations, but rather directly the xy positions of all other agents and objects.

This also seems very similar to "Capture the Flag: the emergence of complex cooperative agents" (https://deepmind.com/blog/article/capture-the-flag-science)?

Regarding the conclusion:

> We’ve provided evidence that human-relevant strategies and skills, far more complex than the seed game dynamics and environment, can emerge from multi-agent competition and standard reinforcement learning algorithms at scale. These results inspire confidence that in a more open-ended and diverse environment, multi-agent dynamics could lead to extremely complex and human-relevant behavior.

This has been well established for a while already, e.g. the DeepMind Capture the Flag paper above, AlphaGo discovering the history of Go openings and techniques as it learns from playing itself, AlphaZero doing the same for chess, etc.

Re: Emergent Tool Use from Multi-Agent Interaction

#37

Post by homieg33 » Tue, Sep 17, 2019, 9:17 PM UTC

visarga wrote: post #25

homieg33 wrote: post #13

I wonder if it’s possible to incorporate a monkey see monkey do aspect to the learning algorithm that could observe human’s playing the game and incorporate that information into its models?

Yes, it's called imitation learning and is a subfield of reinforcement learning. The problem is that even a small error could gradually accumulate and cause the sequence of actions to diverge. RL agents learn not just how to act in a given situation but also to evaluate possible actions, situations and even to model the environment. That way they can adapt dynamically instead of diverging from the optimal actions.

Interesting, ideally it uses the observed human behaviors to seed/inform it’s own attempts as a shortcut to advanced behavior without the many millions of generations needed.

Re: Emergent Tool Use from Multi-Agent Interaction

#38

Post by gdb » Tue, Sep 17, 2019, 9:42 PM UTC

Inufu wrote: post #36

Nice visualizations and explanation! You might want to make it clearer that the agents don't actually receive any visual observations, but rather directly the xy positions of all other agents and objects. This also seems very similar to "Capture the Flag: the emergence of complex cooperative agents" ( https://deepmind.com/blog/article/capture-the-flag-science )? Regarding the conclusion: > We’ve provided evidence tha…

Good catch! Will update the post to be explicit that there are many pre-existing awesome results in this vein.

Re: Emergent Tool Use from Multi-Agent Interaction

#39

Post by The_rationalist » Tue, Sep 17, 2019, 10:17 PM UTC

Am I misunderstanding something?

Instead of teaching the "AI" intelligent rules or rules for creating rules for maximising their goals. They teach them nothing, which means they have 0 usable high level knowledge. And the "AI" pure bruteforce for finding empirically best solutions for this ridiculously simple universe.

How is that advancing research? This is just a showcase of what modern hardware can do, and also a showcase of how far we are from teaching intelligence. My brain understand the semantics of this universe and would have been able to find most strategies without simulating the game more than once in my head. So definitely this is a showcase of how far (bruteforce is like step 0) we (or at least openAI) are from making AGI.

Re: Emergent Tool Use from Multi-Agent Interaction

#40

Post by breck » Tue, Sep 17, 2019, 10:28 PM UTC

vertoc wrote: post #33

boardwaalk wrote: post #32

Earlier quoted context omitted.

They're talking about the visualization, not the physics. The agents aren't getting visual input. That would make things much, much slower.

Right but to fully understand what's going on you need to also visualize the physics in a 3D world - just dots and lines and squares wouldn't fully show what's going on. This may be close to the simplest visualization that made sense

The visualizations look great, but wouldn't run on an N64, which had many physics games. I'm wondering the same thing as the OP--was this advanced level of graphics used during the research, or was the styling added after the fact for readers? A low res visualization seems like it would do the job equally well, but maybe not. Curious what they are finding and whether there are benefits to having a great looking visualization during the EDA phase.