Finally Auotcirricula gets some love! Discussed in some detail in https://www.talkrl.com/episodes/natasha-jaques
Natasha Jaques explains the idea at about 39:50
Emergent Tool Use from Multi-Agent Interaction
21–30 of 64 posts
Re: Emergent Tool Use from Multi-Agent Interaction
#22What is the size of these “strategies”, measured in weights,bytes, or whatever measurement you look at?
1.6 million parameters. There are some details in section 5 and appendix B.7 of https://d4mucfpksywv.cloudfront.net/emergent-tool-use/paper/...
Re: Emergent Tool Use from Multi-Agent Interaction
#23What is the size of these “strategies”, measured in weights,bytes, or whatever measurement you look at?
1.6 million parameters. There are some details in section 5 and appendix B.7 of https://d4mucfpksywv.cloudfront.net/emergent-tool-use/paper/...
Thanks! I like that section on how batch size affects convergence. I wonder how parameter size limits would similarly affect which Stages could be reached. I would not be surprised if you could hit those stages with 100x or fewer params.
Re: Emergent Tool Use from Multi-Agent Interaction
#24This is just adjoint functors. Pls work out automatic integration. Dual numbers is where the path starts.
Re: Emergent Tool Use from Multi-Agent Interaction
#25I wonder if it’s possible to incorporate a monkey see monkey do aspect to the learning algorithm that could observe human’s playing the game and incorporate that information into its models?
Yes, it's called imitation learning and is a subfield of reinforcement learning. The problem is that even a small error could gradually accumulate and cause the sequence of actions to diverge. RL agents learn not just how to act in a given situation but also to evaluate possible actions, situations and even to model the environment. That way they can adapt dynamically instead of diverging from the optimal actions.
Re: Emergent Tool Use from Multi-Agent Interaction
#26I wonder if it’s possible to incorporate a monkey see monkey do aspect to the learning algorithm that could observe human’s playing the game and incorporate that information into its models?
[deleted]
Re: Emergent Tool Use from Multi-Agent Interaction
#27Looks awesome. I tried coding up a multi-agent system for my CS degree and it was incredibly complicated. I was trying to implement an algorithm I found to give each agent emotions of fear, anger, happiness and sadness in order to change their behaviours... it was way more difficult than I expected but you can read more about it here if you're also interested in this stuff. The 3D graphics in this example are way cooler than my 2D shapes.
https://medium.com/@dshields/working-with-emotional-models-i...
Re: Emergent Tool Use from Multi-Agent Interaction
#28I have a friend who observed similar emergent behavior in an a-life (gene-based from what I understand) simulation he created, in an environment of "tanks in a maze" (or something like that).
The "genes" consisted a simplified assembler (run on a VM) that could describe a program the tank would use to control itself - it could sense other tanks within line-of-sight to a certain degree, it could sense walls, it could fire its cannon, move in a particular direction, sense when another tank had a bearing (cannon pointed) on itself, etc.
He set up 100 random tanks (with random "genes"/programs) and let the simulation run. Top scorers (who had the most kills) would be used to seed the next "generation", using a form of sexual "mating" and (pseudo-) random mutation. Then that generation would run.
He said he ran the simulation for days at a time. One day he noticed something odd. He started to notice that certain tanks had "evolved" the means to "teleport" from location to location on the map. He didn't design this possibility in - what had happened was (he later determined) that a bug he had left in the VM was being exploited to allow the tanks to instantaneously move within their environment. He thought it was interesting, so he left it as-is and let the simulation continue.
After a long period of running, my friend then noticed something very odd. Some tanks were "wiggling" their turrets - other tanks would "wiggle" in a similar fashion. After a while all he could deduce was that in some manner, they were communicating with each other, similar to "bee dancing", and starting to form factions against each other...
...it was at that point he decided things were getting much too strange, and he stopped the experiment.
Sadly, he no longer has a copy of this software, but I believe his story, simply because I have seen quite a bit of other code and have worked closely with him on various projects since (as an adult) to know that such a system was well within his capability of creating.
At the time, he was probably only 16 or 17 years old, the computer was a 386, and this was sometime in the early 1990s. I believe the software was likely a combination of QuickBasic 4.5 and 8086 assembler running under DOS, as that was his preferred environment at the time.
I've often considered recreating the experiment, using today's technology, just to see what would happen (at the time he related this to me, as an adult, he asked me how difficult it would be to make a more physical version of this "game"; I'm still not sure if he meant scale model tanks, or full-sized - knowing him, though, he would have loved to play with the latter).
Re: Emergent Tool Use from Multi-Agent Interaction
#29The animations are nice, compared to a default visualization with dots and lines moving around. Was this done just for the public release, or was it worth it to researchers to have an eye-pleasing visualization while doing the experiments?
The environment was actually an important part of the project. It does physics simulation. Having such a 'realistic' environment allowed the agents to discover all sorts of cheats (they appear at the end of the article).
Re: Emergent Tool Use from Multi-Agent Interaction
#30I love how the 3D visualization and game selection make their research immediately relatable - right down to the cute little avatars!
"We’ve shown that agents can learn sophisticated tool use in a high fidelity physics simulator"
I always suspected to evolve intelligence you need an environment rich in complexity. Intelligence we're familiar with (e.g. humans) evolved in a primordial soup packed with possibilities and building blocks (e.g. elaborate rules of physics, amino acids, etc). It's great to see this concept being explored.
It reminds me of Adrian Thompson's experiments in the 90's running generational genetic algorithms on a real FPGA instead of mere simulations [1].
After 5000 generations he coaxed out a perfect tone recognizer. He was able to prune 70% of the circuit (lingering remnants of earlier mutations?) to find it still worked with only 32 gates - an unimaginable feat! Engineers were baffled when they reverse-engineered what remained: if I recall correctly, transistors were run outside of saturation mode, and EM effects were being exploited between adjacent components. In short, the system took a bunch of components designed for digital logic but optimized them using the full range of analog quirks they exhibited.
More recent attempts to recreate his work have reportedly been hampered by modern FPGA's which make it harder to exploit those effects as they don't allow reconfiguration at the raw wiring level [2].
In Thompson's own words:
"Evolution has been free to explore the full repertoire of behaviours available from the silicon resources provided, even being able to exploit the subtle interactions between adjacent components that are not directly connected.... A 'primordial soup' of reconfigurable electronic components has been manipulated according to the overall behavior it exhibits"
---
[1] Paper: http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=669...
Less technical article: https://www.damninteresting.com/on-the-origin-of-circuits/
[2] https://www.reddit.com/r/MachineLearning/comments/2t5ozk/wha...