Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Generally capable agents emerge from open-ended play

deepmind.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Generally capable agents emerge from open-ended play

#41

Post by chrisweekly »

gryn wrote: post #18
nostrademons wrote: post #9

Earlier quoted context omitted.

Watching my 3-year-old, I suspect humans do behave this way, and without regards to pain or fatigue. He'll hit himself on the head with duplos or bang his head against the wall just to see what happens. The pain is just one more signal for reinforcement learning. I recall a paper in a non-CS journal (psych or neuroscience) that posited that the optimal way to gain large quantities of information about an uncertain en…

was that paper about the complexities of the exploration/exploitation trade-off in different environments ?

Your phrasing it this way reminds me of the excellent book Algorithms to Live By. Highly recommended!

Re: Generally capable agents emerge from open-ended play

#42

Post by whimsicalism »

eutectic wrote: post #10
modeless wrote: post #7

Agents trained in simulation like this often flail about seemingly randomly, and when they achieve their goals it seems almost accidental. Rather than this being some kind of limitation of the learning algorithm, I think it might be the optimal strategy, and humans would behave that way too if there was no such thing as fatigue or pain. If we want agents to behave more realistically and move with more apparent intent…

It's proven that if you want polynomial sample complexity in the size of the state space you need directed exploration. The algorithms 'flail' because they are initialized with random policies.

Couldn't any directed exploration also be produced as the result of random flailing?

Or is this average sample complexity?

Re: Generally capable agents emerge from open-ended play

#43

Post by CPLX »

nostrademons wrote: post #9
modeless wrote: post #7

Agents trained in simulation like this often flail about seemingly randomly, and when they achieve their goals it seems almost accidental. Rather than this being some kind of limitation of the learning algorithm, I think it might be the optimal strategy, and humans would behave that way too if there was no such thing as fatigue or pain. If we want agents to behave more realistically and move with more apparent intent…

Watching my 3-year-old, I suspect humans do behave this way, and without regards to pain or fatigue. He'll hit himself on the head with duplos or bang his head against the wall just to see what happens. The pain is just one more signal for reinforcement learning. I recall a paper in a non-CS journal (psych or neuroscience) that posited that the optimal way to gain large quantities of information about an uncertain en…

This is certainly insightful. But I think a complete explanation would have to include the power of narrative as well. Somehow.

It seems obvious and intuitive that the human brain is in some way wired to recognize patterns in data and weave narratives around those patterns, and also that it’s possible to skip the data part and convey intelligence through narrative and metaphor.

Re: Generally capable agents emerge from open-ended play

#45

Post by galdosdi »

lifefeed wrote: post #35
modeless wrote: post #7

Agents trained in simulation like this often flail about seemingly randomly, and when they achieve their goals it seems almost accidental. Rather than this being some kind of limitation of the learning algorithm, I think it might be the optimal strategy, and humans would behave that way too if there was no such thing as fatigue or pain. If we want agents to behave more realistically and move with more apparent intent…

"Throwing shit against the wall and seeing what works" is a widely used strategy in all walks of life. If you ever go to a doctor for a problem, and their first solution doesn't work, you're in for a wild ride of an eduction.

This comment speaks to me SO HARD.

I'm on my fifth antibiotic and this one is making me feel like I'm on LSD. Really. And both Dr. Google and my real doctor agree this is somewhat normal or okay, but it's still deeply disturbing.

Hilariously, I'm taking it to reverse the serious side effects of the fourth antibiotic.

Just to give an example of a wild ride you can end up on when doctors don't figure it out the first time.

Doctors have good domain knowledge, but actual diagnosis skill varies greatly.

Don't fucking get sick. That's your best shot.

Re: Generally capable agents emerge from open-ended play

#46

Post by anigbrowl »

exporectomy wrote: post #33
jashmenn wrote: post #14

- "Tag" - shoots other player - "Capture the flag" - shoots other player - "Hide and seek" - shoots other player As colorful as this world is, these capabilities terrify me because they're obviously going to be used as powerful weapons of war. It is a tiny technological leap to install this learning into a Boston Robotics Spot attached to a firearm. I'm pro-tech, pro-crypto, pro-ml and these videos fill me with dread…

Don't worry about it. We can already do far worse using either humans or land mines or nuclear bombs or whatever. People are the real danger.

This completely misses the point. The problem isn't that people do worse, but that automating warfare changes the cost function. Technologically superior and determined enemies can still be dissuaded by incurring significant numbers of casualties or losses to their infrastructure. A small number of casualties stiffens resolve, a large number of casualties or a never ending trickle of them eventually breaks or wears it away.

If military actors can reliably change outcomes by the relatively low-cost expedient of throwing in autonomous weapons platforms that cost about as much as a washing machine, they will and they'll do it at scale, and (in the short term at least) their political backers will cheer and get off on it. In the longer run it will lead to a considerable increase in terrorism against the technologically advanced power.

Sure, people ultimately make these decisions and deploy such technologies, but so what? it's not like that can change in any way because you can't take people out of the equation and you can't just wish away political forces by pinning the blame on select individuals. Rather than retreating into truisms, it's more important to assess the impact of this emerging force multiplier and develop countermeasures.

Re: Generally capable agents emerge from open-ended play

#47

Post by phreeza »

jcims wrote: post #5
Tarq0n wrote: post #4

Earlier quoted context omitted.

"Aware" is probably overly anthropomorphized language there. What they mean to say is that all these things have become parameterized within the model.

It would be interesting to see what would happen if they added social dynamics between the agents...like some space for theory of mind (what is that agent thinking), mimicry, communication, etc.

There is also some other work from deepmind in this direction: https://deepmind.com/research/publications/machine-theory-mi...

Re: Generally capable agents emerge from open-ended play

#48

Post by nradov »

galdosdi wrote: post #45
lifefeed wrote: post #35

Earlier quoted context omitted.

"Throwing shit against the wall and seeing what works" is a widely used strategy in all walks of life. If you ever go to a doctor for a problem, and their first solution doesn't work, you're in for a wild ride of an eduction.

This comment speaks to me SO HARD. I'm on my fifth antibiotic and this one is making me feel like I'm on LSD. Really. And both Dr. Google and my real doctor agree this is somewhat normal or okay, but it's still deeply disturbing. Hilariously, I'm taking it to reverse the serious side effects of the fourth antibiotic. Just to give an example of a wild ride you can end up on when doctors don't figure it out the first t…

Sometimes regardless of diagnostic skill the only way to make an accurate diagnosis is to try various treatments and see which one is effective. Many conditions have no definitive test.

Re: Generally capable agents emerge from open-ended play

#49

Post by im3w1l »

Jarwain wrote: post #38
modeless wrote: post #7

Agents trained in simulation like this often flail about seemingly randomly, and when they achieve their goals it seems almost accidental. Rather than this being some kind of limitation of the learning algorithm, I think it might be the optimal strategy, and humans would behave that way too if there was no such thing as fatigue or pain. If we want agents to behave more realistically and move with more apparent intent…

I've been thinking about an "energy" idea, where actions consume energy but completing goals replenishes it

I think the issue with

   goals - energy used
is that it requires tuning to make sure that completing goals is better than just idling. A variant that might work better is to optimize "mileage".

    goals / (energy used + epsilon)
Where epsilon has two purposes: it prevents division by zero, and ensures that more goals is (marginally) better than less

Re: Generally capable agents emerge from open-ended play

#50

Post by d110af5ccf »

anigbrowl wrote: post #46
exporectomy wrote: post #33

Earlier quoted context omitted.

Don't worry about it. We can already do far worse using either humans or land mines or nuclear bombs or whatever. People are the real danger.

This completely misses the point. The problem isn't that people do worse, but that automating warfare changes the cost function. Technologically superior and determined enemies can still be dissuaded by incurring significant numbers of casualties or losses to their infrastructure. A small number of casualties stiffens resolve, a large number of casualties or a never ending trickle of them eventually breaks or wears i…

You left out the bit where the various superpowers inevitably have to try using the shiny new technology against their rivals because it's never been tried so we can't be certain it's a bad idea. That's the bit that worries me the most - I'd rather do without a rerun of World War 1.

> it's more important to assess the impact of this emerging force multiplier and develop countermeasures

What is there to do other than develop your own equivalent systems though?

Post reply on HN