Generally capable agents emerge from open-ended play • Hacker News Board

Re: Generally capable agents emerge from open-ended play

#21

Post by marcosdumay » Tue, Jul 27, 2021, 7:32 PM UTC

seph-reed wrote: post #16

modeless wrote: post #7

Agents trained in simulation like this often flail about seemingly randomly, and when they achieve their goals it seems almost accidental. Rather than this being some kind of limitation of the learning algorithm, I think it might be the optimal strategy, and humans would behave that way too if there was no such thing as fatigue or pain. If we want agents to behave more realistically and move with more apparent intent…

I think you're right about pain/fatigue, but scoring them from an external perspective is rather oppressive. And oppression like that is often not conducive to creativity. So perhaps this: instead of goals being binary (wherin no pleasure is derived until fulfillment), they could be on a gradient (so every step that x gets closer to y releases some amount of fulfillment). The fulfillment meter should always be slowly…

Well, it's a good recipe for avoiding burn-out, but you can't have a gradient for all problems, and adding proxies so you get one goes against the principle of letting the AI learn what's best without your biases.

Re: Generally capable agents emerge from open-ended play

#22

Post by [deleted] » Tue, Jul 27, 2021, 7:45 PM UTC

jashmenn wrote: post #14

- "Tag" - shoots other player - "Capture the flag" - shoots other player - "Hide and seek" - shoots other player As colorful as this world is, these capabilities terrify me because they're obviously going to be used as powerful weapons of war. It is a tiny technological leap to install this learning into a Boston Robotics Spot attached to a firearm. I'm pro-tech, pro-crypto, pro-ml and these videos fill me with dread…

[deleted]

Re: Generally capable agents emerge from open-ended play

#23

Post by dmer » Tue, Jul 27, 2021, 7:46 PM UTC

jashmenn wrote: post #14

- "Tag" - shoots other player - "Capture the flag" - shoots other player - "Hide and seek" - shoots other player As colorful as this world is, these capabilities terrify me because they're obviously going to be used as powerful weapons of war. It is a tiny technological leap to install this learning into a Boston Robotics Spot attached to a firearm. I'm pro-tech, pro-crypto, pro-ml and these videos fill me with dread…

I disagree this should have been flagged. The authors do claim the tasks are "general" but they do have a lot in common with each other, including not-so-far-away misuse... which also happened with AlphaDogfight... and that was transferring from playing Go to flying jets. This is clearly not as big of a leap as the author points out.

Aka - notice that none of the agents in this example are folding proteins. They're all engaged in inherently combat-relevant skills. :)

Re: Generally capable agents emerge from open-ended play

#24

Post by woeirua » Tue, Jul 27, 2021, 8:16 PM UTC

modeless wrote: post #11

xcodevn wrote: post #3

"Analysing the agent’s internal representations, we can say that by taking this approach to reinforcement learning in a vast task space, our agents are aware of the basics of their bodies and the passage of time and that they understand the high-level structure of the games they encounter." Wow, really amazing if true. P.S.: After looking into their paper, it's not that impressive. They use agent's internal states (L…

> it's not that impressive. They use agent's internal states (LSTM cells, attention outputs, etc.) to predict whether it is early in the episode, or whether the agent is holding an object. That seems like a decent definition of awareness to me. The agent has learned to encode information about time and its body in its internal state, which then influences its decisions. How else would you define awareness? Qualia or…

By that definition wouldn't a regular RNN or LSTM also possess awareness?

Re: Generally capable agents emerge from open-ended play

#25

Post by pault » Tue, Jul 27, 2021, 8:39 PM UTC

modeless wrote: post #7

Agents trained in simulation like this often flail about seemingly randomly, and when they achieve their goals it seems almost accidental. Rather than this being some kind of limitation of the learning algorithm, I think it might be the optimal strategy, and humans would behave that way too if there was no such thing as fatigue or pain. If we want agents to behave more realistically and move with more apparent intent…

Flailing about seemingly randomly is a good description of how I learn complex software. Blender, Ableton, etc. After a few days I'll reach for the structured educational resources.

Re: Generally capable agents emerge from open-ended play

#26

Post by woeirua » Tue, Jul 27, 2021, 9:12 PM UTC

A couple questions I have after skimming the paper, so forgive me if they were answered somewhere in the 54 page manuscript:

They mention in A.3 that they explicitly reject dynamically generated training worlds/games that collide with their evaluation sets, but do they ensure that dynamic training games are sufficiently "distant" from their evaluation sets regardless of whether or not there's a direct collision? If not, you might still end up training on something quite similar to your test dataset. Figure 27 kind of suggests that might happen for some games given that the vast majority of the held out games have relatively poor transfer performance but a few are really good.

Speaking of Figure 27, while the reward looks good it would have been really nice to show some examples of what these "zero-shot" games look like versus the fine tuned version. Is the gap in the reward between the raw vs fine tuned version significant?

Wouldn't we expect the internal state representation to be more definitive in classifying the state of the agent during the simulation as the agent moves around the environment? From their examples: Figure 20,21, and 22 it almost looks like it either flags the state as "early" or "success." Not sure we're getting the expected performance out of it.

Re: Generally capable agents emerge from open-ended play

#27

Post by dougmwne » Tue, Jul 27, 2021, 9:18 PM UTC

jcelerier wrote: post #6

Tarq0n wrote: post #4

Earlier quoted context omitted.

"Aware" is probably overly anthropomorphized language there. What they mean to say is that all these things have become parameterized within the model.

the main question of course being, aren't we anthropomorphizing ourselves too much ?

When people say this kind of stuff, I wonder whether there might not be philosophical zombies among us.

Re: Generally capable agents emerge from open-ended play

#28

Post by lazide » Tue, Jul 27, 2021, 9:41 PM UTC

jeantherapy wrote: post #15

modeless wrote: post #7

Agents trained in simulation like this often flail about seemingly randomly, and when they achieve their goals it seems almost accidental. Rather than this being some kind of limitation of the learning algorithm, I think it might be the optimal strategy, and humans would behave that way too if there was no such thing as fatigue or pain. If we want agents to behave more realistically and move with more apparent intent…

you seem to reduce it down to pain. how is your notion of "pain" any different from not scoring well and being moved away from?

Some rewards can be worth certain penalties (pain) if within certain thresholds. Exhaustion fatigue can also play in.

You can of course boil them down to a single number, it just produces less nuanced types of decisions/operations, as it can’t differentiate between a cheap, painful, but bountiful choice and a expensive, no pain, mediocre choice.

Re: Generally capable agents emerge from open-ended play

#29

Post by modeless » Tue, Jul 27, 2021, 9:49 PM UTC

woeirua wrote: post #24

modeless wrote: post #11

Earlier quoted context omitted.

> it's not that impressive. They use agent's internal states (LSTM cells, attention outputs, etc.) to predict whether it is early in the episode, or whether the agent is holding an object. That seems like a decent definition of awareness to me. The agent has learned to encode information about time and its body in its internal state, which then influences its decisions. How else would you define awareness? Qualia or…

By that definition wouldn't a regular RNN or LSTM also possess awareness?

I think it would be perfectly reasonable to describe any RNN as being "aware" of information that it learned and then used to make a decision.

"Possess awareness" seems like loaded language though, evoking consciousness. In that direction I'd just quote Dijkstra: "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim."

Re: Generally capable agents emerge from open-ended play

#30

Post by pcl » Tue, Jul 27, 2021, 10:00 PM UTC

modeless wrote: post #29

woeirua wrote: post #24

Earlier quoted context omitted.

By that definition wouldn't a regular RNN or LSTM also possess awareness?

I think it would be perfectly reasonable to describe any RNN as being "aware" of information that it learned and then used to make a decision. "Possess awareness" seems like loaded language though, evoking consciousness. In that direction I'd just quote Dijkstra: "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim."

Ooh, that’s a great quote.

I’d say that it’s no less interesting, either.