DeepMind: A Generalist Agent
deepmind.com
DeepMind: A Generalist Agent
1–10 of 345 posts
Re: DeepMind: A Generalist Agent
#2Abstract: Inspired by progress in large-scale language modeling, we apply a similar approach towards building a
single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a
multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights
can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based
on its context whether to output text, joint torques, button presses, or other tokens. In this report we
describe the model and the data, and document the current capabilities of Gato.
Direct Link to Paper: https://dpmd.ai/Gato-paper
Re: DeepMind: A Generalist Agent
#3If I'm following correctly, they trained a single model with multiple training paradigms and then the single model could perform token predictions for multiple dissimilar token sequences for specific tasks. Seems like it is a straightforward result.
Re: DeepMind: A Generalist Agent
#4"The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens."
This is rather mind blowing. Does it also mean that the generalist network is smaller than the sum of all specialist networks that are equivalent? Even if not, I find the idea that a single network can be used for such diverse tasks at all highly fascinating.
Re: DeepMind: A Generalist Agent
#5I’m not sure how to word my excitement about the progress we see in AI research in the last years. If you haven’t read it, give Tim Urbans classic piece a slice of your attention: https://waitbutwhy.com/2015/01/artificial-intelligence-revol...
It’s a very entertaining read from a couple of years ago (I think I’ve read it in 2017), and man, have things happened in the field since then. If feels like things truly start coming together. Transformers and then some incremental progress look like a very, very promising avenue. I deeply wonder in which areas this will shape the future more than we are able to anticipate beforehand.
Re: DeepMind: A Generalist Agent
#6"The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens." This is rather mind blowing. Does it also mean that the generalist network is smaller than the sum of all specialist networks that are equivalent? Even if not, I find the idea that a single…
Many networks just predict the next integer in a sequence of integers. It sounds like this model identifies what category of problem a sequence of integers falls into and then makes an accurate prediction for that sequence, as you would expect given what it was trained on.
Re: DeepMind: A Generalist Agent
#7Abstract: Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding base…
> we refer to as Gato
First, humanity built enormous statues worshiping cats.
Then, we let cats populate the largest amount of "image-bits" on the Internet.
Now, we name the next closest thing to general AI after them.
These damn felines sure are mysterious.
Re: DeepMind: A Generalist Agent
#8This sounds exciting, but the example outputs look quite bad. E.g. from the interactive conversation sample:
> What is the capital of France? > Marseille
And many of the generated image captions are inaccurate.
Re: DeepMind: A Generalist Agent
#9"The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens." This is rather mind blowing. Does it also mean that the generalist network is smaller than the sum of all specialist networks that are equivalent? Even if not, I find the idea that a single…
I don't find it surprising that a single network can do all those things with appropriate formatting of the data. In itself it just means the network has a large enough capacity to learn all the different tasks.
The interesting questions imo, which they studied, is what kind of added generalization takes place by learning across the different tasks. For example, does learning multiple tasks make it better at a given task than a model that is just trained for one task, and can it generalize to new tasks (out of distribution).
They looked at how it performed on held out tasks (see fig 9 in the paper). I'm still getting my head around the result though so couldn't summarize their finding yet.
Edit: the paper is here https://storage.googleapis.com/deepmind-media/A%20Generalist...
There is currently another submission on the front page that links to it directly.
Re: DeepMind: A Generalist Agent
#10given that the same model can both:
1. tell me about a cat (given a prompt such as "describe a cat to me")
2. recognize a cat in a photo, and describe the cat in the photo
does the model understand that a cat that it sees in an image is related to a cat that it can describe in natural language?
As in, are these two tasks (captioning an image and replying to a natural language prompt) so distinct that a "cat" in an image excites different neurons than a "cat" that I ask it about? Or is there overlap? Or we don't know :)
I wonder if you could mix the type of request. Like, provide a prompt that is both text and image. Such as "Here is a picture of a cat. Explain what breed of cat it is and why you think so." Possibly this is too advanced for the model but the idea makes me excited.