Voyager: An Open-Ended Embodied Agent with LLMs
voyager.minedojo.org
Voyager: An Open-Ended Embodied Agent with LLMs
1–10 of 28 posts
Re: Voyager: An Open-Ended Embodied Agent with LLMs
#2TLDR. An AI system with an IQ of 110-130, with some careful prompting can generate code to play Minecraft through an API.
Re: Voyager: An Open-Ended Embodied Agent with LLMs
#3TLDR. An AI system with an IQ of 110-130, with some careful prompting can generate code to play Minecraft through an API.
>130 IQ
>mines straight up and down
Re: Voyager: An Open-Ended Embodied Agent with LLMs
#4I wonder how many prompts this uses in a minute.
Interestingly, they mod the server so that the game pauses while waiting for a response from GPT-4. That's a nice way to get around the delays.
Re: Voyager: An Open-Ended Embodied Agent with LLMs
#5This is very cool despite the most important caveat:
“Note that we do not directly compare with prior methods that take Minecraft screen pixels as input and output low-level controls [54–56]. It would not be an apple-to-apple comparison, because we rely on the high-level Mineflayer [53] API to control the agent. Our work’s focus is on pushing the limits of GPT-4 for lifelong embodied agent learning, rather than solving the 3D perception or sensorimotor control problems. VOYAGER is orthogonal and can be combined with gradient-based approaches like VPT [8] as long as the controller provides a code API.”
Re: Voyager: An Open-Ended Embodied Agent with LLMs
#6This is kind of amazing given that obviously, GPT-4 never contained such tasks and data.
I think it puts an end to the claim that "language models are only stochastic parrots and cannot do any reasoning".
No, this is 100% a form of reasoning and furthermore, learning that is more similar to how humans learn (gradient-less).
I still don't understand it and it blows my mind - how such properties emerge just from compressing the task of next word prediction. (Yes, I know this is oversimplification, but not a misleading one).
Re: Voyager: An Open-Ended Embodied Agent with LLMs
#7TLDR. An AI system with an IQ of 110-130, with some careful prompting can generate code to play Minecraft through an API.
>130 IQ >mines straight up and down
I’m not sure what is the point that you are making. GPT-4 does tend to pass various IQ tests with the scores in the range of 110 to 130, with outliers between 90 to 150.
Re: Voyager: An Open-Ended Embodied Agent with LLMs
#8This is kind of amazing given that obviously, GPT-4 never contained such tasks and data. I think it puts an end to the claim that "language models are only stochastic parrots and cannot do any reasoning". No, this is 100% a form of reasoning and furthermore, learning that is more similar to how humans learn (gradient-less). I still don't understand it and it blows my mind - how such properties emerge just from compre…
While I do believe LLMs can perform some reasoning, I'm not sure this is the best example as all the reasoning you would ever need for Minecraft is well contained in the data set used to train it. A lot has been written about minecraft.
To me, it would be more convincing if they developed an enterly new game with somewhat novel and arbitrary rules and saw if the embodied agent could learn this game.
Re: Voyager: An Open-Ended Embodied Agent with LLMs
#9This is kind of amazing given that obviously, GPT-4 never contained such tasks and data. I think it puts an end to the claim that "language models are only stochastic parrots and cannot do any reasoning". No, this is 100% a form of reasoning and furthermore, learning that is more similar to how humans learn (gradient-less). I still don't understand it and it blows my mind - how such properties emerge just from compre…
Looking at the paper, as I understand it they're using Mineflayer https://github.com/PrismarineJS/mineflayer and passing parts of the state of the game as JSON to the LLM that are used for code generation to complete tasks.
> I still don't understand it and it blows my mind - how such properties emerge just from compressing the task of next word prediction.
The Mineflayer library is very popular, so all the relevant tasks are likely already extant in the training data.
Re: Voyager: An Open-Ended Embodied Agent with LLMs
#10Earlier quoted context omitted.
>130 IQ >mines straight up and down
I’m not sure what is the point that you are making. GPT-4 does tend to pass various IQ tests with the scores in the range of 110 to 130, with outliers between 90 to 150.
It's a joke about playing the game "right." Mining straight up/down is a rather suboptimal strategy, as:
- mining straight up means you either seal your path behind you, or are limited how high up you can go
- mining straight down likely traps you in a pit
- mining straight down far enough can drop you straight into lava, as many Minecraft players learn early on