A General Goal-Conditioned Minecraft Model • Hacker News Board

Re: A General Goal-Conditioned Minecraft Model

#11

Post by superb_dev » Wed, Jul 15, 2026, 7:43 PM UTC

agajews wrote: post #3

Hey everyone! I'm Alex, one of the founders of Pantograph. We've spent the last six months building a pretty smart Minecraft model, coming soon to a server near you! We trained it on about 500k hours of Minecraft videos, and it learned how to fight creepers, build walls and other structures, and explore to find visual goals. We're considering putting up a public API for larger models like this one, let us know if you…

I would love to be able to add one of these bots to my Minecraft server! I’ve got a family server and we’ve been talking about how to get an AI in there for a while. I tried out some open source harnesses to allow a generic LLM to join, but none of theme were particularly good.

Re: A General Goal-Conditioned Minecraft Model

#12

Post by SirHackalot » Wed, Jul 15, 2026, 8:37 PM UTC

agajews wrote: post #4

xnx wrote: post #2

Reminds me of Google's AI Dreamer that could mine diamonds in Minecraft ( https://www.scientificamerican.com/article/google-deepmind-t... ) or OpenAI training on video to play Minecraft ( https://openai.com/index/vpt/ )

Very much inspired by those papers! One of the things that's interesting about our model is it's goal-conditioned, so it can do any task at inference time without training on it. We had a lot of fun making eval environments after we trained the model trying to find interesting things it can do, and that was all after we trained the model. More like prompting an LLM. (Versus Dreamer, which needs to be trained on a han…

How does this compare to non-stochastic, non-ML AI in games currently?

Re: A General Goal-Conditioned Minecraft Model

#13

Post by ryanhecht » Wed, Jul 15, 2026, 8:56 PM UTC

agajews wrote: post #3

Hey everyone! I'm Alex, one of the founders of Pantograph. We've spent the last six months building a pretty smart Minecraft model, coming soon to a server near you! We trained it on about 500k hours of Minecraft videos, and it learned how to fight creepers, build walls and other structures, and explore to find visual goals. We're considering putting up a public API for larger models like this one, let us know if you…

Hi Alex! I'm Ryan, I run a Minecraft theme park recreation server. While our gamemode is very far from vanilla Minecraft (it plays more like a MMORPG), I've been interested in standing up a model like this to help automate agentic testing of new software features in my monolith Bungee/Paper codebase.

In its current state, do you think it can handle non-vanilla Minecraft tasks like interacting with custom UI's, running commands, or doing other unusual tasks in the world?

Re: A General Goal-Conditioned Minecraft Model

#14

Post by agajews » Wed, Jul 15, 2026, 9:02 PM UTC

ryanhecht wrote: post #13

agajews wrote: post #3

Hey everyone! I'm Alex, one of the founders of Pantograph. We've spent the last six months building a pretty smart Minecraft model, coming soon to a server near you! We trained it on about 500k hours of Minecraft videos, and it learned how to fight creepers, build walls and other structures, and explore to find visual goals. We're considering putting up a public API for larger models like this one, let us know if you…

Hi Alex! I'm Ryan, I run a Minecraft theme park recreation server. While our gamemode is very far from vanilla Minecraft (it plays more like a MMORPG), I've been interested in standing up a model like this to help automate agentic testing of new software features in my monolith Bungee/Paper codebase. In its current state, do you think it can handle non-vanilla Minecraft tasks like interacting with custom UI's, runnin…

Probably not the current model, but one of the benefits of doing internet-scale pretraining is that the model has seen a lot of mods already! We think the bigger models will be able to handle some more custom Minecraft worlds.

Re: A General Goal-Conditioned Minecraft Model

#15

Post by agajews » Wed, Jul 15, 2026, 9:03 PM UTC

superb_dev wrote: post #11

agajews wrote: post #3

Hey everyone! I'm Alex, one of the founders of Pantograph. We've spent the last six months building a pretty smart Minecraft model, coming soon to a server near you! We trained it on about 500k hours of Minecraft videos, and it learned how to fight creepers, build walls and other structures, and explore to find visual goals. We're considering putting up a public API for larger models like this one, let us know if you…

I would love to be able to add one of these bots to my Minecraft server! I’ve got a family server and we’ve been talking about how to get an AI in there for a while. I tried out some open source harnesses to allow a generic LLM to join, but none of theme were particularly good.

Awesome! We should be a lot better than ordinary LLMs, especially at tasks that require making a lot of decisions in real-time.

Re: A General Goal-Conditioned Minecraft Model

#16

Post by agajews » Wed, Jul 15, 2026, 9:06 PM UTC

SirHackalot wrote: post #12

agajews wrote: post #4

Earlier quoted context omitted.

Very much inspired by those papers! One of the things that's interesting about our model is it's goal-conditioned, so it can do any task at inference time without training on it. We had a lot of fun making eval environments after we trained the model trying to find interesting things it can do, and that was all after we trained the model. More like prompting an LLM. (Versus Dreamer, which needs to be trained on a han…

How does this compare to non-stochastic, non-ML AI in games currently?

Pretty different--obviously a lot less consistent today, but capable of a lot more diverse kinds of behavior. As the models get bigger, they'll be easier to prompt, and you could imagine a game designer writing a long prompt for different kinds of NPCs. (E.g. GPT-3 was pretty difficult to get to do a particular thing, although it was very general, but over the years the instruction tuning has gotten a lot better and now it's very easy to ask questions to ChatGPT)