Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Genie 2: A large-scale foundation world model

deepmind.google

Post reply on HN
Print view View original

291–300 of 436 posts

Re: Genie 2: A large-scale foundation world model

#291

Post by achierius »

wongarsu wrote: post #160
vessenes wrote: post #4

This is.. super impressive. I'd like to know how large this model is. I note that the first thing they have it do is talk to agents who can control the world gen; geez - even robots get to play video games while we work. That said; I cannot find any: - architecture explanation - code - technical details - API access information Feels very DeepMind / 2015, and that's a bummer. I think the point of the "we have no moat…

We already knew it's possible from AI minecraft ( https://oasis.decart.ai ). This is just a more impressive version of that, trained on a wider range of games and with more context frames (Oasis has about a second of context, this one a minute). Even the architecture seems to be about the same. Had they released this two months earlier it would have been incredibly impressive. Now it's still cool and inspiring, but n…

This is far beyond Oasis. Oasis had approximately 0 continuity, and the generated world was a blurry mess. This on the other hand actually approaches usability.

Re: Genie 2: A large-scale foundation world model

#293

Post by momojo »

xcodevn wrote: post #21

On a very similar theme, here is the work from World Lab (founded by Fei-Fei Li, ImageNet dataset, et al.) about creating 3D worlds: https://www.worldlabs.ai/blog

I find this work much more exciting. They're not just teaching a model to hallucinate given WASD input. They're generating durable, persistent point clouds. It looks so similar to Genie2 yet they're worlds apart.

Re: Genie 2: A large-scale foundation world model

#294

Post by sangnoir »

mosdl wrote: post #265
sangnoir wrote: post #125

Earlier quoted context omitted.

> Interacting with the actual physical world is very different from creating a video game The major difference being the former scales very poorly for generating training data compared to the latter. Genie 2 is not even a video game and has worse fidelity that video games, the upside is it probably scales even better than video games for generating training scenarios. If you want androids in teal life, Genie 2 (or si…

How does turning an image into a game help with robots? Robots don't need to guess what they can't see, they would have sensors to tell them exactly what is there (like a self driving car).

> Robots don't need to guess what they can't see, they would have sensors to tell them exactly what is there (like a self driving car).

Self driving vars have cameras as part of their sensor suite, and have models to make sense of sensor data. Video will help with perception and classification (understanding the world) with no agency needed. Game-playing will help with planning, execution, and evaluation. Both functions are necessary, and those that come after rely on earlier capabilities

Re: Genie 2: A large-scale foundation world model

#295

Post by UncleOxidant »

echelon wrote: post #29
bix6 wrote: post #14

Genuine question: What is the point of telling us about this if we can’t use it? Is it just to flex on everyone?

To stop competing startups from getting funding. Decart (Oasis) raised $25 million at $500 million valuation. World Labs raised $230 million.

Not sure about that. Sometimes Google legitimates a field. I was in a kite power startup company back in 2019. Before Google canceled it's Makani kite power project VCs and angels would at least talk to us - it gave them some frame of reference: "Oh, this is like the kite power thing Goggle is doing?" "Right, but on a much smaller scale". After they canceled Makani in the summer of 2019 it was crickets. We folded by the end of 2019. They figured if Google couldn't make it work then it probably wasn't something to invest in.

Re: Genie 2: A large-scale foundation world model

#296

Post by BlueTemplar »

wongarsu wrote: post #135
erulabs wrote: post #128

It’s interesting to me that we continue to see such pressure on video and world generation, despite the fact that for years now we’ve gotten games and movies that have beautiful worlds filled with lousy, limited, poorly written stories. Star Wars movies have looked phenomenal for a decade, full of bland stories we’ve all heard a thousand times. Are there any game developers working on infinite story games? I don’t ca…

Dwarf Fortress is the state of the art in procedural interactive story generation. Youtube channels like kruggsmash show how great it is in that role if you actually read all the text. But that doesn't translate well to websites, trailers or demos. It's easier to wow people with graphics.

I think that would be Rimworld, which is laser-focused on this aspect to the point of allowing you to pick different kinds of "narrators" ?

(Dwarf Fortress being much more focused on generating a whole world.)

Re: Genie 2: A large-scale foundation world model

#297

Post by com2kid »

erulabs wrote: post #128

It’s interesting to me that we continue to see such pressure on video and world generation, despite the fact that for years now we’ve gotten games and movies that have beautiful worlds filled with lousy, limited, poorly written stories. Star Wars movies have looked phenomenal for a decade, full of bland stories we’ve all heard a thousand times. Are there any game developers working on infinite story games? I don’t ca…

IMHO Humans will still create the overarching stories, what LLMs will do is help fill in the expensive blanks that make adding stories to a world hard.

For example, right now if you save an entire village from an attacking tribe of orcs, only a handful of NPCs even say anything, just a nice little "thanks for saving our town!" and then 2 villages over the NPCs are completely unaware of a mighty hero literally solo tanking an entire invading army.

Why is that?

Well you'd need lots of, somewhat boring but important, dialogue written, and you'd need tons of voice lines recorded.

Both those are now solvable problems with generative AI. AI generated dialogue is now reasonably high quality, not "main character story arc" high quality, but "idle shop keeper chit chat" quality for sure, it won't break immersion at least. And the quality of writing from AI is fine for 2 or 3 sentences here and there.

I'll be soon releasing a project showing this off at https://www.tinytown.ai/ the NPC dialogue is generated on a small LLM that can be ran locally, and the secret of even high quality voice models is that they don't require a lot of memory to run.

I predict that in another 4 or 5 years we'll see a lot of models ran at the edge on video game consoles and home PCs, fleshing out game worlds.

Re: Genie 2: A large-scale foundation world model

#298

Post by Philpax »

miltonlost wrote: post #185
visarga wrote: post #177

Earlier quoted context omitted.

Actually, all you need to do is to apply structured randomness to get diversity from a LLM. For example in TinyStories paper, a precursor of the Phi models: > We collected a vocabulary consisting of about 1500 basic words, which try to mimic the vocabulary of a typical 3-4 year-old child, separated into nouns, verbs, and adjectives. In each generation, 3 words are chosen randomly (one verb, one noun, and one adjectiv…

A story is not just words crammed together that sound plausible. Is the AI going to know about pacing? About character motivations? About interconnecting disparate plots? That paper sounds like it has a scientist’s conception that a story is just words, and not complex trade offs between the start of a story and its end and middle, complexity and planning that won’t come from any sort of next-token generation. These…

> Is the AI going to know about pacing? About character motivations? About interconnecting disparate plots?

Yes. This is an active research area. See https://github.com/yingpengma/Awesome-Story-Generation, which is not up to date.

Re: Genie 2: A large-scale foundation world model

#299

Post by lifeformed »

Neat tech but people might mistake this as being useful for game development, where it'll be less helpful than useless.

Games are about interactions, and this actively works against it. You don't want the model to infer mechanics, the designer needs deep control over every aspect of it.

People mentioned using this for prototyping a game, but that's completely meaningless. What would it even mean to use this to prototype something? It doesn't help you figure out anything mechanically or visually. It's just, "what if you were an avatar in a world?" What do you do after you run around with your random character controller in your random environments?

I think the most useful part of this is the world generation part, not the mechanics inference part.

Re: Genie 2: A large-scale foundation world model

#300

Post by teamonkey »

singularity2001 wrote: post #252
jjice wrote: post #2

I don't understand this space very well, but this seems incredible. Something I find interesting about generative AI is how it adds a huge layer of flexibility, but at the cost of lots of computation, while a very narrow set of constraints (a traditional program) is comparatively incredibly efficient. If someone spent a ton of time building out something simple in Unity, they could get the same thing running with a s…

Makes me wonder if there's any company which is trying to train a model to produce three D worlds within Unity (not as a video like oasis).

This at least is a bit more realistic than what’s being presented by Google.

There are already a number of techniques for procedurally-generating a world (including Markov-based systems).

The problem with replacing procedural world generation with LLM generation are a) you need to obtain a data set to train it, which doesn’t commercially exist, or train it yourself, b) there’s a fundamental need to iterate on the design, which LLMs do not cope with well at all, c) you need to somehow debug issues and fix them. That’s quite apart from the quality issues, cost and power usage.

Post reply on HN