Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Project Genie: Experimenting with infinite, interactive worlds

blog.google

Post reply on HN
Print view View original

311–320 of 347 posts

Re: Project Genie: Experimenting with infinite, interactive worlds

#311

Post by phailhaus »

dyauspitr wrote: post #275
phailhaus wrote: post #144

Earlier quoted context omitted.

They don't though, they're hallucinated videos. They're feeding models tons and tons of 2D videos and hoping they figure out physics from them, instead of just using a game engine and having the LLM write something up that works 100% of the time.

On the flip side, the emergent properties that come from some of these wouldn’t be replicable by an engine. A moss covered rock realistically shedding moss as it rolls down a hill. Condensation aggregating into beads and rivulets on glass. An ant walking on a pitcher plant and being able to walk inside it and see bugs drowned from its previous meal. You’re missing the forest for the trees.

And then the rivulets disappear or change completely because you looked away. The reason this is a dead end is because computationally, there is absolutely no way for the model to keep track of everything that it decided. Everything is kept "in its head" rather than persisted. So what you get is a dream world, useless for training world models. It's great for prototyping, terrible for anything more durable.

Re: Project Genie: Experimenting with infinite, interactive worlds

#312

Post by pcthrowaway »

forrestthewoods wrote: post #261

SPIN THE CAMERA 1080 DEGREES YOU COWARDS The only test I ever want to see with these frame-gen models is a full 1080 degree camera spin. Miss me with that 30 degree back and forth crap. I want multiple full turns. Some jitter and a little back-and-forth wobble is fine. But I want multiple full spins. I’m pretty sure I know why they don’t do this :)

> SPIN THE CAMERA 1080 DEGREES YOU COWARDS

> a full 1080 degree camera spin

Do you mean 3 full turns, or do you mean 180 (one half-turn)?

Re: Project Genie: Experimenting with infinite, interactive worlds

#314

Post by wasmainiac »

ForceBru wrote: post #299
wasmainiac wrote: post #272

Earlier quoted context omitted.

Have a source for that? I think you are anthropomorphising the AI too much. Imagination is inspired by reality, which AI does not have. Introducing a reality which the AI fully controls (looking beyond issues of vision and physics simulation) would only induce psychosis in the AI itself since false assumptions would only be amplified.

> psychosis in the AI itself I think you're anthropomorphising the AI too much: what does it mean for an LLM to have psychosis? This implies that LLMs have a soul, or a consciousness, or a psyche. But... do they? Speaking of reality, one can easily become philosophical and say that we humans don't exactly "have" a reality either. All we have are sensor readings. LLMs' sensors are texts and images they get as input. T…

> I think you're anthropomorphising the AI too much

I don’t get it. Is that supped to be a gotchya? Have you tried maliciously messing with an LLM? You can get it into a state that resembles psychosis. I mean you give it a context that is removed from reality, yet close enough to reality to act on and it willl give you crazy output.

Re: Project Genie: Experimenting with infinite, interactive worlds

#315

Post by dyauspitr »

phailhaus wrote: post #311
dyauspitr wrote: post #275

Earlier quoted context omitted.

On the flip side, the emergent properties that come from some of these wouldn’t be replicable by an engine. A moss covered rock realistically shedding moss as it rolls down a hill. Condensation aggregating into beads and rivulets on glass. An ant walking on a pitcher plant and being able to walk inside it and see bugs drowned from its previous meal. You’re missing the forest for the trees.

And then the rivulets disappear or change completely because you looked away. The reason this is a dead end is because computationally, there is absolutely no way for the model to keep track of everything that it decided. Everything is kept "in its head" rather than persisted. So what you get is a dream world, useless for training world models. It's great for prototyping, terrible for anything more durable.

A dreamworld where the overwhelming number of things are consistent is better that something low detail and always consistent.

Re: Project Genie: Experimenting with infinite, interactive worlds

#316

Post by ghtbircshotbe »

namanyayg wrote: post #269
jlhawn wrote: post #167

Now I can't stop thinking about _The Experience Machine_ by Andy Clark. It theorizes that this is how humans navigate and experience the real world: Our brains generate what we think the world around is like and our senses don't so much directly process visual information but instead act like a kind of loss function for our internal simulations. Then we use that error to update our internal model of the world. In thi…

This is one of my fundamental beliefs about the nature of consciousness. We are never able to interact with the physical world directly, we first perceive it and then interpret those perceptions. More often than not, our interpretation ignores and modifies those perceptions, so we really are just living in a world created by our own mental chatter. This is one of the core tenets of Buddhism, and it's also expounded o…

I'm not sure this is unique to consciousness (whatever that is). What would it even mean to directly interact with the physical world? Even the most precise scientific experiments are a series of indirect measurements of something that perhaps in some sense is fundamentally unknowable.

Re: Project Genie: Experimenting with infinite, interactive worlds

#317

Post by unicorn_cowboy »

Can someone explain the actual use cases for this that folks are excited about?

Nobody wants to be responsible for writing the film they're watching, as they watch it. Nobody wants to engage with a virtual world that has no story, narrative, meaning, purpose.

What is the actual use case for this, once we move beyond the tech demo stage?

The idea it will wholesale replace existing 3D graphics/rendering pipelines and processes any time soon seems so far-fetched to me (even just logistically) that I can't wrap my head around what people are thinking here.

This strikes me as a fancy parlor trick. Unless I'm mistaken, we need proper world models to do the things people are erroneously assuming this to be capable of one day.

Re: Project Genie: Experimenting with infinite, interactive worlds

#318

Post by ollin »

danielwmayer wrote: post #290
montebicyclelo wrote: post #6

Reminds me of this [1] HN post from 9 months ago, where the author trained a neural network to do world emulation from video recordings of their local park — you can walk around in their interactive demo [2]. I don't have access to the DeepMind demo, but from the video it looks like it takes the idea up a notch. (I don't know the exact lineage of these ideas, but a general observation is that it's a shame that it's t…

I was immediately struck when I looked down at just the boardwalk how similar it felt to being on LSD. I am continually astounded with how similar these systems end up seeming to how our brain works. May just be happy coincidences but I am pretty sold on there being true parallels that will only become more and more apparent.

A lot of people mentioned this! The "dreamlike" comparison is common as well. In both cases, you have a network of neurons rendering an image approximating the real world :) so it sort of makes sense.

Regarding the specific boiling-textures effect: there's a tradeoff in recurrent world models between jittering (constantly regenerating fine details to avoid accumulating error) and drifting (propagating fine details as-is, even when that leads to accumulating error and a simplified/oversaturated/implausible result). The forest trail world is tuned way towards jittering (you can pause with `p` and step frame-by-frame with `.` to see this). So if the effect resembles LSD, it's possible that LSD applies some similar random jitter/perturbation to the neurons within your visual cortex.

Re: Project Genie: Experimenting with infinite, interactive worlds

#319

Post by stephenjayakar »

YetAnotherNick wrote: post #270
qwertyi0k wrote: post #121

Earlier quoted context omitted.

Ref: Yann lecun post on linkedin, 3years ago: FAIR now stand for "Fundamental AI Research"

I literally linked the official site and it currently says Facebook. I have known FAIR for many years and I have always know it as Facebook. Can you link any official source of changing the name.

I gotcha. https://www.linkedin.com/posts/yann-lecun_big-changes-for-ai..., https://ai.meta.com/blog/fair-10-year-anniversary-open-scien...

Re: Project Genie: Experimenting with infinite, interactive worlds

#320

Post by caspar »

unicorn_cowboy wrote: post #317

Can someone explain the actual use cases for this that folks are excited about? Nobody wants to be responsible for writing the film they're watching, as they watch it. Nobody wants to engage with a virtual world that has no story, narrative, meaning, purpose. What is the actual use case for this, once we move beyond the tech demo stage? The idea it will wholesale replace existing 3D graphics/rendering pipelines and p…

Presumably others will write the prompts (or equivalent directing mechanism) that will steer the generation, such that you can act out whatever fantasies interest you.
Post reply on HN