Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Project Genie: Experimenting with infinite, interactive worlds

blog.google

Post reply on HN
Print view View original

61–70 of 347 posts

Re: Project Genie: Experimenting with infinite, interactive worlds

#61

Post by ezst »

observationist wrote: post #26
sy26 wrote: post #10

I have been confused for a long time why FB is not motivated enough to invest in world models, it IS the key to unblock their "metaverse" vision. And instead they let go Yann LeCun.

LeCun wasn't producing results. He was obstinate and insistent on his own theories and ideas which weren't, and possibly aren't, going anywhere. He refused to engage with LLMs and compete in the market that exists, and spent all his effort and energy on unproven ideas and research, which split the company's mission and competitiveness. They lost their place as one of the top 4 AI companies, and are now a full generat…

Since a hot take is as good as the next one: LLMs are by the day more and more clearly understood as a "local maximum" with flawed capabilities, limited efficiency, a $trillion + a large chunk of the USA's GDP wasted, nobody even turning a profit from that nor able to build something that can't be reproduced for free within 6 months.

When the right move (strategically, economically) is to not compete, the head of the AI division acknowledging the above and deciding to focus on the next breakthrough seems absolutely reasonable.

Re: Project Genie: Experimenting with infinite, interactive worlds

#62

Post by qwertyi0k »

observationist wrote: post #26
sy26 wrote: post #10

I have been confused for a long time why FB is not motivated enough to invest in world models, it IS the key to unblock their "metaverse" vision. And instead they let go Yann LeCun.

LeCun wasn't producing results. He was obstinate and insistent on his own theories and ideas which weren't, and possibly aren't, going anywhere. He refused to engage with LLMs and compete in the market that exists, and spent all his effort and energy on unproven ideas and research, which split the company's mission and competitiveness. They lost their place as one of the top 4 AI companies, and are now a full generat…

To be fair, this was his job description: Fundamental AI Research (FAIR) lab. Not AI products division. You can't expect marketable products from a fundamental AI research lab.

Re: Project Genie: Experimenting with infinite, interactive worlds

#63

Post by avaer »

in-silico wrote: post #52

Everyone here seems too caught up in the idea that Genie is the product, and that its purpose is to be a video game, movie, or VR environment. That is not the goal. The purpose of world models like Genie is to be the "imagination" of next-generation AI and robotics systems: a way for them to simulate the outcomes of potential actions in order to inform decisions.

Soft disagree; if you wanted imagination you don't need to make a video model. You probably don't need to decode the latents at all. That seems pretty far from information-theoretic optimality, the kind that you want in a good+fast AI model making decisions.

The whole reason for LLMs inferencing human-processable text, and "world models" inferencing human-interactive video, is precisely so that humans can connect in and debug the thing.

I think the purpose of Genie is to be a video game, but it's a video game for AI researchers developing AIs.

I do agree that the entertainment implications are kind of the research exhaust of the end goal.

Re: Project Genie: Experimenting with infinite, interactive worlds

#64

Post by nozbufferHere »

WarmWash wrote: post #33

The actual breakthrough with Genie is being able to turn around and look back, and seeing the same scene that was there before. A few other labs have similar world simulators, but they all struggle badly with keeping coherence of things not in view. Hence why they always walk forwards and never look around.

Still amazed it took ML people so long to realize they needed and explicit representation to cache stuff.

Re: Project Genie: Experimenting with infinite, interactive worlds

#65

Post by observationist »

qwertyi0k wrote: post #53
halfmatthalfcat wrote: post #30

Earlier quoted context omitted.

Were you there or just an attentive outsider?

Most serious researchers want to work on interesting problems like reinforcement learning or robotics or RNN or dozen other avant-garde subjects. None want to work on "boring" LLM technology, requiring significant engineering effort and huge dataset wrangling effort.

This is true - Ilya got an exit and is engaged in serious research, but research is by its nature unpredictable. Meta wanted a product and to compete in the AI market, and JEPA was incompatible with that. Now LeCun has a lab and resources to pursue his research, and Meta has refocused efforts on LLMs and the marketplace - it remains to be seen if they'll be able to regain their position. I hope they do - open models and relatively open research are important, and the more serious AI labs that do this, the more it incentivizes others to do the same, and keeps the ones that have committed to it honest.

Re: Project Genie: Experimenting with infinite, interactive worlds

#66

Post by mikewittie »

aurumque wrote: post #16
srameshc wrote: post #9

What’s the endgame here? For a small gaming studio, what are the actual implications?

I would think that building a environment which can be managed by a game engine is the first pass. In a few years when we are able to render more than 60 seconds it could very well replace the game engine entirely by just rendering everything in realtime based on user interactions. The final phase is just prompts which turn directly into interactive games, maybe even multiplayer. When I see the progress we've made on…

We seem to think that Genie is good at the creative part, but bad at the consistency and performance part. How hard would it be to take 60 seconds of Genie output and pipe it into a model that generates a consistent and performant 3D environment?

Re: Project Genie: Experimenting with infinite, interactive worlds

#67

Post by falcor84 »

jsheard wrote: post #15
0xcb0 wrote: post #7

I keep on repeating myself, but it feels like I'm living in the future. Can't wait to hook this up to my old Oculus glasses and let Genie create a fully realistic sailing simulator for me, where I can train sailing with realistic conditions. On boats I'd love to sail. If making games out of these simulations work, it't be the end for a lot of big studios, and might be the renaissance for small to one person game stud…

Isn't this still essentially "vibe simulation" inferred from videos? Surface-level visual realism is one thing, but expecting it to figure out the exact physical mechanics of sailing just by watching boats, and usefully abstract that into a gamified form, is another thing entirely.

Why wouldn't it just hook it into something like physx?

Re: Project Genie: Experimenting with infinite, interactive worlds

#68

Post by observationist »

richard___ wrote: post #41
observationist wrote: post #32

Earlier quoted context omitted.

Attentive outsider and acquaintance of a couple people who are or were employed there. Nothing I'm saying is particularly inside baseball, though, it's pretty well covered by all the blogs and podcasts.

What podcast?

Machine Learning Street Talk and Dwarkesh are excellent. Various discord communities, forums, and blogs downstream of the big podcasts, and following researchers on X keeps you in the loop on a lot of these things, and then you can watch for random interviews and presentations on youtube when you know who the interesting people and subjects are.

Re: Project Genie: Experimenting with infinite, interactive worlds

#69

Post by in-silico »

avaer wrote: post #63
in-silico wrote: post #52

Everyone here seems too caught up in the idea that Genie is the product, and that its purpose is to be a video game, movie, or VR environment. That is not the goal. The purpose of world models like Genie is to be the "imagination" of next-generation AI and robotics systems: a way for them to simulate the outcomes of potential actions in order to inform decisions.

Soft disagree; if you wanted imagination you don't need to make a video model. You probably don't need to decode the latents at all. That seems pretty far from information-theoretic optimality, the kind that you want in a good+fast AI model making decisions. The whole reason for LLMs inferencing human-processable text, and "world models" inferencing human-interactive video, is precisely so that humans can connect in…

Sufficiently informative latents can be decoded into video.

When you simulate a stream of those latents, you can decode them into video.

If you were trying to make an impressive demo for the public, you probably would decode them into video, even if the real applications don't require it.

Converting the latents to pixel space also makes them compatible with existing image/video models and multimodal LLMs, which (without specialized training) can't interpret the latents directly.

Re: Project Genie: Experimenting with infinite, interactive worlds

#70

Post by JeremyNT »

jsheard wrote: post #15
0xcb0 wrote: post #7

I keep on repeating myself, but it feels like I'm living in the future. Can't wait to hook this up to my old Oculus glasses and let Genie create a fully realistic sailing simulator for me, where I can train sailing with realistic conditions. On boats I'd love to sail. If making games out of these simulations work, it't be the end for a lot of big studios, and might be the renaissance for small to one person game stud…

Isn't this still essentially "vibe simulation" inferred from videos? Surface-level visual realism is one thing, but expecting it to figure out the exact physical mechanics of sailing just by watching boats, and usefully abstract that into a gamified form, is another thing entirely.

Yeah I have a whole lot of trouble imagining this replacing traditional video games any time soon; we have actually very good and performant representations of how physics work, and games are tuned for the player to have an enjoyable experience.

There's obviously something insanely impressive about these google experiments, and it certainly feels like there's some kind of use case for them somewhere, but I'm not sure exactly where they fit in.

Post reply on HN