Given the near-impossibility of predicting something as "simple" as a stock market due to its recursive nature, I'm not sure I see how it would be possible to simulate an infinitely more complicated "world"
The dawn of a world simulator
51–60 of 60 posts
Re: The dawn of a world simulator
#52Earlier quoted context omitted.
> Visually, they are stunning. The input images are stunning, model's result is another disappointing trip to uncanny valley. But we feel Ok as long as the sequence doesn't horribly contradict the original image or sound. That is the world model.
> But we feel Ok as long as the sequence doesn't horribly contradict the original image or sound. Is the error I pointed out not "horribly contradicting"? > That is the world model. I would say that if it is non-physical[0] then it's hard to call it a /world/ model. A world is consistent and has a set of rules that must be followed. I've yet to see a claimed world model that actually captures this behavior. Yet it's…
> The major problem with these systems is that they just hope that the physics is recovered through enough examples of videos. Yet if one studied physics (beyond your basic college courses) you'd understand the naïveté of that. It took a long time to develop physics due to these specific limitations. These models don't even have the advantage of being able to interact with the environment. They have no mechanisms to form beliefs and certainly no means to test them. It's essentially impossible to develop physics through observation alone
Sounds like these world models are speed running from Platonic ideals to empiricism.
Re: The dawn of a world simulator
#53Earlier quoted context omitted.
> But we feel Ok as long as the sequence doesn't horribly contradict the original image or sound. Is the error I pointed out not "horribly contradicting"? > That is the world model. I would say that if it is non-physical[0] then it's hard to call it a /world/ model. A world is consistent and has a set of rules that must be followed. I've yet to see a claimed world model that actually captures this behavior. Yet it's…
> The major problem with these systems is that they just hope that the physics is recovered through enough examples of videos. Yet if one studied physics (beyond your basic college courses) you'd understand the naïveté of that. It took a long time to develop physics due to these specific limitations. These models don't even have the advantage of being able to interact with the environment. They have no mechanisms to…
[dead]
Re: The dawn of a world simulator
#54I feel like there's a bit if a disconnect with the cool video demos demonstrated here and say, the type of world models someone like Yann Lecunn is talking about. A proper world model like Jepa should be predicting in latent space where the representation of what is going on is highly abstract. Video generation models by definition are either predicting in noise or pixel space (latent noise if the diffuser is diffusi…
> Video generation models by definition are either predicting in noise or pixel space I don't see that this follows "by definition" at all. Just because your output is pixel values doesn't mean your internal world model is in pixel space.
You need to train a decoder either end to end or conditioned on latents.
In either case the impressiveness of that decoder can be far removed from the effectiveness of your world model or involve no world model at all
Re: The dawn of a world simulator
#55Earlier quoted context omitted.
See the demo on their homepage. Calling it a world simulator is a marketing gimmick. It's a worse video generator but you can interact with it in real time and direct the video a little bit. Next version of this thing will be worth looking, this one isnt.
why would you assume anything about "the next version"?
Based it on other video models, all the ones I have seen keep improving. This one should too. Infact, Google is doing it already with their Genie (IIRC). That one is high quality and interactive.
Re: The dawn of a world simulator
#56Earlier quoted context omitted.
> poorly defined. Poorly defined is not the same as undefined. There are bounds and we have a decent understanding of what this means. Not having the details all worked out is not the same. Though that lack of precision is being used to get away with more slop. > I need to find the paper that deepmind produced I've seen that paper and the results pretty close to the action. I've even personally talked with people tha…
I don't think I'm disagreeing, just adding more colour. > It very frequently "forgets" what is outside its view This was the observations that I saw when we were testing it. My former lab was late to pivoting to robotics, so we were looking at the current state of play to see what machine perception stuff is out there for robotics.
Have you looked at Titan and MIRAS where they use online/updating associative memory that happens to be read out via next-token prediction?
https://research.google/blog/titans-miras-helping-ai-have-lo...
https://arxiv.org/abs/2501.00663
https://arxiv.org/pdf/2504.13173
Much research is going into these directions, but I'm more interested in mind-wandering tangents, involving both attentional control and additional mechanisms (memory retrieval, self-referential processing).
Re: The dawn of a world simulator
#57Earlier quoted context omitted.
I don't think I'm disagreeing, just adding more colour. > It very frequently "forgets" what is outside its view This was the observations that I saw when we were testing it. My former lab was late to pivoting to robotics, so we were looking at the current state of play to see what machine perception stuff is out there for robotics.
Have you looked at Titan and MIRAS where they use online/updating associative memory that happens to be read out via next-token prediction? https://research.google/blog/titans-miras-helping-ai-have-lo... https://arxiv.org/abs/2501.00663 https://arxiv.org/pdf/2504.13173 Much research is going into these directions, but I'm more interested in mind-wandering tangents, involving both attentional control and additional me…
Memory in world models is interesting. But I think the main issue is that its holding everything in pixel space (its not, but it feels like that) rather than concept space. Thats why its hard for it to synthesise consistently.
However I am not qualified really to make that assertion.
Re: The dawn of a world simulator
#58Earlier quoted context omitted.
> Video generation models by definition are either predicting in noise or pixel space I don't see that this follows "by definition" at all. Just because your output is pixel values doesn't mean your internal world model is in pixel space.
You need to train a decoder either end to end or conditioned on latents. In either case the impressiveness of that decoder can be far removed from the effectiveness of your world model or involve no world model at all
Making convincing videos of the world without having a world model would be like writing convincing essays about computing without understanding computing.
Re: The dawn of a world simulator
#59The reason they are called "world models" is because the internal representation of what they display represents a "world" instead of a video frame or image. The model needs to "understand" geometry and physics to output a video. Just because there are errors in this doesn't mean it isn't significant. If a machine learning model understands how physical objects interact with each other that is very useful.
> The model needs to "understand" geometry and physics to output a video. No it doesn't. It merely needs to mimic.
Here's a recent paper showing that models trained to generate videos develop strong geometric representations and understanding:
Re: The dawn of a world simulator
#60Earlier quoted context omitted.
You need to train a decoder either end to end or conditioned on latents. In either case the impressiveness of that decoder can be far removed from the effectiveness of your world model or involve no world model at all
Making convincing videos of the world without having a world model would be like writing convincing essays about computing without understanding computing.
There's a few things to consider here:
- there are many aspects to the video that are not convincing, indicating these videogen models do not grok the world the same way a typical human does
- A 6 year old child is almost certainly incapable of recreating pixel level fidelity video footage, yet understands the world extremely well... far beyond what current robotics is capable of.
The two facts above should be indicative that predicting noise (as with DDPM diffusion models), or predicting pixel level (or even VAE latent "pixel") information is probably not the optimal path to world understanding. Probably not even a good path to good world models.