Does anyone know why most of the videos are in slow motion?
Video generation models as world simulators
121–130 of 171 posts
Re: Video generation models as world simulators
#122SORA.....the entire movie industry is now out of a job.
Did you actually read the post? This is not the case, yet.
Also the concept of learning to simulating the world seems more important than just the media and content implications.
Re: Video generation models as world simulators
#123Maybe this says more about me than about the technology, but I found the consistency of the Minecraft simulation super impressive.
I was wondering how feasible it would be to make a Minecraft agent that had a running feed of the past few seconds, continued it off w/ SORA, fed the continuation into a (relatively) simple policy translator that just pulled out what the video showed as player inputs, and the inputted that. Presumably, this would work for non-minecraft applications, but Minecraft has a really standardized interface layer
It’s interesting to think about what the internal model of a NN of a game might be like, and whether it could be exposed with an API that someone could program against.
Re: Video generation models as world simulators
#124I find it wild that this model does not have explicit 3D prior, yet learns to generate videos with such 3D consistency, you can directly train a 3D representation (NeRF-like) from those videos: https://twitter.com/BenMildenhall/status/1758224827788468722
You aren't looking carefully enough. I find so many inconsistencies in these examples. Perspectives that are completely wrong when the camera rotates. Windows that shift perspective, patios that are suddenly deep/shallow. Shadows that appear/disappear as the camera shifts. In other examples; paths, objects, people suddenly appearing or disappearing out of nowhere. A stone turning into a person. A horse that suddenly…
Yeah, it seems to have a hard time processing lens distortion in particular which gives a very weird quality. It's actually bending things, or trying to fill in the gaps, instead of distorting the image in the "correct" way.
Re: Video generation models as world simulators
#125Re: Video generation models as world simulators
#126This is the second Sora announcement I've seen. Am I missing how I can play with it? The examples in the papers are all well and good but I want to get my hands on it and try it.
Re: Video generation models as world simulators
#127Re: Video generation models as world simulators
#128I think people might be missing what this enables. It can make plausible continuations of video, with realistic physics. What happens if this gets fast enough to work _in real time_. Connect this to a robot that has a real time camera feed. Have it constantly generate potential future continuations of the feed that it's getting -- maybe more than one. You have an autonomous robot building a real time model of the wor…
FWIW, you've basically described at a high level exactly what autonomous driving systems have been doing for several years. I don't think anyone would say that Waymo's cars are really close to AGI.
Re: Video generation models as world simulators
#129Earlier quoted context omitted.
> Connect this to a robot that has a real time camera feed. Have it constantly generate potential future continuations of the feed that it's getting -- maybe more than one. You have an autonomous robot building a real time model of the world around it and predicting the future. Give it some error correction based on well each prediction models the actual outcome and I think you're _really_ close to AGI. In theory, ye…
Tbh I don't think an AI for Civ would that impressive, my experience is that most of time you can get away with making locally optimal decisions I.e growing your economy and fighting weaker states. The problem with current civ AI is that their economies can be often structured nonsensically, but optimized economies is usually just the matter of stacking bonuses together into specialized production zones, which can be…
Maybe, but a lot of people would like better AIs in strategy video games, it only adds to the frustration when people say "it wouldn't be that impressive". It's like saying "that would be easy... but it's not going to happen." (And I'm not focused on Civilization, it's just a well known example, I'd like to see a strong AI in any similar strategy game.)
I think it might be harder than StarCraft or Dota. Civilization is all about slow decision making (no APM advantages for the AI), and all the decisions are quite different, and you have to make them in a competitive environment where an opponent can raid and capture your cities.
Re: Video generation models as world simulators
#130Earlier quoted context omitted.
> Connect this to a robot that has a real time camera feed. Have it constantly generate potential future continuations of the feed that it's getting -- maybe more than one. You have an autonomous robot building a real time model of the world around it and predicting the future. Give it some error correction based on well each prediction models the actual outcome and I think you're _really_ close to AGI. In theory, ye…
Tbh I don't think an AI for Civ would that impressive, my experience is that most of time you can get away with making locally optimal decisions I.e growing your economy and fighting weaker states. The problem with current civ AI is that their economies can be often structured nonsensically, but optimized economies is usually just the matter of stacking bonuses together into specialized production zones, which can be…
The problem with game AI is that they "cheat". They don't play like a human. The civ AI straight up gets extra resources, AlphaStar in SC2 performed inhuman feats like giving commands in two different areas of the map simultaneously or spiking actions per minute to inhuman levels briefly. But even with all of that the AI still eventually loses. And then they start losing consistently as players play more against them.
Why? Because AI doesn't learn on the fly. The AI does things a certain way and beating it becomes a puzzle game. It doesn't feel like playing against a human opponent (although AlphaStar in SC2 probably came pretty close).
Learning on the fly is probably the biggest thing that (game) AI is lacking in. I'm not sure there's an easy solution for it.