How OpenAI's Sora Model Works
factorialfunds.com
How OpenAI's Sora Model Works
1–10 of 29 posts
Re: How OpenAI's Sora Model Works
#2[deleted]
Re: How OpenAI's Sora Model Works
#3What’s the status on companies building AI models to build actual 3D backend behind these generative videos. Anyone working on something similar? Imagine that’d be far more productive. For example, lookdev mlop is pretty low hanging fruit. Not sure why we don’t already have models from Autodesk, Epic or even Adobe (with resources ie A100/H100) where you upload an image/video and the model spits out workable 3D scaffolds.
Re: How OpenAI's Sora Model Works
#4What’s the status on companies building AI models to build actual 3D backend behind these generative videos. Anyone working on something similar? Imagine that’d be far more productive. For example, lookdev mlop is pretty low hanging fruit. Not sure why we don’t already have models from Autodesk, Epic or even Adobe (with resources ie A100/H100) where you upload an image/video and the model spits out workable 3D scaffo…
If I’m not mistaken, Stability just released something like that a few days ago.
Re: How OpenAI's Sora Model Works
#5I don't get how transformers can replace convolutional networks. My understanding is patches get fed in, and the transformer will do the same thing that a convolution layer does. But transformers deal with sequential data and I don't see any of that here?
Re: How OpenAI's Sora Model Works
#6I don't get how transformers can replace convolutional networks. My understanding is patches get fed in, and the transformer will do the same thing that a convolution layer does. But transformers deal with sequential data and I don't see any of that here?
I think it just treats the patches like it would be sequentially in memory or disk, but also has coordinates. And they have overlapping patches at an offset to catch features that would span a patch and be missed at that level.
Re: How OpenAI's Sora Model Works
#7I don't get how transformers can replace convolutional networks. My understanding is patches get fed in, and the transformer will do the same thing that a convolution layer does. But transformers deal with sequential data and I don't see any of that here?
[deleted]
Re: How OpenAI's Sora Model Works
#8What’s the status on companies building AI models to build actual 3D backend behind these generative videos. Anyone working on something similar? Imagine that’d be far more productive. For example, lookdev mlop is pretty low hanging fruit. Not sure why we don’t already have models from Autodesk, Epic or even Adobe (with resources ie A100/H100) where you upload an image/video and the model spits out workable 3D scaffo…
This is a good question, and the answer is that from a tech side it is surprisingly easier to solve the problem in the reverse direction.
As in, making workable 3d models is harder than making video.
And it is easier to make a 3d model by generating a video of the object instead.
Why is that? I don't know. But that's the current state of the industry. 3D model generation is simply harder.
Re: How OpenAI's Sora Model Works
#9What’s the status on companies building AI models to build actual 3D backend behind these generative videos. Anyone working on something similar? Imagine that’d be far more productive. For example, lookdev mlop is pretty low hanging fruit. Not sure why we don’t already have models from Autodesk, Epic or even Adobe (with resources ie A100/H100) where you upload an image/video and the model spits out workable 3D scaffo…
If I’m not mistaken, Stability just released something like that a few days ago.
Looks like I completely overlooked threestudio released last year. Thank you for pointing it out.
Re: How OpenAI's Sora Model Works
#10I don't understand, from the article, how Sora works when handling a rotation of an object on another object (the leaves in the leaf covered elefant for example). The explanation goes only to the diffussion model, but not to how, from that model, a correct geometry deformation is derived at each step.