This is very cool thanks for sharing
Learnings from 4 months of Image-Video VAE experiments
11–19 of 19 posts
Re: Learnings from 4 months of Image-Video VAE experiments
#12its cool to see the iterative improvements to your model laid out, but for everything that workedm i imagine there were at least a million other things you also tried but didnt work out. whats your process of trying these different techniques/architectures? do you just wait for one experiment to finish and visually inspect the results everytime. seems hard since these take a while to train. how do you shorten the fee…
honestly, it's really hard to shorten the feedback loop in this space. For this, we really just did run one experiment at a time and visually inspect the results everywhere. when you're going 0 -> 1, you're looking for "signs of life" to make sure the basic thing is working. when it comes to testing which (of the infinite levers) to the pull, a lot of it comes from intuition (which i know isn't the most fun answer). we spent a week or so just running experiments on the amount of compression we could squeeze out the VAE without significant degradation in the final results). In hindsight, spending a week on that seems like a waste, since we got the 8x spatial, 4x compression within the first 1-2 days. But in the moment, you're often unsure WHAT will be the key unlock. So, when you're in the middle of storm you're running a quick bayesian process in your head, measuring what you might learn from the outcome of the experiment vs. the time/money it would take to run the experiment. And you, hope that your intuitions become stronger over time, as you take more repetitions. More money, might help the problem (e.g. parallel experiments, more detailed explorations). But, I don't think money is a cure-all. At some point, you get lost in the sauce trying to tie the threads between all the empirical findings you have at your finger tips. Maybe one day AI models could help here integrating these all results. As it stands, they still struggle to reason about this stuff, in context of other research papers and findings (likely because all the context on arxiv is so noisy; you can't trust any particular finding and verifying findings is so hard to do, that it's hard to meta-reason about your experiments correctly).
Re: Learnings from 4 months of Image-Video VAE experiments
#13As someone currently working on their own VAE, you reasoning for why you went with WAN 2.1 and your learnings for what you think you did wrong really resonated with me, specifically:
> Looking back, we should have just filtered out these samples from the dataset and moved on.
I hadn't even considered to look and see if poor data quality was resulting in an inability to recreate. This is a good gotchya to look out for. Appreciate the deep dive here!
Re: Learnings from 4 months of Image-Video VAE experiments
#14[dead]
Re: Learnings from 4 months of Image-Video VAE experiments
#15It’s been a while but I’m pretty sure the original deepfake used VAE as well. Super powerful idea and architecture
Re: Learnings from 4 months of Image-Video VAE experiments
#16Hi HN, I’m one of the two authors of the post and the Linum v2 text-to-video model ( https://news.ycombinator.com/item?id=46721488 ). We're releasing our Image-Video VAE (open weights) and a deep dive on how we built it. Happy to answer questions about the work!
Great work! I have been wondering what would it take to train with higher image bit depth (10 or 12b) and/or using camera footage only, not already heavily processed images? The usefulness of video generation in most professional use cases is limited because models are too end to end and completely contaminated with stock footage. Maybe quantities of training material needed is simply not there?
Not blaming you, but asking as I don’t usually have access to professionals working with video training.
Re: Learnings from 4 months of Image-Video VAE experiments
#17Hi HN, I’m one of the two authors of the post and the Linum v2 text-to-video model ( https://news.ycombinator.com/item?id=46721488 ). We're releasing our Image-Video VAE (open weights) and a deep dive on how we built it. Happy to answer questions about the work!
Great work! I have been wondering what would it take to train with higher image bit depth (10 or 12b) and/or using camera footage only, not already heavily processed images? The usefulness of video generation in most professional use cases is limited because models are too end to end and completely contaminated with stock footage. Maybe quantities of training material needed is simply not there? Not blaming you, but…
It’s a great question. In terms of pre-training even if they were was enough data at that quality, storing it and either demuxing it into raw frames OR compressing it with a sufficiently powerful encoder likely would cost a lot of $. But there’s a case to potentially use a much smaller subset of that data to dial in aesthetics towards the end of training. The gotcha there would come in terms of data diversity. Often you see that models will adapt to the new distribution and forget patterns from the old data. It’s hard to disentangle a model learning clarity of detail from concepts, so you might forget key ideas when picking up these details. Nevertheless maybe there is a way to use small amounts of this data in a RL finetuning setup? In our experience RL post training changes very little in the underlying model weights — so it might be a “light” enough touch to elicit the the desired details.
Re: Learnings from 4 months of Image-Video VAE experiments
#18Happy to see this architecture stil being handy for other people. Right now, I am using a heavily modified VAE for generation of synthetic satellite imagery. I am working with very limited dataset sizes, so I have chosen VAE over diffusion as a starting point.
Re: Learnings from 4 months of Image-Video VAE experiments
#19That's really cool work!
I've done some work in this area and here are my two cents:
1. Convolution-based architectures are terrible: I've trained Convolution based architectures and they were almost never scalable. Lately I've switched to transformer based AE and they are soo much better! We even managed to get Chinchilla-style scaling laws out of Transformer AE.
2. VAEs are terrible for downstream tasks: We've tried training Video diffusion models out of MAE and VAE (same architecture) and the MAE is hands down better.
3. This whole field is not science. There is no rigorous way of defining what a "good latent" really is. End-to-end methods (such as PixNerd) are the future, since they eliminate the need to hand-design and optimize the interface between separate components. That being said I've never seen a neural-field based video model and I've done some limited experiments on it with underwhelming results