Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Text-to-video model from scratch (2 brothers, 2 years, 2B params)

huggingface.co

Post reply on HN
Print view View original

11–20 of 27 posts

Re: Show HN: Text-to-video model from scratch (2 brothers, 2 years, 2B params)

#11

Post by schopra909 »

dsrtslnd23 wrote: post #10
schopra909 wrote: post #7

Earlier quoted context omitted.

Per the RAM comment, you may able to get it run locally with two tweaks: https://github.com/Linum-AI/linum-v2/blob/298b1bb9186b5b9ff6... 1) Free up the t5 as soon as the text is encoded, so you reclaim GPU RAM 2) Manual Layer Offloading; move layers off GPU once they're done being used to free up space for the remaining layers + activations

Any idea on the minimum VRAM footprint with those tweaks? 20GB seems high for a 2B model. I guess the T5 encoder is responsible for that.

T5 Encoder is ~5B parameters so back of the envelope would be ~10GB of VRAM (it's in bfloat16). So, for 360p should take ~15 GB RAM (+/- a few GB based on the duration of video generated).

We can update the code over the next day or two to provide the option for delete VAE after the text encoding is computed (to save on RAM). And then report back the GB consumed for 360p, 720p 2-5 seconds on GitHub so there are more accurate numbers.

Beyond the 10 GB from the T5, there's just a lot of VRAM taken up by the context window of 720p video (even though the model itself is 2B parameters).

Re: Show HN: Text-to-video model from scratch (2 brothers, 2 years, 2B params)

#16

Post by storystarling »

schopra909 wrote: post #11
dsrtslnd23 wrote: post #10

Earlier quoted context omitted.

Any idea on the minimum VRAM footprint with those tweaks? 20GB seems high for a 2B model. I guess the T5 encoder is responsible for that.

T5 Encoder is ~5B parameters so back of the envelope would be ~10GB of VRAM (it's in bfloat16). So, for 360p should take ~15 GB RAM (+/- a few GB based on the duration of video generated). We can update the code over the next day or two to provide the option for delete VAE after the text encoding is computed (to save on RAM). And then report back the GB consumed for 360p, 720p 2-5 seconds on GitHub so there are more…

The 5B text encoder feels disproportionate for a 2B video model. If the text portion is dominating your VRAM usage it really hurts the inference economics.

Have you tried quantizing the T5? In my experience you can usually run these encoders in 8-bit or even 4-bit with negligible quality loss. Dropping that memory footprint would make this much more viable for consumer hardware.

Re: Show HN: Text-to-video model from scratch (2 brothers, 2 years, 2B params)

#19

Post by schopra909 »

whywhywhywhy wrote: post #18

> We kept a “lab notebook” of all our experiments in Notion Couldn't find a link to this, is this public?

Not public yet — we’re going to clean it up so it’s readable and release it as blog posts. First one will be everything you need to know on building a VAE for image and video. Should be out in a few weeks. We’re figuring out the write balance between spending time writing and all the work we have on our plate for the next model.

If you’re interested in this stuff, keep an eye on field notes (our blog).

Re: Show HN: Text-to-video model from scratch (2 brothers, 2 years, 2B params)

#20

Post by schopra909 »

schopra909 wrote: post #19
whywhywhywhy wrote: post #18

> We kept a “lab notebook” of all our experiments in Notion Couldn't find a link to this, is this public?

Not public yet — we’re going to clean it up so it’s readable and release it as blog posts. First one will be everything you need to know on building a VAE for image and video. Should be out in a few weeks. We’re figuring out the write balance between spending time writing and all the work we have on our plate for the next model. If you’re interested in this stuff, keep an eye on field notes (our blog).

https://www.linum.ai/field-notes
Post reply on HN