Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Text-to-video model from scratch (2 brothers, 2 years, 2B params)

huggingface.co

Post reply on HN
Print view View original

21–27 of 27 posts

Re: Show HN: Text-to-video model from scratch (2 brothers, 2 years, 2B params)

#21

Post by schopra909 »

taherchhabra wrote: post #17

I want to build my own video model, just for learning purposes, is there any course which can teach end to end

I think YC just release video on the basics of diffusion, but honestly I don’t have a good end to end guide.

We’re going to write up going 0->1 on a video model (all the steps) over the coming months. But it likely won’t be a class or anything like that.

https://www.linum.ai/field-notes

We want to share our learnings with folks who are curious about the space - but don’t have time to make it a full class experience.

Hopefully karpathy does that with his courses in the future!

Re: Show HN: Text-to-video model from scratch (2 brothers, 2 years, 2B params)

#22

Post by schopra909 »

storystarling wrote: post #16
schopra909 wrote: post #11

Earlier quoted context omitted.

T5 Encoder is ~5B parameters so back of the envelope would be ~10GB of VRAM (it's in bfloat16). So, for 360p should take ~15 GB RAM (+/- a few GB based on the duration of video generated). We can update the code over the next day or two to provide the option for delete VAE after the text encoding is computed (to save on RAM). And then report back the GB consumed for 360p, 720p 2-5 seconds on GitHub so there are more…

The 5B text encoder feels disproportionate for a 2B video model. If the text portion is dominating your VRAM usage it really hurts the inference economics. Have you tried quantizing the T5? In my experience you can usually run these encoders in 8-bit or even 4-bit with negligible quality loss. Dropping that memory footprint would make this much more viable for consumer hardware.

Great idea! We haven’t tried it but def interested to see if that works as well.

When we started down this path, T5 was the standard (back in 2024).

Likely won’t be the text encoder for subsequent models, given its size (per your point) and age

Re: Show HN: Text-to-video model from scratch (2 brothers, 2 years, 2B params)

#23

Post by schopra909 »

storystarling wrote: post #16
schopra909 wrote: post #11

Earlier quoted context omitted.

T5 Encoder is ~5B parameters so back of the envelope would be ~10GB of VRAM (it's in bfloat16). So, for 360p should take ~15 GB RAM (+/- a few GB based on the duration of video generated). We can update the code over the next day or two to provide the option for delete VAE after the text encoding is computed (to save on RAM). And then report back the GB consumed for 360p, 720p 2-5 seconds on GitHub so there are more…

The 5B text encoder feels disproportionate for a 2B video model. If the text portion is dominating your VRAM usage it really hurts the inference economics. Have you tried quantizing the T5? In my experience you can usually run these encoders in 8-bit or even 4-bit with negligible quality loss. Dropping that memory footprint would make this much more viable for consumer hardware.

That all being said, you can just delete the T5 from memory after encoding the text so save on memory.

The 2B parameters will take up 4 Gb of memory but activations will be a lot more given size of context windows for video.

A 720p 5 second video is roughly 100K tokens of context

Re: Show HN: Text-to-video model from scratch (2 brothers, 2 years, 2B params)

#25

Post by mandeepj »

taherchhabra wrote: post #17

I want to build my own video model, just for learning purposes, is there any course which can teach end to end

> I want to build my own video model, just for learning purposes

Sorry, it might sound like a cliche, but try that as a prompt to a deep thinking and learning model, and see what comes out.

An expensive option: Look at Project #5 at https://bytebyteai.com/

Post reply on HN