Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

constant.meiring.nz

Post reply on HN
Print view View original

41–50 of 141 posts

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#41

Post by hwers »

SuperCuber wrote: post #35
hwers wrote: post #30

If you have even just 4gb stable diffusion will run fine if u go for 448x448 instead (basically the same quality).

I feel like I'm going insane. Everyone says 512x512 should work with 8gb but when I do it I get: CUDA out of memory. Tried to allocate 3.00 GiB (GPU 0; 8.00 GiB total capacity; 5.62 GiB already allocated; 0 bytes free; 5.74 GiB reserved in total by PyTorch) any ideas? I have a 3060ti with 8gb vram... with 448x448 I get: CUDA out of memory. Tried to allocate 902.00 MiB (GPU 0; 8.00 GiB total capacity; 6.73 GiB already…

Oh sorry I guess i need to mention that you need to put the text encoder on the cpu (or precompute the text embedding somehow). (Im using a custom codebase to make that possible idk how trivial that is to achieve with StableDiffusionPipeline.) Only the unet and vae should be on the gpu.

For your case with 8 gb you shouldn’t need to do either of those things (run it all on gpu), just make sure you have batch size 1 and are using the fp16 version.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#42

Post by moffkalast »

SuperCuber wrote: post #35
hwers wrote: post #30

If you have even just 4gb stable diffusion will run fine if u go for 448x448 instead (basically the same quality).

I feel like I'm going insane. Everyone says 512x512 should work with 8gb but when I do it I get: CUDA out of memory. Tried to allocate 3.00 GiB (GPU 0; 8.00 GiB total capacity; 5.62 GiB already allocated; 0 bytes free; 5.74 GiB reserved in total by PyTorch) any ideas? I have a 3060ti with 8gb vram... with 448x448 I get: CUDA out of memory. Tried to allocate 902.00 MiB (GPU 0; 8.00 GiB total capacity; 6.73 GiB already…

> I feel like I'm going insane.

That's the world of running machine learning models for you. Why would anything ever work the first time right? Or at least the 10th time...

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#43

Post by andybak »

alkonaut wrote: post #27

What’s the easiest way of using SD on a Windows box? Can I run it off a Linux live USB or can it run directly under Windows? Edit: never mind this is the missing guide I had been looking for

On Windows the app Visions of Chaos (mostly)-automates the installs for dozens of ML models including SD: https://softology.pro/tutorials/tensorflow/tensorflow.htm and provides a fairly respectable UI.

It's also updated almost daily and tracks the latest features where possible.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#44

Post by blagie »

fragmede wrote: post #34
Llamamoe wrote: post #31

Earlier quoted context omitted.

Who's paying for all the Google Collab notebooks I've been seeing around? Can I really just start and keep using it for free?

Yup, totally free (with a Google account). It's run as a learning resource and there's an upsell to Collab+ and Collab Pro, but for running StableDiffusion it makes it very easy to get started!

I think it's less of an upsell and more of a data collection and a market positioning. Google would like to, for example:

1. Be central in the machine learning ecosystem. This has broad ripple effects, such as recruiting.

2. Doing things there means Google can track how you use machine learning. This can be used for everything from understanding trends in machine learning, to, again, robustly identifying individuals for recruiting efforts.

It seems like the cost is nominal at Google scale for what Google is getting. I suspect the pricing for the higher-end services is less a money-making scheme, as at some point, free is no longer sustainable (and if unlimited CPU were free, that would be prone to abuse / misuse / overuse / wasteful use). The amount of money Google makes there is nominal at Google scale.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#46

Post by mugivarra69 »

qayxc wrote: post #17
mugivarra69 wrote: post #6

anyone tried to quantize or use bfloat?

blfoat would indeed be nice. It's supported on a wide range of hardware (basically all mid-range to high-end Intel CPUs since 2013, AMD MI5 and up compute cards, ARM NEON and NVIDIA cards since Pascal [10-series, 2016!]). It could speed up calculations and significantly reduce memory requirements. I'd expect slightly worse results, though. edit: also https://github.com/basujindal/stable-diffusion/pull/103

neat. thanks!

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#47

Post by SuperCuber »

jeroenhd wrote: post #40
SuperCuber wrote: post #35

Earlier quoted context omitted.

I feel like I'm going insane. Everyone says 512x512 should work with 8gb but when I do it I get: CUDA out of memory. Tried to allocate 3.00 GiB (GPU 0; 8.00 GiB total capacity; 5.62 GiB already allocated; 0 bytes free; 5.74 GiB reserved in total by PyTorch) any ideas? I have a 3060ti with 8gb vram... with 448x448 I get: CUDA out of memory. Tried to allocate 902.00 MiB (GPU 0; 8.00 GiB total capacity; 6.73 GiB already…

Others may have reduced the batch size (n_samples) to reduce the memory load. A lower batch size will significantly help with the memory consumption. This comment: https://news.ycombinator.com/item?id=32710550 talks about running SD with 8GiB of VRAM and mentions needing to reduce this parameter to 1 to get it to output right.

This helped and I finally generated something larger than 256x256 :D thanks

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#48

Post by baobabKoodaa »

It's unfortunate that this article doesn't specify the amount of VRAM needed, other than specifying it's "less than 10Gb". I have 6,1Gb of VRAM and I tried to follow the article until eventually encountering an "unable to allocate memory" error. (I'm now trying to run basujindal's repo as an alternative.)

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#49

Post by capableweb »

baobabKoodaa wrote: post #48

It's unfortunate that this article doesn't specify the amount of VRAM needed, other than specifying it's "less than 10Gb". I have 6,1Gb of VRAM and I tried to follow the article until eventually encountering an "unable to allocate memory" error. (I'm now trying to run basujindal's repo as an alternative.)

Reduce the resolution and run with half-precision instead of full-precision and you should be able to avoid OOM errors. Author seems to have had 8GB VRAM available, so I'm guessing that's the "minimum required" for their solution.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#50

Post by sp332 »

WithinReason wrote: post #16
DarthNebo wrote: post #12

Always wondered why we can't virtualize VRAM like how we did for VMs.

Good question. Bandwidth of dual channel DDR4-3600: 48 GB/s Bandwidth of PCIe 4 x16: 26 GB/s Bandiwdth of 3090 GDDR6X memory: 935.8 GB/s Since neural network evaluation is usually bandwidth limited, it's possible that pushing the data through PCI-E from CPU to GPU is actually slower than doing the evaluation on CPU only for typical neural networks. https://www.microway.com/knowledge-center-articles/performan... https…

Ok, but at least it would run.
Post reply on HN