Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

constant.meiring.nz

Post reply on HN
Print view View original

81–90 of 141 posts

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#81

Post by barrkel »

WhereWhyWhat wrote: post #61
barrkel wrote: post #21

I started out using my old GTX 1080 on Thursday, could generate 512x512 just fine. That's in 8G of VRAM. It worked well on the hlky branch using webui (built using gradio). Seeing that training etc. is much more memory intensive, and wanting to get faster results, I bought an RTX 3090, which has 24G of VRAM. However it maxes out at about 1024x512, only twice as many pixels. Observing the card with GPUZ, it never actu…

How many it/s do you get with 3090 compared to the 1080? My 1080 Ti gets around 2.5it/s with the k_lms sampler.

Where the 1080 would do about 2 it/sec, the 3090 does about 10 it/sec.

When I do batches, it slows down but not linearly; if 1x does 10 it/sec, 2x does about 6 it/sec. Batching is the other upside of more VRAM.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#82

Post by rrobukef »

mordymoop wrote: post #80
SuperCuber wrote: post #35

Earlier quoted context omitted.

I feel like I'm going insane. Everyone says 512x512 should work with 8gb but when I do it I get: CUDA out of memory. Tried to allocate 3.00 GiB (GPU 0; 8.00 GiB total capacity; 5.62 GiB already allocated; 0 bytes free; 5.74 GiB reserved in total by PyTorch) any ideas? I have a 3060ti with 8gb vram... with 448x448 I get: CUDA out of memory. Tried to allocate 902.00 MiB (GPU 0; 8.00 GiB total capacity; 6.73 GiB already…

For some reason — no idea why — this problem went away when I set n_samples to 1 and scale to 10.0 or less. Why these parameters would impact memory usage, I don’t know, but the image quality seems fine, afaict.

n_samples is the batching number. Total memory used scales like "Model Mem Size + n_samples * Batch Mem Size". The memory needed for a batch is smaller than the model but not trivial.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#83

Post by cbozeman »

tbalsam wrote: post #73
barrkel wrote: post #21

I started out using my old GTX 1080 on Thursday, could generate 512x512 just fine. That's in 8G of VRAM. It worked well on the hlky branch using webui (built using gradio). Seeing that training etc. is much more memory intensive, and wanting to get faster results, I bought an RTX 3090, which has 24G of VRAM. However it maxes out at about 1024x512, only twice as many pixels. Observing the card with GPUZ, it never actu…

Recommendations: - Linux w/ Display drivers on CPU (and just...ditch miniconda please) - Use lower FP precision mode if available to use the tensor cores (also to double "effective" memory) - Batch things! - I don't know what the max resolution of the diffusion network is, you may have to just simply tile it past a certain point (with overlap please! ;P Hope that helps somewhat. A 3090 should be more than enough for…

I ordered an NVIDIA Tesla K80 off eBay (and the power adapter... and the blower fan shroud, etc.) and intend to install it when it arrives around Thursday or Friday. I'm hoping that after I install the NVIDIA Linux datacenter drivers I'll be able to use the card with SD.

My only worry is that because the K80 is two GPUs on one board, that it might only utilize one of them, with only 12 GB of VRAM instead of both chips and all 24 GB.

5992 CUDA cores and 24 GB VRAM would be a pretty decent SD accelerator for only $150.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#84

Post by rrobukef »

baobabKoodaa wrote: post #71
prettydeep wrote: post #68

Earlier quoted context omitted.

Use the original SD repo. But modify the txt2img.py according to: https://github.com/CompVis/stable-diffusion/issues/86#issuec...

I now did everything I could to constrain the memory usage of the original SD repo, I was finally able to get it to run, and it produced green squares as output :( What I did: - scripts/txt2img.py, function - load_model_from_config, line - 63, change from: model.cuda() to model.cuda().half() - removed invisible watermarking - reduced n_samples to 1 - reduced resolution to 256x256 - removed sfw filter Just can't get i…

Your model is overflowing/underflowing generating NaNs. I got it with memory optimised, increased resolution (multiples of 32, 384 x 384) and full precision while keeping it in 4 GB.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#85

Post by hedora »

I had good luck with these directions, which let you run inside a docker container:

https://github.com/AshleyYakeley/stable-diffusion-rocm

I had to make the one line change suggested in issue #3 to get it to run under 8GB.

radeontop suggests 4GB might work.

I also had to add this environment variable to make it work on my unsupported radeon 6600xt:

HSA_OVERRIDE_GFX_VERSION=10.3.0

It takes under two minutes per batch of 5 images with the --turbo option.

(Base OS is manjaro; using the distro's version of docker; not the flatpack docker package.)

If you don't have a GPU, paperspace will rent you an appropriate VM.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#86

Post by hedora »

baobabKoodaa wrote: post #48

It's unfortunate that this article doesn't specify the amount of VRAM needed, other than specifying it's "less than 10Gb". I have 6,1Gb of VRAM and I tried to follow the article until eventually encountering an "unable to allocate memory" error. (I'm now trying to run basujindal's repo as an alternative.)

See my other comment in this article. I'm running with 8, but it never uses more than 6GB.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#87

Post by jeroenhd »

SuperCuber wrote: post #47
jeroenhd wrote: post #40

Earlier quoted context omitted.

Others may have reduced the batch size (n_samples) to reduce the memory load. A lower batch size will significantly help with the memory consumption. This comment: https://news.ycombinator.com/item?id=32710550 talks about running SD with 8GiB of VRAM and mentions needing to reduce this parameter to 1 to get it to output right.

This helped and I finally generated something larger than 256x256 :D thanks

If you're okay waiting a while linger and have plenty of RAM, https://github.com/bes-dev/stable_diffusion.openvino has a somewhat CPU-optimized version as well that relies on system memory rather than VRAM.

My laptop takes about 6 seconds per iteration so it's significantly slower, but if you're willing to wait I bet you'll have a much easier time plugging more RAM into your system than adding VRAM.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#88

Post by TrinaryWorksToo »

barrkel wrote: post #21

I started out using my old GTX 1080 on Thursday, could generate 512x512 just fine. That's in 8G of VRAM. It worked well on the hlky branch using webui (built using gradio). Seeing that training etc. is much more memory intensive, and wanting to get faster results, I bought an RTX 3090, which has 24G of VRAM. However it maxes out at about 1024x512, only twice as many pixels. Observing the card with GPUZ, it never actu…

WSL2 is a thing you should look in to.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#89

Post by XorNot »

baobabKoodaa wrote: post #54
capableweb wrote: post #49

Earlier quoted context omitted.

Reduce the resolution and run with half-precision instead of full-precision and you should be able to avoid OOM errors. Author seems to have had 8GB VRAM available, so I'm guessing that's the "minimum required" for their solution.

It's not possible to halve the precision further. The precision was already dropped from float32 to float16 in the OP. I now used parameters to drop the resolution to 256x256, and now it's running, but it's somehow broken. Every output image it produces is literally a green square.

You need to be in full precision mode in that case. Running on my AMD card this was necessary.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#90

Post by [deleted] »

baobabKoodaa wrote: post #53
schleck8 wrote: post #39

Earlier quoted context omitted.

Use halfprecision float and/or the optimized forks https://github.com/basujindal/stable-diffusion https://github.com/neonsecret/stable-diffusion Or the hlky webui, that is optimized too. http://rentry.co/kretard

I've been trying to get the basujindal fork to work, but it seems to be putting all work on the CPU. I've been running the example txt2img prompt for 30 minutes now and it's still not finished. It has reserved 4Gb memory from the GPU, but the GPU doesn't appear to be doing any work, only CPU is doing work.

[deleted]
Post reply on HN