Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

constant.meiring.nz

Post reply on HN
Print view View original

61–70 of 141 posts

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#61

Post by WhereWhyWhat »

barrkel wrote: post #21

I started out using my old GTX 1080 on Thursday, could generate 512x512 just fine. That's in 8G of VRAM. It worked well on the hlky branch using webui (built using gradio). Seeing that training etc. is much more memory intensive, and wanting to get faster results, I bought an RTX 3090, which has 24G of VRAM. However it maxes out at about 1024x512, only twice as many pixels. Observing the card with GPUZ, it never actu…

How many it/s do you get with 3090 compared to the 1080?

My 1080 Ti gets around 2.5it/s with the k_lms sampler.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#62

Post by pixl97 »

barrkel wrote: post #21

I started out using my old GTX 1080 on Thursday, could generate 512x512 just fine. That's in 8G of VRAM. It worked well on the hlky branch using webui (built using gradio). Seeing that training etc. is much more memory intensive, and wanting to get faster results, I bought an RTX 3090, which has 24G of VRAM. However it maxes out at about 1024x512, only twice as many pixels. Observing the card with GPUZ, it never actu…

Maybe look in to booting off a USB stick as a means to test this. I wouldn't be surprised if there were some kind of driver reservation I Windows causing this issue.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#63

Post by capableweb »

WhereWhyWhat wrote: post #61
barrkel wrote: post #21

I started out using my old GTX 1080 on Thursday, could generate 512x512 just fine. That's in 8G of VRAM. It worked well on the hlky branch using webui (built using gradio). Seeing that training etc. is much more memory intensive, and wanting to get faster results, I bought an RTX 3090, which has 24G of VRAM. However it maxes out at about 1024x512, only twice as many pixels. Observing the card with GPUZ, it never actu…

How many it/s do you get with 3090 compared to the 1080? My 1080 Ti gets around 2.5it/s with the k_lms sampler.

On a 2080 Ti I get around 8 it/s with k_lms

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#64

Post by MattRix »

barrkel wrote: post #21

I started out using my old GTX 1080 on Thursday, could generate 512x512 just fine. That's in 8G of VRAM. It worked well on the hlky branch using webui (built using gradio). Seeing that training etc. is much more memory intensive, and wanting to get faster results, I bought an RTX 3090, which has 24G of VRAM. However it maxes out at about 1024x512, only twice as many pixels. Observing the card with GPUZ, it never actu…

Have you tried something like 1024x768? Going to full 1024x1024 would double your VRAM usage so I can see why that wouldn’t work.

For my uses, the real benefit of having more VRAM is that you can generate more images simultaneously. My 3080 can generate only one 512x512 in 7 seconds but three 384x384 in that same timeframe. It’s allowed me to generate grids of hundreds of images in just a few minutes.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#65

Post by cube2222 »

capableweb wrote: post #60
Karuma wrote: post #33

Earlier quoted context omitted.

Google is paying, and yes, you can, but they will disconnect you after a while. And if you abuse it too much, you won't be able to use it until the following day... You can also buy Colab Pro and Colab Pro+, which have fewer limitations and faster GPUs.

How fast is the Colab stuff? Is Colab Pro/Pro+ a lot faster too? I run it locally and can generate images with 50 steps in about 6 seconds per image, would it be faster for me to use Colab Free/Pro/Pro+?

In my usage Colab and Colab Pro were similar, with plain Colab occasionally OOMing during model loading. That said I've actually been seeing times slower than yours on Colab and I think they're slower than on my RTX 3080. ~15 secs per image. I'm not sure why, though.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#66

Post by vaughnegut »

SuperCuber wrote: post #35
hwers wrote: post #30

If you have even just 4gb stable diffusion will run fine if u go for 448x448 instead (basically the same quality).

I feel like I'm going insane. Everyone says 512x512 should work with 8gb but when I do it I get: CUDA out of memory. Tried to allocate 3.00 GiB (GPU 0; 8.00 GiB total capacity; 5.62 GiB already allocated; 0 bytes free; 5.74 GiB reserved in total by PyTorch) any ideas? I have a 3060ti with 8gb vram... with 448x448 I get: CUDA out of memory. Tried to allocate 902.00 MiB (GPU 0; 8.00 GiB total capacity; 6.73 GiB already…

On my 3070 I get that error unless I set my batch size to 1. My typical setup is to do six batches of one and it works fine (although I minimize the number of visible things on my screen while it's running). This reliably produces one image every 7-8 seconds.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#67

Post by XorNot »

I have this running on my fairly mundane Radeon 5600XT at about 1 minute per image generated (under rootless podman, which is the real cool news to me) which isn't bad all things considered. Definitely get some interesting sounds from coil whine when it's going.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#68

Post by prettydeep »

baobabKoodaa wrote: post #53
schleck8 wrote: post #39

Earlier quoted context omitted.

Use halfprecision float and/or the optimized forks https://github.com/basujindal/stable-diffusion https://github.com/neonsecret/stable-diffusion Or the hlky webui, that is optimized too. http://rentry.co/kretard

I've been trying to get the basujindal fork to work, but it seems to be putting all work on the CPU. I've been running the example txt2img prompt for 30 minutes now and it's still not finished. It has reserved 4Gb memory from the GPU, but the GPU doesn't appear to be doing any work, only CPU is doing work.

Use the original SD repo. But modify the txt2img.py according to:

https://github.com/CompVis/stable-diffusion/issues/86#issuec...

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#70

Post by WithinReason »

sp332 wrote: post #50
WithinReason wrote: post #16

Earlier quoted context omitted.

Good question. Bandwidth of dual channel DDR4-3600: 48 GB/s Bandwidth of PCIe 4 x16: 26 GB/s Bandiwdth of 3090 GDDR6X memory: 935.8 GB/s Since neural network evaluation is usually bandwidth limited, it's possible that pushing the data through PCI-E from CPU to GPU is actually slower than doing the evaluation on CPU only for typical neural networks. https://www.microway.com/knowledge-center-articles/performan... https…

Ok, but at least it would run.

It already does, on the CPU.
Post reply on HN