Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

constant.meiring.nz

Post reply on HN
Print view View original

71–80 of 141 posts

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#71

Post by baobabKoodaa »

prettydeep wrote: post #68
baobabKoodaa wrote: post #53

Earlier quoted context omitted.

I've been trying to get the basujindal fork to work, but it seems to be putting all work on the CPU. I've been running the example txt2img prompt for 30 minutes now and it's still not finished. It has reserved 4Gb memory from the GPU, but the GPU doesn't appear to be doing any work, only CPU is doing work.

Use the original SD repo. But modify the txt2img.py according to: https://github.com/CompVis/stable-diffusion/issues/86#issuec...

I now did everything I could to constrain the memory usage of the original SD repo, I was finally able to get it to run, and it produced green squares as output :(

What I did:

- scripts/txt2img.py, function - load_model_from_config, line - 63, change from: model.cuda() to model.cuda().half()

- removed invisible watermarking

- reduced n_samples to 1

- reduced resolution to 256x256

- removed sfw filter

Just can't get it to work and it's not producing an error message or anything that I could debug it with.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#72

Post by ladberg »

barrkel wrote: post #21

I started out using my old GTX 1080 on Thursday, could generate 512x512 just fine. That's in 8G of VRAM. It worked well on the hlky branch using webui (built using gradio). Seeing that training etc. is much more memory intensive, and wanting to get faster results, I bought an RTX 3090, which has 24G of VRAM. However it maxes out at about 1024x512, only twice as many pixels. Observing the card with GPUZ, it never actu…

Just checking, are you also using the same GPU for rendering your desktop? If so then try switching over to your integrated GPU or the 1080 if it’s still attached so you can leave 100% of the 3090 available to the network.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#73

Post by tbalsam »

barrkel wrote: post #21

I started out using my old GTX 1080 on Thursday, could generate 512x512 just fine. That's in 8G of VRAM. It worked well on the hlky branch using webui (built using gradio). Seeing that training etc. is much more memory intensive, and wanting to get faster results, I bought an RTX 3090, which has 24G of VRAM. However it maxes out at about 1024x512, only twice as many pixels. Observing the card with GPUZ, it never actu…

Recommendations: - Linux w/ Display drivers on CPU (and just...ditch miniconda please) - Use lower FP precision mode if available to use the tensor cores (also to double "effective" memory) - Batch things! - I don't know what the max resolution of the diffusion network is, you may have to just simply tile it past a certain point (with overlap please! ;P

Hope that helps somewhat. A 3090 should be more than enough for what you're doing, I'm stuck with P100s at best for me! (Cost :'( )

Best of luck! :D :)

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#74

Post by vimy »

SuperCuber wrote: post #35
hwers wrote: post #30

If you have even just 4gb stable diffusion will run fine if u go for 448x448 instead (basically the same quality).

I feel like I'm going insane. Everyone says 512x512 should work with 8gb but when I do it I get: CUDA out of memory. Tried to allocate 3.00 GiB (GPU 0; 8.00 GiB total capacity; 5.62 GiB already allocated; 0 bytes free; 5.74 GiB reserved in total by PyTorch) any ideas? I have a 3060ti with 8gb vram... with 448x448 I get: CUDA out of memory. Tried to allocate 902.00 MiB (GPU 0; 8.00 GiB total capacity; 6.73 GiB already…

How much ram is your gpu using before you start stable diffusion? You can check with ‘nvidia-smi’ in terminal.

The not-optimized release works with my 2070 with 8 gb ram.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#75

Post by raphlinus »

naillo wrote: post #55
moffkalast wrote: post #42

Earlier quoted context omitted.

> I feel like I'm going insane. That's the world of running machine learning models for you. Why would anything ever work the first time right? Or at least the 10th time...

Which is so silly since ML models should be the most portable thing in the world. It's just a series of math operations, not a bunch of OS/hardware specific API calls or something like that. We should be at a stage where each ML model is boiled down to a simple executable with zero dependencies at this point.

Agree 100% and I spend a fair amount of time wondering why this hasn't happened. I built piet-gpu-hal because I couldn't find any abstraction layer over compute shaders that supports precompiled shaders. A motivated person absolutely could write shaders to do all the operations needed by Stable Diffusion, and ship a binary in the megabyte range (obviously not counting the models themselves). That would support Metal, Vulkan, and D3D12. The only thing holding this back is a will to build it.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#76

Post by yoru-sulfur »

tbalsam wrote: post #73
barrkel wrote: post #21

I started out using my old GTX 1080 on Thursday, could generate 512x512 just fine. That's in 8G of VRAM. It worked well on the hlky branch using webui (built using gradio). Seeing that training etc. is much more memory intensive, and wanting to get faster results, I bought an RTX 3090, which has 24G of VRAM. However it maxes out at about 1024x512, only twice as many pixels. Observing the card with GPUZ, it never actu…

Recommendations: - Linux w/ Display drivers on CPU (and just...ditch miniconda please) - Use lower FP precision mode if available to use the tensor cores (also to double "effective" memory) - Batch things! - I don't know what the max resolution of the diffusion network is, you may have to just simply tile it past a certain point (with overlap please! ;P Hope that helps somewhat. A 3090 should be more than enough for…

> and just...ditch miniconda please

Why the recommendation to stop using miniconda?

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#77

Post by LanternLight83 »

baobabKoodaa wrote: post #54
capableweb wrote: post #49

Earlier quoted context omitted.

Reduce the resolution and run with half-precision instead of full-precision and you should be able to avoid OOM errors. Author seems to have had 8GB VRAM available, so I'm guessing that's the "minimum required" for their solution.

It's not possible to halve the precision further. The precision was already dropped from float32 to float16 in the OP. I now used parameters to drop the resolution to 256x256, and now it's running, but it's somehow broken. Every output image it produces is literally a green square.

The green square issue has been well known, particularly on AMD cards, and I believe the solution is... full precision :c But idk, I haven't had that issue. My issue's that I can run it in <4GB VRAM, but can only do a couple dozen images before some memory leak or smth drives it out of memory (effects my 2070S too, but only after many more images). Restarting it isn't too bad, but it's enough to have me looking to using either if two AMD APU's that I have on hand.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#78

Post by the-golden-one »

MattRix wrote: post #64
barrkel wrote: post #21

I started out using my old GTX 1080 on Thursday, could generate 512x512 just fine. That's in 8G of VRAM. It worked well on the hlky branch using webui (built using gradio). Seeing that training etc. is much more memory intensive, and wanting to get faster results, I bought an RTX 3090, which has 24G of VRAM. However it maxes out at about 1024x512, only twice as many pixels. Observing the card with GPUZ, it never actu…

Have you tried something like 1024x768? Going to full 1024x1024 would double your VRAM usage so I can see why that wouldn’t work. For my uses, the real benefit of having more VRAM is that you can generate more images simultaneously. My 3080 can generate only one 512x512 in 7 seconds but three 384x384 in that same timeframe. It’s allowed me to generate grids of hundreds of images in just a few minutes.

I can squeeze 640x512 out of my Mobile RTX 3080 with 8Gb VRAM (as the secondary GPU). Anything more than that fails.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#80

Post by mordymoop »

SuperCuber wrote: post #35
hwers wrote: post #30

If you have even just 4gb stable diffusion will run fine if u go for 448x448 instead (basically the same quality).

I feel like I'm going insane. Everyone says 512x512 should work with 8gb but when I do it I get: CUDA out of memory. Tried to allocate 3.00 GiB (GPU 0; 8.00 GiB total capacity; 5.62 GiB already allocated; 0 bytes free; 5.74 GiB reserved in total by PyTorch) any ideas? I have a 3060ti with 8gb vram... with 448x448 I get: CUDA out of memory. Tried to allocate 902.00 MiB (GPU 0; 8.00 GiB total capacity; 6.73 GiB already…

For some reason — no idea why — this problem went away when I set n_samples to 1 and scale to 10.0 or less. Why these parameters would impact memory usage, I don’t know, but the image quality seems fine, afaict.
Post reply on HN