Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

constant.meiring.nz

Post reply on HN
Print view View original

11–20 of 141 posts

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#11

Post by forgingahead »

blfr wrote: post #5

Is there a similar guide for Linux/Ubuntu with some sort of light sandboxing, at least python virtual virtual environment?

https://github.com/basujindal/stable-diffusion

I use this on my Ubuntu 18 machine, works nicely on a GPU with 8GB VRAM.

As usual, some python dependency nonsense to sort out even with Anaconda, but pretty quick and easy to get up and running.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#13

Post by kernelsanderz »

DarthNebo wrote: post #12

Always wondered why we can't virtualize VRAM like how we did for VMs.

You kind of can - projects like deepspeed (https://www.deepspeed.ai/) enable running a model that is larger than in VRAM through various tricks like moving weights from regular system RAM into VRAM between layers. Can come with a performance hit though depending on the model, of course.

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#14

Post by Datagenerator »

T0Bi wrote: post #3
BTCarel wrote: post #2

Can't believe how awesome these generated images are. Thank you for the guide!

If you want to have a really good experience using stable diffusion, use this guide: https://rentry.org/GUItard - includes a nice GUI - txt2img and img2img - upscaling, face correction - many more

Can this be used with the optimizedSD by basujindal?

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#16

Post by WithinReason »

DarthNebo wrote: post #12

Always wondered why we can't virtualize VRAM like how we did for VMs.

Good question.

    Bandwidth of dual channel DDR4-3600: 48 GB/s
    Bandwidth of PCIe 4 x16: 26 GB/s
    Bandiwdth of 3090 GDDR6X memory: 935.8 GB/s
Since neural network evaluation is usually bandwidth limited, it's possible that pushing the data through PCI-E from CPU to GPU is actually slower than doing the evaluation on CPU only for typical neural networks.

https://www.microway.com/knowledge-center-articles/performan...

https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#17

Post by qayxc »

mugivarra69 wrote: post #6

anyone tried to quantize or use bfloat?

blfoat would indeed be nice. It's supported on a wide range of hardware (basically all mid-range to high-end Intel CPUs since 2013, AMD MI5 and up compute cards, ARM NEON and NVIDIA cards since Pascal [10-series, 2016!]).

It could speed up calculations and significantly reduce memory requirements. I'd expect slightly worse results, though.

edit: also https://github.com/basujindal/stable-diffusion/pull/103

Re: Running Stable Diffusion on Your GPU with Less Than 10Gb of VRAM

#20

Post by matsemann »

DarthNebo wrote: post #12

Always wondered why we can't virtualize VRAM like how we did for VMs.

For training you can often divide the batch size by n (and then only apply the backprop gradient stuff after each n batches for it to be mathematically equivalent). At a cost of speed, though.
Post reply on HN