21–30 of 38 posts
Earlier quoted context omitted.
Why am I with a 3090 @ 3 it/s? Am I doing something heavily wrong? All through WSL2
is/s depends on resolution and other factors like batch size. What are you getting for 512x image?
[1]: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issu...
I accidentally stumbled upon this and did not expect such a speedup. It seems anything less than cu118 does not properly support the RTX4090 (or H100). Bumping to cuda12.2 with pytorch2.0.1+cu118 made my SDXL go 50% faster and ESRGAN 80% faster on the 4090.
So few people have the technology that I knew I'd be spending significant time figuring out solutions to problems.
The other reason is that I'd wait a few years and get some 6090 with 4x the VRAM.
Earlier quoted context omitted.
You can also run PyTorch cu121 nightly builds, These also allow `torch.compile` to function properly with dynamic input, which should net another 30%+ boost to SD.
This comment brings a tear to my eye.
This is an example of a setup.py that correctly installs the accelerated PyTorch for your platform:
https://github.com/comfyanonymous/ComfyUI/blob/9aeaac4af5e19...
As you can see, never merged. For philosophical reasons I believe. The author wanted to merge it earlier and changed his mind.
Like why make end users deal with this at all? The ROI from a layperson choosing these details is very low.
Python has a packaging problem, this is well known. Fixing setuptools would be highest yield. Other package tooling can't install PyTorch, for example: https://github.com/python-poetry/poetry/issues/6409#issuecom....
PyTorch itself is wonkily packaged. But I'm sure they have a good reason for this. Anyway, it goes to show that you can put a huge amount of effort into fixing this particular problem that everyone touching this technology has, and the maintainers everywhere will go nowhere with it. And I don't think this is a "me" problem, because there is so much demand for packaging PyTorch correctly - all the easy UIs, etc.
Earlier quoted context omitted.
This comment brings a tear to my eye.
The underlying problem is the community's decision to make users manage this in the first place. This is an example of a setup.py that correctly installs the accelerated PyTorch for your platform: https://github.com/comfyanonymous/ComfyUI/blob/9aeaac4af5e19... As you can see, never merged. For philosophical reasons I believe. The author wanted to merge it earlier and changed his mind. Like why make end users deal wit…
CUDA and ROCM make this an intractable problem. Basically there is no way to sanely package everything users need, and the absolutely enormous, cude/rocm versioned pytorch packages with missing libs are already a compromise.
TBH the whole ecosystem is not meant to be for end user inference anyway.
I accidentally stumbled upon this and did not expect such a speedup. It seems anything less than cu118 does not properly support the RTX4090 (or H100). Bumping to cuda12.2 with pytorch2.0.1+cu118 made my SDXL go 50% faster and ESRGAN 80% faster on the 4090.
You can also run PyTorch cu121 nightly builds, These also allow `torch.compile` to function properly with dynamic input, which should net another 30%+ boost to SD.
Edit: After a bit more research it looks like scaled dot product attention in Pytorch 2 provides much the same benefit as xformers without the need for xformers proper. Nice.
Earlier quoted context omitted.
This comment brings a tear to my eye.
The underlying problem is the community's decision to make users manage this in the first place. This is an example of a setup.py that correctly installs the accelerated PyTorch for your platform: https://github.com/comfyanonymous/ComfyUI/blob/9aeaac4af5e19... As you can see, never merged. For philosophical reasons I believe. The author wanted to merge it earlier and changed his mind. Like why make end users deal wit…
I am talking about dynamic shapes in torch.compile.
You seem to be talking about software packaging. You also make heavy use of the word "this" without it being clear what "this" is.
Earlier quoted context omitted.
The underlying problem is the community's decision to make users manage this in the first place. This is an example of a setup.py that correctly installs the accelerated PyTorch for your platform: https://github.com/comfyanonymous/ComfyUI/blob/9aeaac4af5e19... As you can see, never merged. For philosophical reasons I believe. The author wanted to merge it earlier and changed his mind. Like why make end users deal wit…
Sorry, no idea what you are talking about. I am talking about dynamic shapes in torch.compile. You seem to be talking about software packaging. You also make heavy use of the word "this" without it being clear what "this" is.
For instance, the torch.compile thing we are talking about is (last I checked) totally irrelevant for those UIs because they are still using the Stability AI implementation, not diffusers package that Huggingface checks for graph breaks. This may extend to SDXL.
Earlier quoted context omitted.
You can also run PyTorch cu121 nightly builds, These also allow `torch.compile` to function properly with dynamic input, which should net another 30%+ boost to SD.
Is there a trick to getting pytorch+cu121 and xformers to play nicely together? All the xformers packages I can find are torch==2.01+cu118. Edit: After a bit more research it looks like scaled dot product attention in Pytorch 2 provides much the same benefit as xformers without the need for xformers proper. Nice.
xformers still has a tiny performance benefit (especially at higher resolutions IIRC), but yeah, PyTorch's SDP is good.