Is python in the fast path? Why not rewrite in a performant language for a XXX% speedup?
The += operator is almost certainly calling some method on sends out the real work to some tuned hardware-specific framework written in a fast language.
Speedup from switch to +=
11–20 of 85 posts
Re: Speedup from switch to +=
#12I guess this is the beauty of making a model open source.
it is a hell of a good case study that is for sure.
Re: Speedup from switch to +=
#13If they're seeing these kinds of gains from relatively minor changes to their Python code, I can't help but wonder how much faster the model would run in a compiled language or a language with a good JIT (way more optimization work's gone into the mainstream Javascript runtimes than CPython). I'd assumed that overall performance in Stable Diffusion was limited by the code running on the GPU, with Python performance b…
[deleted]
Re: Speedup from switch to +=
#14One comment asks about putting it all on one line, and this is where interpreted languages without a JIT kinda blow.
Many times I have had to decide if my Python code would be more legible or get free performance.
The thing I like about JavaScript is that I can _usually_ trust the JIT to make my code faster than I could, meaning I can focus entirely on writing clean code.
P.S. you can always hand optimize. If you do, just comment the heck out of it.
Re: Speedup from switch to +=
#15I wonder what version of Python they were using? I'm wondering, because recent version have improved performance a lot. 3.11 is much faster than 3.10, and what's in 3.12 is already much faster than 3.11.
The upstream Stable Diffusion uses python 3.8: https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a...
Thanks.
Re: Speedup from switch to +=
#16Is python in the fast path? Why not rewrite in a performant language for a XXX% speedup?
In this case I believe python is faster by a few months.
Jokes aside this is pytorch so this is compiled to C++ or cuda, the problem likely comes from the different functions that are called for += vs +
Re: Speedup from switch to +=
#17It’s not clear a JIT compiled language would help much here unless the operations were baked into the JIT itself (which would have to identify the memory savings of an in-place call).
Re: Speedup from switch to +=
#18If they're seeing these kinds of gains from relatively minor changes to their Python code, I can't help but wonder how much faster the model would run in a compiled language or a language with a good JIT (way more optimization work's gone into the mainstream Javascript runtimes than CPython). I'd assumed that overall performance in Stable Diffusion was limited by the code running on the GPU, with Python performance b…
I don't know anything about stable diffusion, but I've been optimizing a lot of prime-field arithmetic in Rust lately, and we experienced a similar speedup going from `+ x` to `+= x` (for scalars and especially for composite structures like vectors and polynomials).
Re: Speedup from switch to +=
#19If they're seeing these kinds of gains from relatively minor changes to their Python code, I can't help but wonder how much faster the model would run in a compiled language or a language with a good JIT (way more optimization work's gone into the mainstream Javascript runtimes than CPython). I'd assumed that overall performance in Stable Diffusion was limited by the code running on the GPU, with Python performance b…
I don't think this is necessarily a minor change -- += and + are operators. I have no familiarity with this library, but I think _forward(...) works on tensors of something like that, they are probably big chunky data structures. += probably saves a copy or whatever.
Re: Speedup from switch to +=
#20If they're seeing these kinds of gains from relatively minor changes to their Python code, I can't help but wonder how much faster the model would run in a compiled language or a language with a good JIT (way more optimization work's gone into the mainstream Javascript runtimes than CPython). I'd assumed that overall performance in Stable Diffusion was limited by the code running on the GPU, with Python performance b…
This is PyTorch code, so the Python is setting up a bunch of kernels that are executed on the GPU. The switch from + to += might allow two of those kernels to be fused together or something, and that could lead to the large performance gain.
The Python part only runs a handful of times so JIT vs. non-JIT doesn't really make a difference.