Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speedup from switch to +=

github.com

Post reply on HN
Print view View original

21–30 of 85 posts

Re: Speedup from switch to +=

#21

Post by luizfzs »

JonathonW wrote: post #7

If they're seeing these kinds of gains from relatively minor changes to their Python code, I can't help but wonder how much faster the model would run in a compiled language or a language with a good JIT (way more optimization work's gone into the mainstream Javascript runtimes than CPython). I'd assumed that overall performance in Stable Diffusion was limited by the code running on the GPU, with Python performance b…

I've always assumed Python was interpreted until I heard Nuitka [1].

It would be interesting to get a benchmark using CPython vs Nuitka related to this change.

[1] https://github.com/Nuitka/Nuitka

Re: Speedup from switch to +=

#22

Post by brrrrrm »

jonas21 wrote: post #20
JonathonW wrote: post #7

If they're seeing these kinds of gains from relatively minor changes to their Python code, I can't help but wonder how much faster the model would run in a compiled language or a language with a good JIT (way more optimization work's gone into the mainstream Javascript runtimes than CPython). I'd assumed that overall performance in Stable Diffusion was limited by the code running on the GPU, with Python performance b…

This is PyTorch code, so the Python is setting up a bunch of kernels that are executed on the GPU. The switch from + to += might allow two of those kernels to be fused together or something, and that could lead to the large performance gain. The Python part only runs a handful of times so JIT vs. non-JIT doesn't really make a difference.

The Python code is run every time.

Re: Speedup from switch to +=

#23

Post by dahfizz »

bee_rider wrote: post #11
dahfizz wrote: post #5

Is python in the fast path? Why not rewrite in a performant language for a XXX% speedup?

The += operator is almost certainly calling some method on sends out the real work to some tuned hardware-specific framework written in a fast language.

So python is marshalling data to and from an ffi in the fast path? That sounds even worse

Re: Speedup from switch to +=

#24

Post by ironhaven »

Because of operator overloading "+=" can call a more optimized method than "+". If this code was written in a language without operator overloading I don't think this would be a very interesting pull request. THis could be a example of why some people don't like operator overloading and why some programing languages (java, zig, etc) do not implment the feature.

Re: Speedup from switch to +=

#25

Post by stingraycharles »

brrrrrm wrote: post #22
jonas21 wrote: post #20

Earlier quoted context omitted.

This is PyTorch code, so the Python is setting up a bunch of kernels that are executed on the GPU. The switch from + to += might allow two of those kernels to be fused together or something, and that could lead to the large performance gain. The Python part only runs a handful of times so JIT vs. non-JIT doesn't really make a difference.

The Python code is run every time.

Yes but not nearly as much as the GPU code, which I think is what the parent is saying; it’s typically not the bottleneck.

Re: Speedup from switch to +=

#26

Post by thayne »

sshine wrote: post #18
JonathonW wrote: post #7

If they're seeing these kinds of gains from relatively minor changes to their Python code, I can't help but wonder how much faster the model would run in a compiled language or a language with a good JIT (way more optimization work's gone into the mainstream Javascript runtimes than CPython). I'd assumed that overall performance in Stable Diffusion was limited by the code running on the GPU, with Python performance b…

I don't know anything about stable diffusion, but I've been optimizing a lot of prime-field arithmetic in Rust lately, and we experienced a similar speedup going from `+ x` to `+= x` (for scalars and especially for composite structures like vectors and polynomials).

For composite structures that isn't too surprising, but for scalars, I would have expected llvm to optimize the addition and assignment into a single in place addition.

Re: Speedup from switch to +=

#28

Post by lumost »

stingraycharles wrote: post #25
brrrrrm wrote: post #22

Earlier quoted context omitted.

The Python code is run every time.

Yes but not nearly as much as the GPU code, which I think is what the parent is saying; it’s typically not the bottleneck.

Depends on the field, python is almost never the bottleneck in CV - but it almost always a bottleneck in NLP or Search research.

Regardless, the code in the github PR should be running strictly in pytorch for the heavy lifting. Clearly something is touching python that shouldn't be, or must be for frustrating reasons.

Re: Speedup from switch to +=

#29

Post by thrown_22 »

JonathonW wrote: post #7

If they're seeing these kinds of gains from relatively minor changes to their Python code, I can't help but wonder how much faster the model would run in a compiled language or a language with a good JIT (way more optimization work's gone into the mainstream Javascript runtimes than CPython). I'd assumed that overall performance in Stable Diffusion was limited by the code running on the GPU, with Python performance b…

This is like saying that passing a struct vs a pointer to a struct is a minor change for C code. I mean it's just one extra * !

Re: Speedup from switch to +=

#30

Post by bee_rider »

dahfizz wrote: post #23
bee_rider wrote: post #11

Earlier quoted context omitted.

The += operator is almost certainly calling some method on sends out the real work to some tuned hardware-specific framework written in a fast language.

So python is marshalling data to and from an ffi in the fast path? That sounds even worse

I'm not sure this is the conventional use of the phrase "fast path."

But anyway, the idea is usually that the Python code calls out to the framework with operations that are in some way "large," and so the overhead is not so significant.

Python probably doesn't have to do any marshaling, hypothetically the framework could just return an object that represents a pointer. Then the python code sends that pointer to another framework method.

Post reply on HN