Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speedup from switch to +=

github.com

Post reply on HN
Print view View original

51–60 of 85 posts

Re: Speedup from switch to +=

#51

Post by Arnavion »

mgraczyk wrote: post #44
staticassertion wrote: post #42

Earlier quoted context omitted.

Love to see it. A perfect example of why this optimization can't be done automatically - in the case of `else` you're working with a mutable reference to `x` passed in, which means that now your function is mutating something it used to not mutate. A "safe" way to do this is still straightforward, I think. from copy import copy def _forward(self, x, context=None): x = x.contiguous() if x.device.type == 'mps' else x x…

That's not safe if the problem is the in place mutation. You will still mutate x while reading from it.

staticassertion's point is that the current code's usage of `+=` mutates the x that was passed in by the caller, and their suggestion is to copy x into a function local before mutating it, which is similar to how the original `+` code also worked on a function local x (the result of `attn1() + x`).

Re: Speedup from switch to +=

#52

Post by nodja »

Waterluvian wrote: post #14

One comment asks about putting it all on one line, and this is where interpreted languages without a JIT kinda blow. Many times I have had to decide if my Python code would be more legible or get free performance. The thing I like about JavaScript is that I can _usually_ trust the JIT to make my code faster than I could, meaning I can focus entirely on writing clean code. P.S. you can always hand optimize. If you do,…

> Many times I have had to decide if my Python code would be more legible or get free performance.

This is rarely an option that has presented itself to me. If there's a clear performance issue in my code then I probably picked the wrong algorithm or my code has a bug, unless you decided for some reason to do heavy calculations in raw python. If you're doing operations on big chunks of data you should always use something like numpy or jax.

Even OPs issue the clear reason is that it's doing an operation in place instead of creating a copy, for ML models this can only be done at inference time and not training time since you need to keep track of the whole network, hence why the code was in it's unoptimized state.

Re: Speedup from switch to +=

#53

Post by [deleted] »

mgraczyk wrote: post #44
staticassertion wrote: post #42

Earlier quoted context omitted.

Love to see it. A perfect example of why this optimization can't be done automatically - in the case of `else` you're working with a mutable reference to `x` passed in, which means that now your function is mutating something it used to not mutate. A "safe" way to do this is still straightforward, I think. from copy import copy def _forward(self, x, context=None): x = x.contiguous() if x.device.type == 'mps' else x x…

That's not safe if the problem is the in place mutation. You will still mutate x while reading from it.

[deleted]

Re: Speedup from switch to +=

#55

Post by olliej »

Is this a lookup overhead thing or a memcpy based overhead regression? In the case of the latter it seems like this may result in an unexpected mutation of the source data?

Re: Speedup from switch to +=

#56

Post by sdenton4 »

jonas21 wrote: post #20
JonathonW wrote: post #7

If they're seeing these kinds of gains from relatively minor changes to their Python code, I can't help but wonder how much faster the model would run in a compiled language or a language with a good JIT (way more optimization work's gone into the mainstream Javascript runtimes than CPython). I'd assumed that overall performance in Stable Diffusion was limited by the code running on the GPU, with Python performance b…

This is PyTorch code, so the Python is setting up a bunch of kernels that are executed on the GPU. The switch from + to += might allow two of those kernels to be fused together or something, and that could lead to the large performance gain. The Python part only runs a handful of times so JIT vs. non-JIT doesn't really make a difference.

I haven't played much with torch, but the game is generally that you have a graph of computations which gets JIT compiled into GPU ops. The compiler may have more or less competence at finding modifications (eg, 'fusions') to reduce the number of GPU ops required to perform the computation.

See, for example, XLA: https://www.tensorflow.org/xla

It looks like maybe nvFuser is an equivalent library for pytorch? https://pytorch.org/blog/introducing-nvfuser-a-deep-learning...

Re: Speedup from switch to +=

#58

Post by [deleted] »

eru wrote: post #4

I wonder what version of Python they were using? I'm wondering, because recent version have improved performance a lot. 3.11 is much faster than 3.10, and what's in 3.12 is already much faster than 3.11.

[deleted]

Re: Speedup from switch to +=

#59

Post by mgraczyk »

Arnavion wrote: post #51
mgraczyk wrote: post #44

Earlier quoted context omitted.

That's not safe if the problem is the in place mutation. You will still mutate x while reading from it.

staticassertion's point is that the current code's usage of `+=` mutates the x that was passed in by the caller, and their suggestion is to copy x into a function local before mutating it, which is similar to how the original `+` code also worked on a function local x (the result of `attn1() + x`).

That's not the problem though. The problem is that the += operations mutate x in place, but the right hand side reads from x. There is no copy you could insert like that to fix this. You would have to do the following, for example

Instead of x = op(x) + x -> x += op(x)

Do

x_copy = copy(x) x += op(x_copy)

If you do x_copy += op(x_copy)

Then you are still mutating x_copy while op() reads it.

EDIT: I also don't think copy(x) will copy the actual tensor data, although I'm not super familiar with Pytorch.

Re: Speedup from switch to +=

#60

Post by teo_zero »

But wait... x+=y is equivalent to x=x+y not to x=y+x. Only if + is commutative, then the three are equivalent. Are we sure the + operation is commutatve for this type of data? And does the compiler know it?

It would be interesting to check whether changing every expression to x=x+y has a performance more similar to += or to ...+x

Post reply on HN