81–85 of 85 posts
He's the author of the essay "How Perl Saved the Genome Project", the books "Network Programming with Perl" and "Writing Apache Modules with Perl and C", and a number of Perl packages including CGI.pm - which helped power the dot-com era - and GD.pm.
Ok, I work on PyTorch, so probably should clear up some misconceptions in this thread. 1. In PyTorch (and other array programming libraries like Numpy), the operations being passed around are tensors/arrays (i.e. large chunks of memory). Thus, += is overloaded to mean "in-place write" to the arrays. So, `+` vs `+=` is the equivalent of a: float[1000] b: float[1000] for i in [0, 1000]: b[i] = a[i] + 2 vs. a: float[100…
> Generally, it's best to have this optimization be done automatically by an optimizing compiler.
What compiler should be optimizing this operation?
There are comments on the commit reporting errors under certain conditions.
Ok, I work on PyTorch, so probably should clear up some misconceptions in this thread. 1. In PyTorch (and other array programming libraries like Numpy), the operations being passed around are tensors/arrays (i.e. large chunks of memory). Thus, += is overloaded to mean "in-place write" to the arrays. So, `+` vs `+=` is the equivalent of a: float[1000] b: float[1000] for i in [0, 1000]: b[i] = a[i] + 2 vs. a: float[100…
Thanks for the input. Before I start throwing += into my PyTorch code can you explain what you mean here: > Generally, it's best to have this optimization be done automatically by an optimizing compiler. What compiler should be optimizing this operation? There are comments on the commit reporting errors under certain conditions.
There's many different paths to optimizing compilers folks use with PyTorch. One with close integration is NVFuser (see https://www.reddit.com/r/MachineLearning/comments/xa75km/p_p...), although there are other compilers like ONNXRuntime.
Yes, handling autograd (during training) is a whole different thing, and not all compilers support that.
Plot twist: it breaks the code...? > Changing this back to the original implementation fixed an error I was getting when doing textual inversion on Windows https://github.com/lstein/stable-diffusion/commit/62863ac586...
The problem here comes down to not knowing whether you’re allowed to modify a value in-place or not, because it’s not clear who owns it: it wasn’t written down anywhere, and in stable-diffusion alone it was fine to mutate it, but textual-inversion did something so it wasn’t (perhaps passing it something it expected to not be mutated). This is a moderately common type of bug that can be extraordinarily difficult to diagnose—it’s unusually easy to pinpoint here because it promptly raises a RuntimeError—and which is statically impossible in Rust, because the whole “am I allowed to mutate it” thing is resolved in the type system.