God I love einsum so much. I discovered it in grad school and it made my life so much nicer, definitely recommend learning it, you can do some wicked powerful stuff. (One of the first things I asked one of the NVIDIA guys when they were showing off cuPy a few years back was whether it had einsum.)
Einsum in Depth
21–30 of 33 posts
Re: Einsum in Depth
#22I was pretty confused by this for a while. I think the context I was missing is that this is about a function in nympy called ‘einsum’ which is somewhat related to Einstein summation notation. To write a little more: there are two things people mean by ‘tensor’. One is a kind of geometric object that corresponds to a multilinear map between vector spaces (or modules, I suppose), and another is a array indexed by k-tu…
Academic CS, moreso than any of the other STEM subjects, is like that whose line is it anyway meme - all the words are completely made up and absolutely none of the results matter. So a vector isn't a vector, a tensor isn't a tensor, einsum isn't actually Einstein summation, automatic differentiation often gives derivatives for things that aren't differentiable, a neural network has nothing to do with neurons, etc et…
Vec4 is a vector, but yes C++ base vec is not.
In some ways CS is more correct with the modern (abstract) algebra definition of a vector, vs the historical didactics arrow concept.
Tensor has been abused, it is a property of bilinear maps, and as matrix multiplication is a bilinear operation there is a concept overlap there.
Vectors are really just elements of a vector space, not necessarily arrows.
Re: Einsum in Depth
#23Really interesting, I have been confused about the einsum function before. As a former physicist, I would also like to see the actual tensor notation for the examples. So instead of ij,jk something like $A_i^j B_j^k$ (imagine the math here instead of the LaTeX).
This is a good idea, though one problem is that Einsum notation (as realized in Numpy and Pytorch) doesn't support the notion of co-contravariance, and the site is based on their Einsum notation. I could potentially add the variances for the examples, though that would move away from how the site currently works (where the information about the reduction comes only from the einsum input).
Re: Einsum in Depth
#24I've found that thinking of tensors in terms of graphs make Einsums much more natural. For example, a matrix product MN, `a b, b c -> a c` is just two nodes with two edges each: `-a- M -b- N -c-`. Their `b` edges are connected, so the resulting graph has only two "free" edges `a` and `c`. That's how we know the result is another matrix. Once you look at tensors this way, a number of things that are normally tricky wi…
I use a similar notation, but never quite found a satisfactory notation for elementwise operations (e.g. `-M-a + -b`, especially broadcasted ones which I end up doing as `-A-B- + -c 1-`) or for denoting what derivatives are with respect to. Using differentials gets around some of the latter, but still, I was never quite satisfied. Any chance you've found nice options there?
The broadcasting using "1-" (or the order-1 copy/kronecker tensor) is actually pretty nice, I think. It allows a lot of nice manipulations, and make the low rank of the matrices really clear etc.
Addition does feel a bit hacky, but it may just be necessary for any associative notation. At least it means that rules such as distribution works the same as with classical notation.
I also wrote this tensorgrad library to do symbolic calculations using tensor networks: https://github.com/thomasahle/tensorgrad it keeps track of what you are taking derivatives with respect to. But I agree it would be nice to show more explicitly.
Re: Einsum in Depth
#25I've found that thinking of tensors in terms of graphs make Einsums much more natural. For example, a matrix product MN, `a b, b c -> a c` is just two nodes with two edges each: `-a- M -b- N -c-`. Their `b` edges are connected, so the resulting graph has only two "free" edges `a` and `c`. That's how we know the result is another matrix. Once you look at tensors this way, a number of things that are normally tricky wi…
I just opened your book, very nice! I really like the derivatives. You went above and beyond with latex diagrams ^^
At this point I feel like I need to write a tikz library just for tensor diagrams...
Re: Einsum in Depth
#26I've found that thinking of tensors in terms of graphs make Einsums much more natural. For example, a matrix product MN, `a b, b c -> a c` is just two nodes with two edges each: `-a- M -b- N -c-`. Their `b` edges are connected, so the resulting graph has only two "free" edges `a` and `c`. That's how we know the result is another matrix. Once you look at tensors this way, a number of things that are normally tricky wi…
I had been working on a library for visualizing tensor contractions (so-called Penrose tensor diagrams), https://github.com/Quantum-Flytrap/tensor-diagrams, with an example (a tutorial stub) https://p.migdal.pl/art-tensor-diagrams/.
My background is in quantum physics, and it is a useful tool for understanding some aspects of entanglement. Here, the idea was to show its use in machine learning in general and deep learning in particular.
Re: Einsum in Depth
#27Earlier quoted context omitted.
I just opened your book, very nice! I really like the derivatives. You went above and beyond with latex diagrams ^^
At this point I feel like I need to write a tikz library just for tensor diagrams...
There are some libraries, but the style is so not standardized that I understand your desire to write your own.
Back in 2020 I wanted to write a library based on manim to animate the contractions, but I never came around to it (and manim back then was less well-documented than it is now).
Re: Einsum in Depth
#28Earlier quoted context omitted.
At this point I feel like I need to write a tikz library just for tensor diagrams...
There are some libraries, but the style is so not standardized that I understand your desire to write your own. Back in 2020 I wanted to write a library based on manim to animate the contractions, but I never came around to it (and manim back then was less well-documented than it is now).
> There are some libraries, but the style is so not standardized that I understand your desire to write your own.
Do you have some recommendations? I'm currently using tikz' graphdrawing library because it supports subgraphs. This is necessary for handling addition, functions and derivatives. However, the force-based layouting doesn't work with subgraphs, which causes a lot of trouble.
> Back in 2020 I wanted to write a library based on manim to animate the contractions, but I never came around to it (and manim back then was less well-documented than it is now).
Manim is on the TODO list :-) https://github.com/thomasahle/tensorgrad/issues/7 If you feel like writing some code, I think it could work very well with tensorgrad's step-by-step derivations.
Re: Einsum in Depth
#29I've found that thinking of tensors in terms of graphs make Einsums much more natural. For example, a matrix product MN, `a b, b c -> a c` is just two nodes with two edges each: `-a- M -b- N -c-`. Their `b` edges are connected, so the resulting graph has only two "free" edges `a` and `c`. That's how we know the result is another matrix. Once you look at tensors this way, a number of things that are normally tricky wi…
I had been working on a library for visualizing tensor contractions (so-called Penrose tensor diagrams), https://github.com/Quantum-Flytrap/tensor-diagrams , with an example (a tutorial stub) https://p.migdal.pl/art-tensor-diagrams/ . My background is in quantum physics, and it is a useful tool for understanding some aspects of entanglement. Here, the idea was to show its use in machine learning in general and deep l…
Awesome! Are you able to handle sums of diagrams like (-M- + -a b-)-c etc?
Re: Einsum in Depth
#30I found this video to be fantastic