Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Einsum in Depth

einsum.joelburget.com

Post reply on HN
Print view View original

11–20 of 33 posts

Re: Einsum in Depth

#11

Post by joshlk »

captainmuon wrote: post #10

Really interesting, I have been confused about the einsum function before. As a former physicist, I would also like to see the actual tensor notation for the examples. So instead of ij,jk something like $A_i^j B_j^k$ (imagine the math here instead of the LaTeX).

Tensors used in deep learning are not the same as the definition used by Physicists - blame the DL community for this :). So DL tensors are just N-dimensional arrays of data, and there is no concept of covariance and contravariance of the dimensions. You could think of DL tensors as Cartesian tensors and they don't need to conform to the same transformation laws that Physics tensors do.

Re: Einsum in Depth

#12

Post by gus_massa »

dan-robertson wrote: post #3

I was pretty confused by this for a while. I think the context I was missing is that this is about a function in nympy called ‘einsum’ which is somewhat related to Einstein summation notation. To write a little more: there are two things people mean by ‘tensor’. One is a kind of geometric object that corresponds to a multilinear map between vector spaces (or modules, I suppose), and another is a array indexed by k-tu…

I mostly agree.

> an even number is equivalent to reapplying the ‘twice’ rule many times

I never heard that extension and in my opinion is a bad idea. One nice property of the Einstein summation notation is that you can reorder the tensors(with their index), and the result does no change. If you allow 4x repetitions this is not valid).

---

Also, a nice property of tensors written in paper is that you can write the index as subscripsts or superscripts to remember how they change when there is a base change. You can only colapse a subscripst with a superscript, so the result of the operation does not depend on the choice of the base. (It would be very nice that Python can remember and check this too to avoid silly error.)

Re: Einsum in Depth

#13

Post by pama »

captainmuon wrote: post #10

Really interesting, I have been confused about the einsum function before. As a former physicist, I would also like to see the actual tensor notation for the examples. So instead of ij,jk something like $A_i^j B_j^k$ (imagine the math here instead of the LaTeX).

Einsum immediately clicked with me because in my past advanced classical mechanics courses such concise contractions of multi-index creatures were really the only way to make quick sense of complicated problems without the limitations of low-dimensional array notations. Physics tensors are of course different creatures than the simpler multidimensional arrays of PyTorch and co., but the simplified einsum notation still works very well. I ended up sometimes rewriting my Einsum code to plain tensor manipulation code in order to better work with collaborators who didnt like einsum, but I still experiment in my own hacks with einsum when I need to. Occasionally I felt that it would have been nice to also have the Levi-Civita symbol available in einsum, or to be able to use complex numbers and take complex conjugates, but these are all super-specialized requests and there often is a good way around them without modifying einsum.

Re: Einsum in Depth

#14

Post by sakex »

That's something I wish I had when I started looking at einsums. It gets interesting when you start thinking about optimum paths (like in the opt_einsum package), sharded/distributed einsums, and ML accelerators.

Re: Einsum in Depth

#15

Post by pama »

chpatrick wrote: post #6

For anyone who does a lot of einsum I highly recommend the einx package because the stock single-letter syntax very quickly becomes hard to read.

In your experience, why einx over einops+einsum?

Re: Einsum in Depth

#16

Post by dan-robertson »

gus_massa wrote: post #12
dan-robertson wrote: post #3

I was pretty confused by this for a while. I think the context I was missing is that this is about a function in nympy called ‘einsum’ which is somewhat related to Einstein summation notation. To write a little more: there are two things people mean by ‘tensor’. One is a kind of geometric object that corresponds to a multilinear map between vector spaces (or modules, I suppose), and another is a array indexed by k-tu…

I mostly agree. > an even number is equivalent to reapplying the ‘twice’ rule many times I never heard that extension and in my opinion is a bad idea. One nice property of the Einstein summation notation is that you can reorder the tensors(with their index), and the result does no change. If you allow 4x repetitions this is not valid). --- Also, a nice property of tensors written in paper is that you can write the in…

Ah I think I’m just wrong and the 4x thing shouldn’t be allowed

Re: Einsum in Depth

#17

Post by ziofill »

thomasahle wrote: post #7

I've found that thinking of tensors in terms of graphs make Einsums much more natural. For example, a matrix product MN, `a b, b c -> a c` is just two nodes with two edges each: `-a- M -b- N -c-`. Their `b` edges are connected, so the resulting graph has only two "free" edges `a` and `c`. That's how we know the result is another matrix. Once you look at tensors this way, a number of things that are normally tricky wi…

A few years ago I wrote a note (never published) on how many products can be seen in this way https://arxiv.org/abs/1903.01366

Re: Einsum in Depth

#18

Post by thomasahle »

ziofill wrote: post #17
thomasahle wrote: post #7

I've found that thinking of tensors in terms of graphs make Einsums much more natural. For example, a matrix product MN, `a b, b c -> a c` is just two nodes with two edges each: `-a- M -b- N -c-`. Their `b` edges are connected, so the resulting graph has only two "free" edges `a` and `c`. That's how we know the result is another matrix. Once you look at tensors this way, a number of things that are normally tricky wi…

A few years ago I wrote a note (never published) on how many products can be seen in this way https://arxiv.org/abs/1903.01366

This is beautiful! I see we even decided on the same "vectorization tensor" and notation, which makes Kathri-Rao, and all the other "matrix products" much more intuitive!

Re: Einsum in Depth

#19

Post by ziofill »

thomasahle wrote: post #7

I've found that thinking of tensors in terms of graphs make Einsums much more natural. For example, a matrix product MN, `a b, b c -> a c` is just two nodes with two edges each: `-a- M -b- N -c-`. Their `b` edges are connected, so the resulting graph has only two "free" edges `a` and `c`. That's how we know the result is another matrix. Once you look at tensors this way, a number of things that are normally tricky wi…

I just opened your book, very nice! I really like the derivatives. You went above and beyond with latex diagrams ^^

Re: Einsum in Depth

#20

Post by 6gvONxR4sf7o »

thomasahle wrote: post #7

I've found that thinking of tensors in terms of graphs make Einsums much more natural. For example, a matrix product MN, `a b, b c -> a c` is just two nodes with two edges each: `-a- M -b- N -c-`. Their `b` edges are connected, so the resulting graph has only two "free" edges `a` and `c`. That's how we know the result is another matrix. Once you look at tensors this way, a number of things that are normally tricky wi…

I use a similar notation, but never quite found a satisfactory notation for elementwise operations (e.g. `-M-a + -b`, especially broadcasted ones which I end up doing as `-A-B- + -c 1-`) or for denoting what derivatives are with respect to. Using differentials gets around some of the latter, but still, I was never quite satisfied. Any chance you've found nice options there?
Post reply on HN