Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

rockt.github.io

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#21

Post by jesuslop »

krastanov wrote: post #16
Xcelerate wrote: post #6

I remember when I was learning matrix calculus and realized at some point that it was much simpler to convert everything to index notation, perform all operations, then convert everything back to standard notation at the end. It became almost comically simple, because you're "just" working with labeled scalars at that point. To be fair, it's convenient to memorize some of the more commonly used expressions (like ∂tr(…

Roger Penrose (the famous mathematician) has been saying the same thing for a couple of decades (going even further than that, check out Penrose diagrams).

And Penrose diagrams (tensor networks) became a prominent example of String Diagram in the monoidal category of vector spaces over a field.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#22

Post by pishpash »

joppy wrote: post #11
ajkjk wrote: post #10

Earlier quoted context omitted.

Yes, it's incredibly faster. You do a lot of tensor algebra in a relativity class.

To the point where writing one large sigma and listing indices under it at the start of each line would significantly slow you down? I found I had to do this mentally during my own relativity class, just to figure out what each expression meant.

Then you're doing it wrong. The point of notation (or learning any new language) is to operate in that language, not to translate back and forth.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#23

Post by joppy »

krastanov wrote: post #19
mhh__ wrote: post #13

Earlier quoted context omitted.

Absolutely yes. Doing tensor magic without einstein notation will make you shoot yourself, and even you don't someone else will if publish it. The only real problem I have with it personally is the abstraction of upper and lower indices, which I constantly forget the conventions as to which is which.

This is interesting... I love the notation, but mainly because how upper and lower indices make it easy to distinguish vectors and forms (not that it matters in ML where everything is "euclidean").

Upper and lower indices are great, but are still unrelated to the convention of dropping the big sigma at the front.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#24

Post by claudius »

yorwba wrote: post #2

The problem with einsum is that you have to explicitly specify the mapping between dimensions and indices every time, without any way to enforce consistency. It would be more ergonomic if each tensor had labeled dimensions. That would prevent the kind of silly mistake where you mix up the ordering of dimensions and only notice it when you later change the shape of the tensors so the different dimensions no longer mat…

ITensor [1] has somewhat pioneered this concept in tensor networks for condensed matter physics. If I understand correctly, Uni10 [2], a similar project, is even working on a graphical interface for such networks so that you can "draw" the network and have the computer figure out the best contraction order.

In my own code I’ve recently also implemented such named indices (only had "einstein summation"-like contraction specifiers before) and they make tensor contractions so much simpler to write, especially since you can simply overload operator * and have it figure out which legs need to go together.

As a side effect, it also enforces your algorithms to make sense because you can't simply add two tensors living on different (but equal-dimensional) vector spaces together anymore.

[1] https://itensor.org

[2] https://uni10.gitlab.io/

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#26

Post by joppy »

pishpash wrote: post #22
joppy wrote: post #11

Earlier quoted context omitted.

To the point where writing one large sigma and listing indices under it at the start of each line would significantly slow you down? I found I had to do this mentally during my own relativity class, just to figure out what each expression meant.

Then you're doing it wrong. The point of notation (or learning any new language) is to operate in that language, not to translate back and forth.

It's not actually a new notation though, it's literally just deleting a part of the old notation and then getting the reader to fill that part back in. While this is fine for calculations, I would rather that when equations are actually presented, that one extra piece of information is explicitly given.

I guess a similar thing that happens on the programming language side would be leaving out types when the compiler can infer them. This is great, and can make code more concise, but when reading an API I would really like the types written down explicitly, rather than having the play the part of the compiler.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#27

Post by mcabbott »

joppy wrote: post #8

I've never really understood the point of Einstein notation, as a piece of mathematical notation. Is writing something like A[i, j] * B[j, k] really that much faster than writing something like Sum[j](A[i, j] * B[j, k])? Especially when you have to check the left hand side of the equality sign just to know which indices to sum over, it seems like making things less clear for a minuscule saving on ink.

If you have just one term, then leaving out the summation sign saves little. But in larger expressions, the rule applies sums per term, and then the savings in clutter can be quite large. For example, here v_i is not summed, and the term with z_b has two sums:

A_ia x_a + v_i + x_a y_a ( w_i + B_ib z_b )

You should not need to check the LHS, x_a y_a should always mean the dot product of these two. At least this is the convention among the heavy addicts; among lighter users you will sometimes find other setups.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#28

Post by mcabbott »

claudius wrote: post #24
yorwba wrote: post #2

The problem with einsum is that you have to explicitly specify the mapping between dimensions and indices every time, without any way to enforce consistency. It would be more ergonomic if each tensor had labeled dimensions. That would prevent the kind of silly mistake where you mix up the ordering of dimensions and only notice it when you later change the shape of the tensors so the different dimensions no longer mat…

ITensor [1] has somewhat pioneered this concept in tensor networks for condensed matter physics. If I understand correctly, Uni10 [2], a similar project, is even working on a graphical interface for such networks so that you can "draw" the network and have the computer figure out the best contraction order. In my own code I’ve recently also implemented such named indices (only had "einstein summation"-like contractio…

I made a thing to audit that you are consistent with your indices [1] as an alternative to explicit named-tensor objects. And found, but have not used, another package in a similar spirit [2].

Although to be honest, simply writing A[n,μ,ν,c] etc. (using different letters for different spaces) makes it pretty easy to visually check that you are getting this right. This is one of the attractions of the notation, even on paper. Unfortunately np.einsum's string notation makes this harder to see, as the indices aren't adjacent to the variable name.

[1] https://github.com/mcabbott/TensorCast.jl#checking (Julia)

[2] https://github.com/ofnote/tsalib (Python)

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#29

Post by CuriousSkeptic »

yorwba wrote: post #2

The problem with einsum is that you have to explicitly specify the mapping between dimensions and indices every time, without any way to enforce consistency. It would be more ergonomic if each tensor had labeled dimensions. That would prevent the kind of silly mistake where you mix up the ordering of dimensions and only notice it when you later change the shape of the tensors so the different dimensions no longer mat…

Why is it that mathematicians are so comfortable with using ordering to encode semantics?

Evident from this thread it seems many developer recognize the need to use proper labels to simplify reasoning. Doesn’t mathematicians have similar desires?

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#30

Post by stared »

Right now I work on Tensor diagram notation for deep learning (for project "thinking in tensors, writing in PyTorch").

To read more about it, see: https://medium.com/@pmigdal/in-the-topic-of-diagrams-i-did-w... (obviously, I refer to the post).

And if you want to create some, here is a short demo: https://jsfiddle.net/stared/8huz5gy7/

In general, I want to expand that to tensor structure (e.g. n, channel, x, y) plus, translate it to the Einstein summation convention.

Post reply on HN