Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

rockt.github.io

Post reply on HN
Print view View original

11–20 of 50 posts

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#11

Post by joppy »

ajkjk wrote: post #10
joppy wrote: post #8

I've never really understood the point of Einstein notation, as a piece of mathematical notation. Is writing something like A[i, j] * B[j, k] really that much faster than writing something like Sum[j](A[i, j] * B[j, k])? Especially when you have to check the left hand side of the equality sign just to know which indices to sum over, it seems like making things less clear for a minuscule saving on ink.

Yes, it's incredibly faster. You do a lot of tensor algebra in a relativity class.

To the point where writing one large sigma and listing indices under it at the start of each line would significantly slow you down? I found I had to do this mentally during my own relativity class, just to figure out what each expression meant.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#12

Post by lenticular »

joppy wrote: post #8

I've never really understood the point of Einstein notation, as a piece of mathematical notation. Is writing something like A[i, j] * B[j, k] really that much faster than writing something like Sum[j](A[i, j] * B[j, k])? Especially when you have to check the left hand side of the equality sign just to know which indices to sum over, it seems like making things less clear for a minuscule saving on ink.

It's more succinct, and the thing is that in most physics, repeated indices really almost always are sums. When you're doing fluid mechanics or anything like that, it's tensors all the time. The terseness is nice once you get used to it.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#13

Post by mhh__ »

joppy wrote: post #8

I've never really understood the point of Einstein notation, as a piece of mathematical notation. Is writing something like A[i, j] * B[j, k] really that much faster than writing something like Sum[j](A[i, j] * B[j, k])? Especially when you have to check the left hand side of the equality sign just to know which indices to sum over, it seems like making things less clear for a minuscule saving on ink.

Absolutely yes.

Doing tensor magic without einstein notation will make you shoot yourself, and even you don't someone else will if publish it.

The only real problem I have with it personally is the abstraction of upper and lower indices, which I constantly forget the conventions as to which is which.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#14

Post by ssivark »

joppy wrote: post #8

I've never really understood the point of Einstein notation, as a piece of mathematical notation. Is writing something like A[i, j] * B[j, k] really that much faster than writing something like Sum[j](A[i, j] * B[j, k])? Especially when you have to check the left hand side of the equality sign just to know which indices to sum over, it seems like making things less clear for a minuscule saving on ink.

It's not just about the sum. The upper and lower indices are semantically crucial. Eg: vectors have upper indices and gradients have lower indices. It's nonsensical to add the two, so the core step in gradient descent doesn't make sense. To add the two, you need the metric (Hessian) to raise the index of the gradient. And just like that, voila, you've discovered natural gradients!

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#15

Post by frutiger »

joppy wrote: post #11
ajkjk wrote: post #10

Earlier quoted context omitted.

Yes, it's incredibly faster. You do a lot of tensor algebra in a relativity class.

To the point where writing one large sigma and listing indices under it at the start of each line would significantly slow you down? I found I had to do this mentally during my own relativity class, just to figure out what each expression meant.

> To the point where writing one large sigma and listing indices under it at the start of each line would significantly slow you down?

Yes (in my relativity & QFT classes). But the timesaving aspect was not that important for me. The notation enabled the intuition of “zipping” together these somewhat unwieldy mathematical objects, and that was the clincher.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#16

Post by krastanov »

Xcelerate wrote: post #6

I remember when I was learning matrix calculus and realized at some point that it was much simpler to convert everything to index notation, perform all operations, then convert everything back to standard notation at the end. It became almost comically simple, because you're "just" working with labeled scalars at that point. To be fair, it's convenient to memorize some of the more commonly used expressions (like ∂tr(…

Roger Penrose (the famous mathematician) has been saying the same thing for a couple of decades (going even further than that, check out Penrose diagrams).

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#19

Post by krastanov »

mhh__ wrote: post #13
joppy wrote: post #8

I've never really understood the point of Einstein notation, as a piece of mathematical notation. Is writing something like A[i, j] * B[j, k] really that much faster than writing something like Sum[j](A[i, j] * B[j, k])? Especially when you have to check the left hand side of the equality sign just to know which indices to sum over, it seems like making things less clear for a minuscule saving on ink.

Absolutely yes. Doing tensor magic without einstein notation will make you shoot yourself, and even you don't someone else will if publish it. The only real problem I have with it personally is the abstraction of upper and lower indices, which I constantly forget the conventions as to which is which.

This is interesting... I love the notation, but mainly because how upper and lower indices make it easy to distinguish vectors and forms (not that it matters in ML where everything is "euclidean").

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#20

Post by ssivark »

TTPrograms wrote: post #17

How do you einsum convolution? Arguably the single most important linear operation in deep learning?

Convolution is a linear operator (with a certain special structure). So convolutions are represented just like any matrix-vector multiplication. Given the special structure of the convolution operator, it can be represented in a sparse manner, and what's more, the actual computation can also be implemented in an efficient manner, compared to a naive matrix-vector multiplication.

TL;DR: It can be represented easily using Einstein notation. Einstein notation just does not capture the sparsity properties we like; it represents the transformation properties quite nicely.

Post reply on HN