Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

rockt.github.io

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#41

Post by antidesitter »

krastanov wrote: post #19
mhh__ wrote: post #13

Earlier quoted context omitted.

Absolutely yes. Doing tensor magic without einstein notation will make you shoot yourself, and even you don't someone else will if publish it. The only real problem I have with it personally is the abstraction of upper and lower indices, which I constantly forget the conventions as to which is which.

This is interesting... I love the notation, but mainly because how upper and lower indices make it easy to distinguish vectors and forms (not that it matters in ML where everything is "euclidean").

> not that it matters in ML where everything is "euclidean"

Non-euclidean spaces are actually quite common in ML, but many people don’t realize the spaces they’re working in are non-euclidean!

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#43

Post by improbable22 »

antidesitter wrote: post #41
krastanov wrote: post #19

Earlier quoted context omitted.

This is interesting... I love the notation, but mainly because how upper and lower indices make it easy to distinguish vectors and forms (not that it matters in ML where everything is "euclidean").

> not that it matters in ML where everything is "euclidean" Non-euclidean spaces are actually quite common in ML, but many people don’t realize the spaces they’re working in are non-euclidean!

Would be curious to hear what you have in mind here -- could you expand?

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#44

Post by mmmmpancakes »

ssivark wrote: post #14
joppy wrote: post #8

I've never really understood the point of Einstein notation, as a piece of mathematical notation. Is writing something like A[i, j] * B[j, k] really that much faster than writing something like Sum[j](A[i, j] * B[j, k])? Especially when you have to check the left hand side of the equality sign just to know which indices to sum over, it seems like making things less clear for a minuscule saving on ink.

It's not just about the sum. The upper and lower indices are semantically crucial. Eg: vectors have upper indices and gradients have lower indices. It's nonsensical to add the two, so the core step in gradient descent doesn't make sense. To add the two, you need the metric (Hessian) to raise the index of the gradient. And just like that, voila, you've discovered natural gradients!

Gradients are vectors. You can certainly add them to other vectors.

I think what you mean to say is that you can not add vectors and covectors. The differential of a function is a 1-form. Pointwise, it is a covector. This data is the natural starting point from which to compute gradients.

Metrics can be used to convert covectors into vectors and forms into vector fields. The result is the thing you call the natural gradient. If you do this with the standard metric on R^n, then you get the vector field which you call the gradient.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#45

Post by joppy »

ajkjk wrote: post #39
joppy wrote: post #26

Earlier quoted context omitted.

It's not actually a new notation though, it's literally just deleting a part of the old notation and then getting the reader to fill that part back in. While this is fine for calculations, I would rather that when equations are actually presented, that one extra piece of information is explicitly given. I guess a similar thing that happens on the programming language side would be leaving out types when the compiler…

If types are omitted then there's actual missing information from the equation. If summations are omitted, there... isn't. They add nothing (at least in relativity where every sum is over the same 4 values). Anyway, index notation is more of a shorthand for symbolic tensor algebra computations than actual sums. The problem is that when dealing with 2, 3, and 4 index tensors, with co- and contra-variant indexes, it's…

I don't have a problem with using indices for anything, they are fine. Distinguishing upper and lower indices is even better. The exact complaint I have is the implicit sums. For example, a matrix A would be written with one upper and one lower index, which I'll just write as A[i, j]. In einstein (implicit sum) notation, the trace of the matrix is written Tr A = A[i, i]. Is this really better than saying Tr A = Sum[i] A[i, i]?

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#46

Post by joppy »

mcabbott wrote: post #27
joppy wrote: post #8

I've never really understood the point of Einstein notation, as a piece of mathematical notation. Is writing something like A[i, j] * B[j, k] really that much faster than writing something like Sum[j](A[i, j] * B[j, k])? Especially when you have to check the left hand side of the equality sign just to know which indices to sum over, it seems like making things less clear for a minuscule saving on ink.

If you have just one term, then leaving out the summation sign saves little. But in larger expressions, the rule applies sums per term, and then the savings in clutter can be quite large. For example, here v_i is not summed, and the term with z_b has two sums: A_ia x_a + v_i + x_a y_a ( w_i + B_ib z_b ) You should not need to check the LHS, x_a y_a should always mean the dot product of these two. At least this is the…

I still feel that the whole expression could be clarified quite easily be writing a \Sum_{a, b} at the front?

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#47

Post by joppy »

feanaro wrote: post #37
joppy wrote: post #8

I've never really understood the point of Einstein notation, as a piece of mathematical notation. Is writing something like A[i, j] * B[j, k] really that much faster than writing something like Sum[j](A[i, j] * B[j, k])? Especially when you have to check the left hand side of the equality sign just to know which indices to sum over, it seems like making things less clear for a minuscule saving on ink.

As an analogy, why do you write A⋅B instead of Sum[i](A[i] * B[i])?

This is a bit of a red herring - why would people using Einstein notation write A[i] B[i] instead of writing A⋅B?

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#48

Post by feanaro »

joppy wrote: post #47
feanaro wrote: post #37

Earlier quoted context omitted.

As an analogy, why do you write A⋅B instead of Sum[i](A[i] * B[i])?

This is a bit of a red herring - why would people using Einstein notation write A[i] B[i] instead of writing A⋅B?

They wouldn't. My point was that A⋅B is also hiding a sum across all indices, just like Einstein notation. The sum never changes and is therefore visual/cognitive noise, once you've understood it the first time. The dot product is an abstraction that hides it and makes it implied.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#49

Post by mcabbott »

joppy wrote: post #46
mcabbott wrote: post #27

Earlier quoted context omitted.

If you have just one term, then leaving out the summation sign saves little. But in larger expressions, the rule applies sums per term, and then the savings in clutter can be quite large. For example, here v_i is not summed, and the term with z_b has two sums: A_ia x_a + v_i + x_a y_a ( w_i + B_ib z_b ) You should not need to check the LHS, x_a y_a should always mean the dot product of these two. At least this is the…

I still feel that the whole expression could be clarified quite easily be writing a \Sum_{a, b} at the front?

But that would sum everything, not just the parts which need it. You'd actually have to write this:

\sum_a(A_ia x_a) + v_i + \sum_a(x_a y_a) ( w_i + \sum_b(B_ib z_b) )

My example is actually still easier without indices. (Although you lose some information, like the fact that x & y are vectors in the indexed by a,b.) Here it's clear that Ax and x⋅y each involve a sum, which never includes v:

Ax + v + (x⋅y)(w + Bz)

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#50

Post by g82918 »

mhh__ wrote: post #13
joppy wrote: post #8

I've never really understood the point of Einstein notation, as a piece of mathematical notation. Is writing something like A[i, j] * B[j, k] really that much faster than writing something like Sum[j](A[i, j] * B[j, k])? Especially when you have to check the left hand side of the equality sign just to know which indices to sum over, it seems like making things less clear for a minuscule saving on ink.

Absolutely yes. Doing tensor magic without einstein notation will make you shoot yourself, and even you don't someone else will if publish it. The only real problem I have with it personally is the abstraction of upper and lower indices, which I constantly forget the conventions as to which is which.

Co -> low, primes below.
Post reply on HN