Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

rockt.github.io

Post reply on HN
Print view View original

31–40 of 50 posts

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#31

Post by improbable22 »

CuriousSkeptic wrote: post #29
yorwba wrote: post #2

The problem with einsum is that you have to explicitly specify the mapping between dimensions and indices every time, without any way to enforce consistency. It would be more ergonomic if each tensor had labeled dimensions. That would prevent the kind of silly mistake where you mix up the ordering of dimensions and only notice it when you later change the shape of the tensors so the different dimensions no longer mat…

Why is it that mathematicians are so comfortable with using ordering to encode semantics? Evident from this thread it seems many developer recognize the need to use proper labels to simplify reasoning. Doesn’t mathematicians have similar desires?

Well, many different notations are convenient in different contexts. The compiler they target is a bit more flexible than developers', so they are free to just describe what they are doing on page 1.

Human languages face the same trade-off: in some word order is very important, in others less so, but they need to compensate with some kind of case labels.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#32

Post by claudius »

mcabbott wrote: post #28
claudius wrote: post #24

Earlier quoted context omitted.

ITensor [1] has somewhat pioneered this concept in tensor networks for condensed matter physics. If I understand correctly, Uni10 [2], a similar project, is even working on a graphical interface for such networks so that you can "draw" the network and have the computer figure out the best contraction order. In my own code I’ve recently also implemented such named indices (only had "einstein summation"-like contractio…

I made a thing to audit that you are consistent with your indices [1] as an alternative to explicit named-tensor objects. And found, but have not used, another package in a similar spirit [2]. Although to be honest, simply writing A[n,μ,ν,c] etc. (using different letters for different spaces) makes it pretty easy to visually check that you are getting this right. This is one of the attractions of the notation, even o…

> I made a thing to audit that you are consistent with your indices [1] as an alternative to explicit named-tensor objects. And found, but have not used, another package in a similar spirit [2].

Cool, I will have to check this out!

> Although to be honest, simply writing A[n,μ,ν,c] etc. (using different letters for different spaces) makes it pretty easy to visually check that you are getting this right. This is one of the attractions of the notation, even on paper. Unfortunately np.einsum's string notation makes this harder to see, as the indices aren't adjacent to the variable name.

The problem is not so much the letter-space association but also handling the ordering of the spaces inside the tensor. For example in my code, to do a contraction over two indices, you could do prod(a, b, "tlx,tr,p1,p2|tlx,p1,tl|tl,p2,tr") where the result would then have index order tl,p2,tr. The problem was then that changing the index order in one place (e.g. for performance reasons) meant having to re-check all other places where this is used. If you want to contract a tensor network like (d) in [1], this quickly gets complicated. With named indices, the above becomes a * b and if I change the index order in any place, it gets automatically changed there, too.

[1] https://journals.aps.org/prb/article/10.1103/PhysRevB.81.165...

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#33

Post by cdavid »

Xcelerate wrote: post #6

I remember when I was learning matrix calculus and realized at some point that it was much simpler to convert everything to index notation, perform all operations, then convert everything back to standard notation at the end. It became almost comically simple, because you're "just" working with labeled scalars at that point. To be fair, it's convenient to memorize some of the more commonly used expressions (like ∂tr(…

Interesting, I almost always got the exact opposite experience. How do you calculate non trivial matrix derivatives using index notation ?

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#34

Post by cdavid »

krastanov wrote: post #16
Xcelerate wrote: post #6

I remember when I was learning matrix calculus and realized at some point that it was much simpler to convert everything to index notation, perform all operations, then convert everything back to standard notation at the end. It became almost comically simple, because you're "just" working with labeled scalars at that point. To be fair, it's convenient to memorize some of the more commonly used expressions (like ∂tr(…

Roger Penrose (the famous mathematician) has been saying the same thing for a couple of decades (going even further than that, check out Penrose diagrams).

I am interested: do you have a reference that shows this through examples ? In my grad school years, I was always dissatisfied by matrix calculus notations

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#35

Post by mcabbott »

claudius wrote: post #32
mcabbott wrote: post #28

Earlier quoted context omitted.

I made a thing to audit that you are consistent with your indices [1] as an alternative to explicit named-tensor objects. And found, but have not used, another package in a similar spirit [2]. Although to be honest, simply writing A[n,μ,ν,c] etc. (using different letters for different spaces) makes it pretty easy to visually check that you are getting this right. This is one of the attractions of the notation, even o…

> I made a thing to audit that you are consistent with your indices [1] as an alternative to explicit named-tensor objects. And found, but have not used, another package in a similar spirit [2]. Cool, I will have to check this out! > Although to be honest, simply writing A[n,μ,ν,c] etc. (using different letters for different spaces) makes it pretty easy to visually check that you are getting this right. This is one o…

Right, changing the order would be a pain. Although if the reason for doing so is memory layout (for speed), then a lazy permutedims(A) would decouple index order from this. Perhaps when creating a tensor for the first time there ought to be a way to specify the layout? Haven't thought much but something like A[n^4, μ,ν, c^1] := .... would not be hard to do.

In your prod(a, b, ... example, if p1 and p2 are in the same space, how would a*b know which one to contract? Or do they have different names from when a was created?

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#36

Post by mcabbott »

cdavid wrote: post #33
Xcelerate wrote: post #6

I remember when I was learning matrix calculus and realized at some point that it was much simpler to convert everything to index notation, perform all operations, then convert everything back to standard notation at the end. It became almost comically simple, because you're "just" working with labeled scalars at that point. To be fair, it's convenient to memorize some of the more commonly used expressions (like ∂tr(…

Interesting, I almost always got the exact opposite experience. How do you calculate non trivial matrix derivatives using index notation ?

You insert kronecker deltas: ∂M_ab / ∂M_cd = δ_ac δ_bd , and a,b here remain contracted with whatever M was originally contracted with. Then you simplify, δ_ac Z_xya = Z_xyc and so on.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#37

Post by feanaro »

joppy wrote: post #8

I've never really understood the point of Einstein notation, as a piece of mathematical notation. Is writing something like A[i, j] * B[j, k] really that much faster than writing something like Sum[j](A[i, j] * B[j, k])? Especially when you have to check the left hand side of the equality sign just to know which indices to sum over, it seems like making things less clear for a minuscule saving on ink.

As an analogy, why do you write A⋅B instead of Sum[i](A[i] * B[i])?

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#38

Post by claudius »

mcabbott wrote: post #35
claudius wrote: post #32

Earlier quoted context omitted.

> I made a thing to audit that you are consistent with your indices [1] as an alternative to explicit named-tensor objects. And found, but have not used, another package in a similar spirit [2]. Cool, I will have to check this out! > Although to be honest, simply writing A[n,μ,ν,c] etc. (using different letters for different spaces) makes it pretty easy to visually check that you are getting this right. This is one o…

Right, changing the order would be a pain. Although if the reason for doing so is memory layout (for speed), then a lazy permutedims(A) would decouple index order from this. Perhaps when creating a tensor for the first time there ought to be a way to specify the layout? Haven't thought much but something like A[n^4, μ,ν, c^1] := .... would not be hard to do. In your prod (a, b, ... example, if p1 and p2 are in the sa…

> In your prod(a, b, ... example, if p1 and p2 are in the same space, how would a*b know which one to contract? Or do they have different names from when a was created?

ITensor introduced this concept and I mostly just followed their lead – spaces have unique names and tensor legs have a name label and a "prime level". So for example an operator O: A → A would have one leg labelled a[uuid]-prime0 and another leg a[uuid]-prime1. Similar to how one might write O: A → A has elements O_{a a’} when writing it down on paper.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#39

Post by ajkjk »

joppy wrote: post #26
pishpash wrote: post #22

Earlier quoted context omitted.

Then you're doing it wrong. The point of notation (or learning any new language) is to operate in that language, not to translate back and forth.

It's not actually a new notation though, it's literally just deleting a part of the old notation and then getting the reader to fill that part back in. While this is fine for calculations, I would rather that when equations are actually presented, that one extra piece of information is explicitly given. I guess a similar thing that happens on the programming language side would be leaving out types when the compiler…

If types are omitted then there's actual missing information from the equation. If summations are omitted, there... isn't. They add nothing (at least in relativity where every sum is over the same 4 values).

Anyway, index notation is more of a shorthand for symbolic tensor algebra computations than actual sums. The problem is that when dealing with 2, 3, and 4 index tensors, with co- and contra-variant indexes, it's quite cumbersome to notate which component of a tensor contracts with which component of another tensor, and anything you could come up with would end up looking like index notation. Rarely do you actually mean to sum anything.

Re: Einsum Is All You Need – Einstein Summation in Deep Learning (2018)

#40

Post by curiousmiro »

yorwba wrote: post #2

The problem with einsum is that you have to explicitly specify the mapping between dimensions and indices every time, without any way to enforce consistency. It would be more ergonomic if each tensor had labeled dimensions. That would prevent the kind of silly mistake where you mix up the ordering of dimensions and only notice it when you later change the shape of the tensors so the different dimensions no longer mat…

See HarvardNLP's namedtensor which is under active development!

disclaimer: i'm a contributor

Post reply on HN