Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A basic introduction to NumPy's einsum

ajcr.net

Post reply on HN
Print view View original

41–50 of 50 posts

Re: A basic introduction to NumPy's einsum

#41

Post by pizza »

p1esk wrote: post #24
imranq wrote: post #18

I've found einsum to be amazing at consolidating my code into something more readable, particularly for implementing architectures from scratch. Here's a good video that explains why its so good: https://www.youtube.com/watch?v=pkVwUVEHmfI Also check out Lucid Rains Github, who uses it extensively to build transformer architectures from scratch: https://github.com/lucidrains \ * Example: https://github.com/lucidrains…

amazing at consolidating my code into something more readable Readable if you're already familiar with einsum notation. Otherwise there's learning curve. An alternative to einsum is using multiple dot product and reshape ops, hopefully with each one having a comment - this would be a lot more readable imo.

But then to find some mistake you would have to corroborate n * 2 pieces of information, which is quite a bit more, before getting into the issue of whether or not the sequence of operations actually makes sense on top of all that.

Re: A basic introduction to NumPy's einsum

#43

Post by mcabbott »

shaklee3 wrote: post #27
catgary wrote: post #19

Earlier quoted context omitted.

The Tullio library in Julia is a pretty fantastic option for Einstein summation. It’s performance is great, it generates CUDA kernels, and does some clever tricks for automatic differentiation. It’s also a bit more readable than numpy’s einsum function, since you just write: @tullio C[i,j] := A[i,k] * B[k,j]

it's highly unlikely that the performance will be great on GPU if it generates its own kernels. cutensor has been hand optimized to do exactly these types of operations in a way emitting a kernel automatically can't do.

On the GPU, operations which look like matrix multiplication are indeed quite slow. As you say there are many tricks, and Tullio doesn't (right now) know them. For operations which are more like broadcasting it does much better.

On the CPU, the situation is much better. With some help from LoopVectorization.jl (which optimises micro-kernels) it will often beat OpenBLAS at matrix multiplication. The best-case scenario is an operation which would otherwise be permutedims plus matrix multiplication, for which it will often be several times faster, by fusing these.

The notation above is shared by some other packages. TensorOperations.jl is always decomposes to known kernels (including on the GPU) and OMEinsum.jl usually does so (with a fallback to loops), both more like einsum. TensorCast.jl is more like einops, just notation for writing reshape/permute/slice operations.

Re: A basic introduction to NumPy's einsum

#44

Post by mjburgess »

physicsguy wrote: post #42

Oh god why did I not know about this when I did my theoretical physics PhD

I have to imagine python has revolutionised physics education at university.

I taught myself C before arriving (and, eg., esp. via stanford's programming paradigms course) -- if I hadnt, I would be resigned with my classmates to C code projected on OHP slides.

If I had my way today, the whole of any applied science would be taught with both mathematical and programming (python for convenience) notation.

Re: A basic introduction to NumPy's einsum

#45

Post by p1esk »

pizza wrote: post #41
p1esk wrote: post #24

Earlier quoted context omitted.

amazing at consolidating my code into something more readable Readable if you're already familiar with einsum notation. Otherwise there's learning curve. An alternative to einsum is using multiple dot product and reshape ops, hopefully with each one having a comment - this would be a lot more readable imo.

But then to find some mistake you would have to corroborate n * 2 pieces of information, which is quite a bit more, before getting into the issue of whether or not the sequence of operations actually makes sense on top of all that.

So let’s say there’s a bug, and an einsum statement is a suspect. Assume I’m not familiar with einsum notation, but I know dot products and reshapes. How do I debug it?

Re: A basic introduction to NumPy's einsum

#46

Post by p1esk »

hakuseki wrote: post #25
p1esk wrote: post #24

Earlier quoted context omitted.

amazing at consolidating my code into something more readable Readable if you're already familiar with einsum notation. Otherwise there's learning curve. An alternative to einsum is using multiple dot product and reshape ops, hopefully with each one having a comment - this would be a lot more readable imo.

I really don't find that einsum is less intuitive than reshape. Maybe they both take about the same time to learn the first time. And that's less time than it would take me to struggle through even a single function using reshape ops.

How many DL people are familiar with dot products and reshapes, and how many are familiar with einsum?

Re: A basic introduction to NumPy's einsum

#47

Post by pizza »

p1esk wrote: post #45
pizza wrote: post #41

Earlier quoted context omitted.

But then to find some mistake you would have to corroborate n * 2 pieces of information, which is quite a bit more, before getting into the issue of whether or not the sequence of operations actually makes sense on top of all that.

So let’s say there’s a bug, and an einsum statement is a suspect. Assume I’m not familiar with einsum notation, but I know dot products and reshapes. How do I debug it?

Guess you can’t, then. Personally I’m a bigger fan of einops since the variable/index names can be a lot more self-documenting.

Re: A basic introduction to NumPy's einsum

#48

Post by hakuseki »

p1esk wrote: post #46
hakuseki wrote: post #25

Earlier quoted context omitted.

I really don't find that einsum is less intuitive than reshape. Maybe they both take about the same time to learn the first time. And that's less time than it would take me to struggle through even a single function using reshape ops.

How many DL people are familiar with dot products and reshapes, and how many are familiar with einsum?

I really couldn't say.

It does occur to me, though, that you are probably talking about the elementwise product, not the dot product. The dot product includes a reduce_sum step and outputs a scalar.

Re: A basic introduction to NumPy's einsum

#49

Post by physicsguy »

mjburgess wrote: post #44
physicsguy wrote: post #42

Oh god why did I not know about this when I did my theoretical physics PhD

I have to imagine python has revolutionised physics education at university. I taught myself C before arriving (and, eg., esp. via stanford's programming paradigms course) -- if I hadnt, I would be resigned with my classmates to C code projected on OHP slides. If I had my way today, the whole of any applied science would be taught with both mathematical and programming (python for convenience) notation.

I used all of C + Python + NumPy a lot, just didn't know about this feature. Would have helped prototyping various bits and pieces at the Python level.

Re: A basic introduction to NumPy's einsum

#50

Post by TheRealKing »

enriquto wrote: post #7
cschmid wrote: post #2

I'd really like to use einsum more often, because it allows me to code my expressions the same way I derive them on pen and paper. Unfortunately, as mentioned in the article, it's slow, because it converts your formula to a for loop. So usually, I rewrite my formulas into messy combinations of broadcasts, transposes and array multiplications. Is there a package or an algorithm that does this conversion automatically?…

Just use C or C++ or fortran or julia or even lua. The issue of "slow loops" is entirely self-inflicted by some languages. It's getting ridiculous to still worry about this shit in 2022. Simple loops can and should be just as fast as vectorized programs. When they are slower, it is 100% due to deliberate decisions by the language dessigners

or Rust
Post reply on HN