If you squint a bit tensor multiplications are still basically just matrix multiplications. At each coordinate in the output you'll place the sum of a bunch of element-wise products. Here are a few examples, some of which you've seen before and some of which you might not have:
- Vector inner product (dot product) -- n-element vectors V and W are combined to create the single element (V_1 x W_1 + V_2 x W_2 + ...). Let's reframe this as a type of tensor contraction (multiplication). You have an (n,) tensor and another (n,) tensor (both "1D"), and you're getting rid of the first dimension in each of them, so the product is an (,) tensor ("0D"), and each element is computed by summing up n element-wise multiplications.
- Vector outer product -- we have the same vectors V and W as before, but the result is a "2D" (n,n) matrix M where M_a,b = V_a x W_b. Reframing this as a type of tensor contraction, you aren't getting rid of, summing over, or contracting any dimensions, so the product includes every dimension from both inputs -- it's (n,n). Each element is computed by summing up 1 element-wise multiplication.
- Matrix product -- Consider A an (i,j) matrix and B a (j,k) matrix. Standard matrix multiplication yields a result M (i,k). As a tensor multiplication, you're contracting (summing over) the "j" entries, so each output element is the sum of j element-wise multiplications. In particular, M_a,b = (A_a,1 x B_1,b + A_a,2 x B_2,b + ...).
- Kronecker product (block matrix product) -- Consider the same matrices A and B as before. We aren't going to contract over any indices, so the result is a "4D" object M (i,j,j,k). To compute M_a,b,c,d you find all elements from A and B with those indices fixed, pair and multiply them, and sum them together. In particular, M_a,b,c,d = A_a,b x B_c,d.
- Something higher dimensional -- Consider an input tensor A with shape (i,j,k,m,n) and another B with shape (m,n,w,y,z). As I'm sure you've noticed, even for objects as simple as vectors there are several choices of tensor multiplication (contraction) available. As with any other operation, the problem you're describing will govern which one you use (just like how you'd normally have a very clear reason for choosing an inner vs outer product in a vector problem). It's still interesting to consider what our options are though. The general rule is that sizes have to line up on both sides, so for this problem we could contract over {}, {m}, {n}, or {m,n}. The outputs for each of those products would have shapes, respectively, of (i,j,k,m,n,m,n,w,y,z) [everything], (i,j,k,n,n,w,y,z) [no m], (i,j,k,m,m,w,y,z) [no n], or (i,j,k,w,y,z) [no m or n]. I'm going to skip over the first three because they're enough like things you've seen before (kronecker product for the first, matrix product for the other two) that I don't think they're worth the comment space, but the last one is novel in some sense. Our methodology for multiplication is still straightforward though. Suppose the result is called M and we want to compute its value at some coordinate M_a,b,c,d,e,f. We'll pair up elements that look like A_a,b,c,?,? with elements that look like B_?,?,d,e,f (note that the inner, contracted coordinates need to match), multiply them together, and add them up. In particular, we get something like (A_a,b,c,1,1 x B_1,1,d,e,f + A_a,b,c,1,2 x B_1,2,d,e,f + A_a,b,c,2,1 x B_2,1,d,e,f + A_a,b,c,2,2 x B_2,2,d,e,f + ...). There are m x n element-wise products being added up to achieve _each_ coordinate of the output.