Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Gentle Introduction to Tensors (2014) [pdf]

ese.wustl.edu

Post reply on HN
Print view View original

41–50 of 54 posts

Re: A Gentle Introduction to Tensors (2014) [pdf]

#41

Post by soVeryTired »

joppy wrote: post #40
soVeryTired wrote: post #34

I think one of the issues that makes tensors difficult to understand is that different disciplines use the word in different ways. I'm going to generalize here a bit about how each group uses the term, but I suspect not everyone will agree with me. Apologies in advance! For machine-learning types, a tensor is a multidimensaional array of numbers that admits certain operations (addition, subtraction, hadamard product,…

I think that tensors have a much more broad meaning to mathematicians. At least in pure mathematics and algebra, there is much more of a focus on the tensor product operation, a way of taking two (or more) vector spaces and producing a new vector space: their tensor product. Tensors are elements of this new space. In particular they don’t need to be multilinear functions on a product of vector spaces (for finite dime…

That's a fair point about the tensor product being more central in algebra. But I think the 'friendliest' introduction to tensors is the multilinear function viewpoint. IMO it motivates the tensor product.

I'm not familiar with the infinite-dimensional case. Do you have an example of a tensor that isn't a multilinear function on a product of vector spaces? I'd be interested to refine my understanding here.

Re: A Gentle Introduction to Tensors (2014) [pdf]

#42

Post by joppy »

soVeryTired wrote: post #41
joppy wrote: post #40

Earlier quoted context omitted.

I think that tensors have a much more broad meaning to mathematicians. At least in pure mathematics and algebra, there is much more of a focus on the tensor product operation, a way of taking two (or more) vector spaces and producing a new vector space: their tensor product. Tensors are elements of this new space. In particular they don’t need to be multilinear functions on a product of vector spaces (for finite dime…

That's a fair point about the tensor product being more central in algebra. But I think the 'friendliest' introduction to tensors is the multilinear function viewpoint. IMO it motivates the tensor product. I'm not familiar with the infinite-dimensional case. Do you have an example of a tensor that isn't a multilinear function on a product of vector spaces? I'd be interested to refine my understanding here.

The easiest way to see it is cardinality: the space of linear functions on a countably-infinite dimensional vector space is uncountably-infinite dimensional. This is the reason why you can’t necessarily swap out vectors with linear functions on an infinite-dimensional vector space (if you restrict to functions with finite support or something, it’s ok).

A familiar example of a tensor product of countably infinite dimensional vector spaces would be polynomials in multiple variables. Say F[x] is the vector space of polynomials in x with coefficients in the field F, and F[x,y] is the space of polynomials in the variables x, y. For example x^2 - x is an element of F[x], and yx^2 - y^2 is an element of F[x,y]. Then it’s not hard to see that as vector spaces, F[x,y] is isomorphic to (F[x] tensor F[y]). A tensor in this new space is precisely a polynomial in two variables.

In the above it’s important to note that a polynomial has finitely many terms, so a power series like 1 + x + x^2 + … is not a polynomial. The space of power series is isomorphic to the space of linear functions on F[x], and is uncountably-infinite dimensional.

Re: A Gentle Introduction to Tensors (2014) [pdf]

#43

Post by abdullahkhalids »

JadeNB wrote: post #39
abdullahkhalids wrote: post #29

Earlier quoted context omitted.

I taught a module on tensors in my undergrad Mathematical Methods for Physics course last semester. The students had a weak mathematical background, so I had to explain better than any of the books and videos do. The main problem with most introductions to the topic are that they deal with coordinate systems where the basis vectors are orthogonal, so the covariant and contravariant components are the same. You need t…

> The main problem with most introductions to the topic are that they deal with coordinate systems where the basis vectors are orthogonal, so the covariant and contravariant components are the same. You need to deal with non-orthogonal basis vectors, because then you realize that naturally there are two ways of defining basis vectors at a given point in space. Or, preferably, not with basis vectors at all, but that n…

Because most physicists are ultimately interested in predicting the results of experiments with actual numbers. And that demands the use of coordinates.

Re: A Gentle Introduction to Tensors (2014) [pdf]

#44

Post by soVeryTired »

joppy wrote: post #42
soVeryTired wrote: post #41

Earlier quoted context omitted.

That's a fair point about the tensor product being more central in algebra. But I think the 'friendliest' introduction to tensors is the multilinear function viewpoint. IMO it motivates the tensor product. I'm not familiar with the infinite-dimensional case. Do you have an example of a tensor that isn't a multilinear function on a product of vector spaces? I'd be interested to refine my understanding here.

The easiest way to see it is cardinality: the space of linear functions on a countably-infinite dimensional vector space is uncountably-infinite dimensional. This is the reason why you can’t necessarily swap out vectors with linear functions on an infinite-dimensional vector space (if you restrict to functions with finite support or something, it’s ok). A familiar example of a tensor product of countably infinite dim…

> Say F[x] is the vector space of polynomials in x with coefficients in the field F, and F[x,y] is the space of polynomials in the variables x, y. For example x^2 - x is an element of F[x], and yx^2 - y^2 is an element of F[x,y]. Then it’s not hard to see that as vector spaces, F[x,y] is isomorphic to (F[x] tensor F[y]). A tensor in this new space is precisely a polynomial in two variables.

I think I'm missing part of the argument. So we have a polynomial p in F[x,y]. The claim in my previous post basically says that there's always a way to associate p with a linear map that takes two elements of the dual space of F[x] and produces a real number. I don't see why that's impossible here.

Re: A Gentle Introduction to Tensors (2014) [pdf]

#45

Post by joppy »

soVeryTired wrote: post #35
dreamcompiler wrote: post #23

Earlier quoted context omitted.

Here's where this is confusing (to me at least). The dimensionality of a vector space is the number of scalars needed to build a vector in that space. That's easy. I also think of "rank" as synonymous with "dimensionality" but it's not. Or at least it implies a different connotation of the word "dimensionality." It's the number of dimensions in the notation rather than in the space. Or something. And now I'm off the…

Take a look at my definition of the "mathematician's tensor" in the top-level comments. The rank of a tensor is the number of row- and column-vectors you need to feed the tensor to get it to return a real number. AFAIK it's not related to 'rank' in the sense of the rank-nullity theorem. To move from the mathematician's definition to the ML definition, pick a basis for your row and colum vectors. Now if you want the (…

There is in fact another definition of “tensor rank” which has everything to do with the rank of a matrix. For a tensor t in a tensor product of vector spaces VxW, define the rank of t to be the least number of summands possible in an expression t = v1xw1 + … + vnxwn.

If t is zero, then it rank is zero. If the tensor product is Vx(dual V), ie of type (n,m)=(1,1), then a tensor t can be considered as a matrix, and its tensor rank is the same thing as its matrix rank. You’re basically looking for the smallest way of writing the matrix as a sum of outer products of row and column vectors.

If you’re into quantum physics, then tensors of rank 0 or 1 are non-entangled, and tensors of rank 2 or more are entangled states.

Re: A Gentle Introduction to Tensors (2014) [pdf]

#46

Post by JadeNB »

abdullahkhalids wrote: post #43
JadeNB wrote: post #39

Earlier quoted context omitted.

> The main problem with most introductions to the topic are that they deal with coordinate systems where the basis vectors are orthogonal, so the covariant and contravariant components are the same. You need to deal with non-orthogonal basis vectors, because then you realize that naturally there are two ways of defining basis vectors at a given point in space. Or, preferably, not with basis vectors at all, but that n…

Because most physicists are ultimately interested in predicting the results of experiments with actual numbers. And that demands the use of coordinates.

> Because most physicists are ultimately interested in predicting the results of experiments with actual numbers. And that demands the use of coordinates.

Of course, at some point, you need numbers! But there's no reason that those numbers need to infest the whole computation; you can de-coordinatise as soon as possible and re-coordinatise as late as possible, and in between you not only can but must think in a coordinate-free fashion.

(I am a mathematician, not a physicist, and am not presuming to tell physicists how to do their job—this isn't an argument about whether they should do things this way. I'm just pointing out that they could, and that it seems not only possible but advantageous. But of course long-established knowledge of what actually works in physics beats this tyro's guess at what could.)

Re: A Gentle Introduction to Tensors (2014) [pdf]

#47

Post by joppy »

soVeryTired wrote: post #44
joppy wrote: post #42

Earlier quoted context omitted.

The easiest way to see it is cardinality: the space of linear functions on a countably-infinite dimensional vector space is uncountably-infinite dimensional. This is the reason why you can’t necessarily swap out vectors with linear functions on an infinite-dimensional vector space (if you restrict to functions with finite support or something, it’s ok). A familiar example of a tensor product of countably infinite dim…

> Say F[x] is the vector space of polynomials in x with coefficients in the field F, and F[x,y] is the space of polynomials in the variables x, y. For example x^2 - x is an element of F[x], and yx^2 - y^2 is an element of F[x,y]. Then it’s not hard to see that as vector spaces, F[x,y] is isomorphic to (F[x] tensor F[y]). A tensor in this new space is precisely a polynomial in two variables. I think I'm missing part o…

That’s definitely possible (the function should take two elements in F[x], not in its dual space), the problem is that there are strictly more linear functions on the set of polynomials than there are polynomials. For example you will have trouble finding a polynomial representing the “evaluate at x=1” linear function F[x] -> F, since such a polynomial would have to have infinitely many terms.

So every polynomial could be represented as a linear function on polynomials, but not every linear function on polynomials is itself a polynomial.

Re: A Gentle Introduction to Tensors (2014) [pdf]

#48

Post by soVeryTired »

JadeNB wrote: post #38
soVeryTired wrote: post #34

I think one of the issues that makes tensors difficult to understand is that different disciplines use the word in different ways. I'm going to generalize here a bit about how each group uses the term, but I suspect not everyone will agree with me. Apologies in advance! For machine-learning types, a tensor is a multidimensaional array of numbers that admits certain operations (addition, subtraction, hadamard product,…

> For mathematicians, a tensor is a function F. We pass the function an ordered tuple of N vectors and M covectors (or, to keep things simple, N column vectors and M row vectors), and the function returns a scalar. The function F needs to be linear in each of the N vectors and M covectors. In this view, a matrix is a tensor with N=1, M=1. The operations used by machine learning types arise naturally once you crank th…

> if your original vector space is infinite dimensional then the natural embedding into its double dual is not an isomorphism

Ok, so this feels like the crux of the matter. So how does the fact that V** is not isomorphic to V make the tensor product construction a more general concept than the linear function construction?

Re: A Gentle Introduction to Tensors (2014) [pdf]

#49

Post by soVeryTired »

joppy wrote: post #47
soVeryTired wrote: post #44

Earlier quoted context omitted.

> Say F[x] is the vector space of polynomials in x with coefficients in the field F, and F[x,y] is the space of polynomials in the variables x, y. For example x^2 - x is an element of F[x], and yx^2 - y^2 is an element of F[x,y]. Then it’s not hard to see that as vector spaces, F[x,y] is isomorphic to (F[x] tensor F[y]). A tensor in this new space is precisely a polynomial in two variables. I think I'm missing part o…

That’s definitely possible (the function should take two elements in F[x], not in its dual space), the problem is that there are strictly more linear functions on the set of polynomials than there are polynomials. For example you will have trouble finding a polynomial representing the “evaluate at x=1” linear function F[x] -> F, since such a polynomial would have to have infinitely many terms. So every polynomial cou…

Not trying to be stubborn here - I just don't understand.

So we're talking about the case when tensors don’t need to be multilinear functions on a product of vector spaces.

Each element of V* is (trivially) a tensor and a linear function on V. Each element of V is (trivially) a tensor and a linear function on V*. However, not all linear functions on V* are in V. So V** is bigger than V. No problem so far.

But all elements of V** are functions on (trivial) products of vector spaces, and by definition all functions in V** are linear. So how have we misunderstood each other here?

Re: A Gentle Introduction to Tensors (2014) [pdf]

#50

Post by joppy »

joppy wrote: post #47
soVeryTired wrote: post #44

Earlier quoted context omitted.

> Say F[x] is the vector space of polynomials in x with coefficients in the field F, and F[x,y] is the space of polynomials in the variables x, y. For example x^2 - x is an element of F[x], and yx^2 - y^2 is an element of F[x,y]. Then it’s not hard to see that as vector spaces, F[x,y] is isomorphic to (F[x] tensor F[y]). A tensor in this new space is precisely a polynomial in two variables. I think I'm missing part o…

That’s definitely possible (the function should take two elements in F[x], not in its dual space), the problem is that there are strictly more linear functions on the set of polynomials than there are polynomials. For example you will have trouble finding a polynomial representing the “evaluate at x=1” linear function F[x] -> F, since such a polynomial would have to have infinitely many terms. So every polynomial cou…

I guess the problem is that saying "tensor products are spaces of multilinear functions on vector spaces" is tantamount to saying "vector spaces are spaces of multilinear functions on vector spaces", which is simply not true: the second set is strictly smaller than the first. For example, there is no space of linear functions on a vector space which is countably-infinite dimensional: they are all either finite-dimensional or countably infinite dimensional. Said another way, if we're talking about 1-fold tensor products, it is not right to say "a 1-fold tensor product of V is V*", since V itself is a perfectly fine 1-fold tensor product.

In order to have V* = V for an infinite-dimensional vector space V, you need to redefine V^* to some kind of restricted dual, rather than defining it as the set of all linear functions. In the polynomial example, if we take the space of all linear maps g: F[x] -> F such that g(x^n) = g(x^(n+1)) = ... = 0 for some n >> 0, then this restricted dual is isomorphic to F[x] again. But the evaluation map g(f) = f(1) is not in this restricted dual.

There are more reasons why confusing a vector space with its dual is a bad idea. For example you cannot cook up a map V -> V* without extra knowledge, for example a choice of basis of V or something. There are many examples in abstract algebra where there is a perfectly good vector space V, and absolutely no good choice of basis for V, so trying to identify elements of V with V* is unnatural. We may still be able to speak perfectly well of vectors in V or V*, but trying to identify V with V* is still unnatural. A good example is V = (functions R -> R). I can speak easily of elements of V (for example, x + sin(x)), and of elements of V* (for example, f -> integral of xf(x)), but trying to figure out which element in the dual either of these corresponds to is hopeless. We're better off just accepting at some point that there is a real difference between a vector space and its dual.

Post reply on HN