Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why tensors? A beginner's perspective

mfaizan.github.io

Post reply on HN
Print view View original

61–70 of 109 posts

Re: Why tensors? A beginner's perspective

#61

Post by ummonk »

mfn wrote: post #49
ummonk wrote: post #47

Earlier quoted context omitted.

The post was also a poor explanation for someone doing modern physics. [edit: not true actually I should have read the rest of the post - it’s a good post] Wald's approach in General Relativity is much better - he treats Tensors as a multilinear map from vectors and dual vectors to scalars. He then derives the underlying coordinate transformaton rules, for the vector spaces used in differential geometry. But

That’s the approach I used as well in the second half of the article - I just mentioned the transformation law in the beginning since that’s what most physics students encounter first. Most of the article tries to provide some intuition behind why multilinear maps, which sound like a fairly abstract concept, might be relevant in physics. The key link being the importance of coordinate invariance. I didn’t go into der…

Yeah sorry you’re right - I should have read the rest of your post, which is excellent and describes precisely why the coordinates/transformations focused definition is bad for one’s intuition.

Re: Why tensors? A beginner's perspective

#62

Post by ummonk »

cygx wrote: post #60
contravariant wrote: post #58

Earlier quoted context omitted.

Yes I'd also call those tensor fields. The main point I'm trying to make is that the tensor transformation law only makes sense for such fields.

The terms co- and contravariant make sense on a purely algebraic basis, with components of tensors transforming 'the same as' or 'opposite to' the basis vectors. That the basis transformation is induced by transformations of some base manifold is incidental.

Exactly. The fact that the bases are related to coordinates on the manifold is a property of differential geometry but the laws for transformation between bases are more general.

Re: Why tensors? A beginner's perspective

#63

Post by ummonk »

kaashif wrote: post #59
ericphanson wrote: post #3

I was happy to see that this article is actually talking about tensors, not just multidimensional arrays (which for some reasons are often called tensors by machine learning folks).

It's not clear to me what you're annoyed about exactly. The way I see it, there are a few options: You're getting annoyed that people are confusing the map with the territory [1]. Multidimensional arrays with certain properties can be used to represent tensors, but aren't tensors. In the same way a diagram of torus isn't a topological space, or a multiplication table isn't a group, or a matrix is not a linear map. Is…

I personally am a fan of basis free linear algebra.

More importantly though, “tensors” as commonly used in machine learning seem to rely on a single special basis, so they really are just multidimensional arrays. A machine learning algorithm isn’t really invariant under a change of basis. For example, the ReLU activation function is not independent of a change of basis.

Re: Why tensors? A beginner's perspective

#64

Post by ummonk »

uoaei wrote: post #40
icapybara wrote: post #6

Earlier quoted context omitted.

Tensors have additional properties that arrays don't necessarily have. For example, the coordinate system transform rule that the author describes in the beginning of the post. One of my old physics professors taught us to think of tensors as "arrays with units." If it's a vector/matrix/higher dimensional array but has physical units, it's probably a tensor. The fact that it has units means it represents something ph…

This is a pretty tired line, I gotta say. Obviously they are not. One is a linear operator, the other is a data structure for implementing computations using that operator. This description extends to all tensors. It's like saying "queues are not just lists". That is true and also neither insightful nor helpful. I don't see it as mystifying or complicated, what am I missing?

Sure but if you’re working with lists you shouldn’t call them queues. Similarly if you’re working with mere multidimensional arrays don’t call them tensors.

Re: Why tensors? A beginner's perspective

#65

Post by kaashif »

xyzzyz wrote: post #14

That was explanation from a perspective of someone acquainted with modern physics. As such, it will make sense to physicist, but no sense to most everyone else, including mathematicians who don’t know modern physics. For example, in the beginning, author describes tensors as things behaving according to tensor transformation formula. This is already very much a physicist kind of thinking: it assumes that there is som…

> author describes tensors as things behaving according to tensor transformation formula

Yeah, the idea that there are pre-existing things that we're trying to describe is somewhat weird to me when we're trying to come up with a definition of a tensor. The whole point of mathematics is that you come up with the definitions and theorems fall out.

In particular, this comment is funny and speaks to some difference in how I and the author view what we're doing when defining a tensor:

> But why that specific transformation law - why must tensors transform in that way in order to preserve whatever object the tensor represents?

Because we defined it like that! When you make the definition "a tensor is a thing that follows X laws", you don't get to ask why, you just defined it!

Just a funny bit of phrasing, I get what is meant :)

Re: Why tensors? A beginner's perspective

#66

Post by ummonk »

ogogmad wrote: post #11
omarhaneef wrote: post #8

Earlier quoted context omitted.

Since I also thought Tensors were just higher dimension arrays, isn't this really what ML folks think Tensors are, since they (we?) do attach units to the Tensors most of the time?

The physicist's approach is a bit non-conceptual. From a mathematical point of view, a tensor is essentially an arbitrary multi-linear map. Think of the dot product, the determinant of a matrix (which is linear on each column but is not linear on a matrix), the exterior product in exterior algebra (or geometric algebra), a linear map itself (which is obviously a special case of a multilinear map), etc. The coordinate…

Not all physicists make that conflation - Wald’s book in General Relativity emphasizes tensors as abstract concepts, and then details how they can be expressed in terms of a basis.

The OP also emphasizes the abstract interpretation as providing more intuition than the coordinate transformation rule.

Re: Why tensors? A beginner's perspective

#67

Post by edflsafoiewq »

kaashif wrote: post #65
xyzzyz wrote: post #14

That was explanation from a perspective of someone acquainted with modern physics. As such, it will make sense to physicist, but no sense to most everyone else, including mathematicians who don’t know modern physics. For example, in the beginning, author describes tensors as things behaving according to tensor transformation formula. This is already very much a physicist kind of thinking: it assumes that there is som…

> author describes tensors as things behaving according to tensor transformation formula Yeah, the idea that there are pre-existing things that we're trying to describe is somewhat weird to me when we're trying to come up with a definition of a tensor. The whole point of mathematics is that you come up with the definitions and theorems fall out. In particular, this comment is funny and speaks to some difference in ho…

> The whole point of mathematics is that you come up with the definitions and theorems fall out.

That's just how it's presented in textbooks. It's obviously not math is actually done.

Re: Why tensors? A beginner's perspective

#68

Post by cygx »

ogogmad wrote: post #11
omarhaneef wrote: post #8

Earlier quoted context omitted.

Since I also thought Tensors were just higher dimension arrays, isn't this really what ML folks think Tensors are, since they (we?) do attach units to the Tensors most of the time?

The physicist's approach is a bit non-conceptual. From a mathematical point of view, a tensor is essentially an arbitrary multi-linear map. Think of the dot product, the determinant of a matrix (which is linear on each column but is not linear on a matrix), the exterior product in exterior algebra (or geometric algebra), a linear map itself (which is obviously a special case of a multilinear map), etc. The coordinate…

Physicists conflate a tensor with its representation in some coordinate system

Rather, mathematicians that complain about the physicist's approach just haven't advanced far enough in their studies to understand how vector bundles are associated to the frame bundle ;)

Re: Why tensors? A beginner's perspective

#69

Post by mkehrt »

A (d_0 * d_1 * ... * d_{k-1} * d_k) tensor is just a linear map from a (d_0 * d_1 * ... * d_{m-1} * d_{m+1} * ... * d_{k-1} * d_k) tensor to a (d_0 * d_1 * ... * d_{n-1} * d_{n+1} * ... * d_{k-1} * d_k) tensor, where a () tensor is a scalar, right?

(I kid, but I think this is true, right?)

Re: Why tensors? A beginner's perspective

#70

Post by ravi-delia »

edflsafoiewq wrote: post #67
kaashif wrote: post #65

Earlier quoted context omitted.

> author describes tensors as things behaving according to tensor transformation formula Yeah, the idea that there are pre-existing things that we're trying to describe is somewhat weird to me when we're trying to come up with a definition of a tensor. The whole point of mathematics is that you come up with the definitions and theorems fall out. In particular, this comment is funny and speaks to some difference in ho…

> The whole point of mathematics is that you come up with the definitions and theorems fall out. That's just how it's presented in textbooks. It's obviously not math is actually done.

It's presented that way in textbooks because it's way easier to learn it that way (or it's Stockholm syndrome, which I won't deny is possible). The motivated way would require way, way, way more background knowledge.
Post reply on HN