Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why tensors? A beginner's perspective

mfaizan.github.io

Post reply on HN
Print view View original

11–20 of 109 posts

Re: Why tensors? A beginner's perspective

#11

Post by ogogmad »

omarhaneef wrote: post #8
icapybara wrote: post #6

Earlier quoted context omitted.

Tensors have additional properties that arrays don't necessarily have. For example, the coordinate system transform rule that the author describes in the beginning of the post. One of my old physics professors taught us to think of tensors as "arrays with units." If it's a vector/matrix/higher dimensional array but has physical units, it's probably a tensor. The fact that it has units means it represents something ph…

Since I also thought Tensors were just higher dimension arrays, isn't this really what ML folks think Tensors are, since they (we?) do attach units to the Tensors most of the time?

The physicist's approach is a bit non-conceptual. From a mathematical point of view, a tensor is essentially an arbitrary multi-linear map. Think of the dot product, the determinant of a matrix (which is linear on each column but is not linear on a matrix), the exterior product in exterior algebra (or geometric algebra), a linear map itself (which is obviously a special case of a multilinear map), etc.

The coordinate change stuff that physicists talk about stems from observing that a matrix can be used to represent some tensors, but the rule for changing basis changes along with the kind of tensor. So if M is a matrix which represents a linear map and P is a matrix whose columns are basis vectors, then PMP^{-1} is the same linear map as M but in basis P; if on the other hand the matrix M represents a bilinear form as opposed to a linear map, then the basis change formula is actually PMP^T, where we use the matrix transpose. Sylvester's Law Of Inertia is then a non-trivial observation about matrix representations of bilinear forms.

Physicists conflate a tensor with its representation in some coordinate system. Then they show how changing the coordinate system changes the coordinates. This point of view does provide some concrete intuition, though, so it's not all bad. By a coordinate system, I mean a linear basis.

Hope that helps.

Re: Why tensors? A beginner's perspective

#12

Post by jstx1 »

ericphanson wrote: post #3

I was happy to see that this article is actually talking about tensors, not just multidimensional arrays (which for some reasons are often called tensors by machine learning folks).

Same word, different contexts and meaning. In ML tensors are multidimensional arrays (and nothing more). Neither physicists nor ML researchers/developers are confused about what it means.

Re: Why tensors? A beginner's perspective

#14

Post by xyzzyz »

That was explanation from a perspective of someone acquainted with modern physics. As such, it will make sense to physicist, but no sense to most everyone else, including mathematicians who don’t know modern physics.

For example, in the beginning, author describes tensors as things behaving according to tensor transformation formula. This is already very much a physicist kind of thinking: it assumes that there is some object out there, and we’re trying to understand what it is in terms of how it behaves. It also uses the summation notation which is rather foreign to non-physicist mathematicians. Then, when it finally reaches the point where it is all related to tensors in TensorFlow sense, we find that there is no reference made to the transformation formula, purportedly so crucial to understanding tensors. How comes?

The solution here is quite simple: what author (and physicists) call tensors is not what TensorFlow (and mathematicians) call tensors. Instead, author describes what mathematicians call “a tensor bundle”, which is a correspondence that assigns each point of space a unique tensor. That’s where the transformation rule comes from: if we describe this mapping in terms of some coordinate system (as physicist universally do), the transformation rule tells you how to this description changes in terms of change of the coordinates. This setup, of course, has little to do with TensorFlow, because there is no space that its tensors are attached to, they are just standalone entities.

So what are the mathematician’s (and TensorFlow) tensors? They’re actually basically what the author says, after very confusing and irrelevant introduction talking about change of coordinates of underlying space — irrelevant, because TensorFlow tensors are not attached as a bundle to some space (manifold) as they are on physics, so no change of space coordinates ever happens. Roughly, tensors are a sort of universal objects representing multi linear maps: bilinear maps V x W -> R correspond canonically one-to-one to regular linear maps V (x) W -> R, where V (x) W is a vector space called tensor product of V and W, and tensors are simply vectors in this tensor product space.

Basically, the idea is to replace weird multi linear objects with normal linear objects (vectors), that we know how to deal with, using matrix multiplication and stuff. That’s all there is to it.

Re: Why tensors? A beginner's perspective

#15

Post by ajkjk »

omarhaneef wrote: post #8
icapybara wrote: post #6

Earlier quoted context omitted.

Tensors have additional properties that arrays don't necessarily have. For example, the coordinate system transform rule that the author describes in the beginning of the post. One of my old physics professors taught us to think of tensors as "arrays with units." If it's a vector/matrix/higher dimensional array but has physical units, it's probably a tensor. The fact that it has units means it represents something ph…

Since I also thought Tensors were just higher dimension arrays, isn't this really what ML folks think Tensors are, since they (we?) do attach units to the Tensors most of the time?

There are really just two things called tensors.

Physicists' tensors = generalization of arrays with units; have to transform according to certain coordinate laws.

Mathematicians' tensors = generalization of arrays, transformation rules don't matter.

Re: Why tensors? A beginner's perspective

#17

Post by cloogshicer »

jstx1 wrote: post #12
ericphanson wrote: post #3

I was happy to see that this article is actually talking about tensors, not just multidimensional arrays (which for some reasons are often called tensors by machine learning folks).

Same word, different contexts and meaning. In ML tensors are multidimensional arrays (and nothing more). Neither physicists nor ML researchers/developers are confused about what it means.

> Neither physicists nor ML researchers/developers are confused about what it means.

I'm sure this has confused a lot of people, especially beginners. Clashing terminology is one of the main difficulties in interdisciplinary work, in my experience. I don't think it's good to shrug it off like that.

Re: Why tensors? A beginner's perspective

#18

Post by kgwgk »

The_rationalist wrote: post #5
ericphanson wrote: post #3

I was happy to see that this article is actually talking about tensors, not just multidimensional arrays (which for some reasons are often called tensors by machine learning folks).

What is the difference?

Something like the difference between a “vector” of length N which is just a collection of N numbers and one that is a representation of a N-dimensional geometric algebra object.

Re: Why tensors? A beginner's perspective

#19

Post by hnarayanan »

jstx1 wrote: post #12
ericphanson wrote: post #3

I was happy to see that this article is actually talking about tensors, not just multidimensional arrays (which for some reasons are often called tensors by machine learning folks).

Same word, different contexts and meaning. In ML tensors are multidimensional arrays (and nothing more). Neither physicists nor ML researchers/developers are confused about what it means.

I was confused by this when I first got to ML.

Re: Why tensors? A beginner's perspective

#20

Post by mbbutler »

xyzzyz wrote: post #14

That was explanation from a perspective of someone acquainted with modern physics. As such, it will make sense to physicist, but no sense to most everyone else, including mathematicians who don’t know modern physics. For example, in the beginning, author describes tensors as things behaving according to tensor transformation formula. This is already very much a physicist kind of thinking: it assumes that there is som…

Why are you complaining that the author didn't talk about tensors as they are used in tensorflow? Tensorflow is never even mentioned in the piece.

The author is perfectly clear in the first sentence that the piece's focus is about the usefulness of tensors in a physics context.

Post reply on HN