Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tensors, the geometric tool that solved Einstein's relativity problem

quantamagazine.org

Post reply on HN
Print view View original

11–20 of 98 posts

Re: Tensors, the geometric tool that solved Einstein's relativity problem

#11

Post by mitthrowaway2 »

AnotherGoodName wrote: post #9
senderista wrote: post #2

If you have any linear algebra background, then the definition of a tensor is straightforward: given a vector space V over a field K (in physics, K = R or C ), a tensor T is a multilinear (i.e. linear in each argument) function from vectors and dual vectors in V to numbers in K . That's it! A type (p, q) tensor T takes p vectors and q dual vectors as arguments ( p+q is often called the rank of T but is ambiguous comp…

For those without a strong math background but more of a programmers background; You know the matrices you work with in 2D or 3D graphics environments that you can apply to vectors or even other matrices to more easily transform (rotate, translate, scale)? Well tensors are the generalisation of this concept. If you’ve noticed 2D games transformation matrices seem similar (although much simpler) to 3D games transforma…

Is there a difference between a 4x4 matrix and a 4x4 tensor?

Re: Tensors, the geometric tool that solved Einstein's relativity problem

#12

Post by andrewla »

max_likelihood wrote: post #3

I've always thought the use of "Tensor" in the "TensorFlow" library is a misnomer. I'm not too familiar with ML/theory, is there a deeper geometric meaning to the multi-dimensional array of numbers we are multiplying or is "MatrixFlow" a more appropriate name?

Matrices are strictly two-dimensional arrays (together with some other properties, but for a computer scientist that's it). Tensors are the generalization to higher dimensional arrays.

Re: Tensors, the geometric tool that solved Einstein's relativity problem

#13

Post by pyinstallwoes »

AnotherGoodName wrote: post #9
senderista wrote: post #2

If you have any linear algebra background, then the definition of a tensor is straightforward: given a vector space V over a field K (in physics, K = R or C ), a tensor T is a multilinear (i.e. linear in each argument) function from vectors and dual vectors in V to numbers in K . That's it! A type (p, q) tensor T takes p vectors and q dual vectors as arguments ( p+q is often called the rank of T but is ambiguous comp…

For those without a strong math background but more of a programmers background; You know the matrices you work with in 2D or 3D graphics environments that you can apply to vectors or even other matrices to more easily transform (rotate, translate, scale)? Well tensors are the generalisation of this concept. If you’ve noticed 2D games transformation matrices seem similar (although much simpler) to 3D games transforma…

Is there a similar "gimbal lock" problem that people ran into?

Re: Tensors, the geometric tool that solved Einstein's relativity problem

#14

Post by MathMonkeyMan »

max_likelihood wrote: post #3

I've always thought the use of "Tensor" in the "TensorFlow" library is a misnomer. I'm not too familiar with ML/theory, is there a deeper geometric meaning to the multi-dimensional array of numbers we are multiplying or is "MatrixFlow" a more appropriate name?

The joke I learned in a Physics course is "a vector is something that transforms like a vector," and "a tensor is something that transforms like a tensor." It's true, though.

The physicist's tensor is a matrix of functions of coordinates that transform in a prescribed way when the coordinates are transformed. It's a particular application of the chain rule from calculus.

I don't know why the word "tensor" is used in other contexts. Google says that the etymology of the word is:

> early 18th century: modern Latin, from Latin tendere ‘to stretch’.

So maybe the different senses of the word share the analogy of scaling matrices.

Re: Tensors, the geometric tool that solved Einstein's relativity problem

#15

Post by blt »

max_likelihood wrote: post #3

I've always thought the use of "Tensor" in the "TensorFlow" library is a misnomer. I'm not too familiar with ML/theory, is there a deeper geometric meaning to the multi-dimensional array of numbers we are multiplying or is "MatrixFlow" a more appropriate name?

There is no geometric meaning. It's a really bad name.

Re: Tensors, the geometric tool that solved Einstein's relativity problem

#16

Post by dannymi »

max_likelihood wrote: post #3

I've always thought the use of "Tensor" in the "TensorFlow" library is a misnomer. I'm not too familiar with ML/theory, is there a deeper geometric meaning to the multi-dimensional array of numbers we are multiplying or is "MatrixFlow" a more appropriate name?

In the first example on https://www.tensorflow.org/api_docs/python/tf/math/multiply you can see that they use the Hadamard product (not the matrix product):

    x = tf.constant(([1, 2, 3, 4]))
    tf.math.multiply(x, x)
I could stop right here since it's a counterexample to x being a matrix (with a matrix product defined on it; P.S. try tf.matmul(x, x)--it will fail; there's no .transpose either). But that's only technically correct :)

So let's look at tensorflow some more:

The tensorflow tensors should transform like vectors would under change of coordinate system.

In order to see that, let's do a change of coordinate system. To summarize the stuff below: If L1 and W12 are indeed tensors, it should be true that A L1 W12 A^-1 = L1 W12.

Try it (in tensorflow) and see whether the new tensor obeys the tensor laws after the transformation. Interpret the changes to the nodes as covariant and the changes to the weights as contravariant:

    import tensorflow as tf
    # Initial outputs of one layer of nodes in your neural network
    L1 = tf.constant([2.5, 4, 1.2], dtype=tf.float32)
    # Our evil transformation matrix (coordinate system change)
    A = tf.constant([[2, 0, 0], [0, 1, 0], [0, 0, 0.2]], dtype=tf.float32)
    # Weights (no particular values; "random")
    W12 = tf.constant(
        [[-1, 0.4, 1.5],
         [0.8, 0.5, 0.75],
         [0.2, -0.3, 1]], dtype=tf.float32
    )
    # Covariant tensor nature; varying with the nodes
    L1_covariant = tf.matmul(A, tf.reshape(L1, [3, 1]))
    A_inverse = tf.linalg.inv(A)
    # Contravariant tensor nature; varying against the nodes
    W12_contravariant = tf.matmul(W12, A_inverse)
    # Now derive the inputs for the next layer using the transformed node outputs and weights
    L2 = tf.matmul(W12_contravariant, L1_covariant)
    # Compare to the direct way
    L2s = tf.matmul(W12, tf.reshape(L1, [3, 1]))
    #assert L2 == L2s
A tensor (like a vector) is actually a very low-level object from the standpoint of linear algebra. It's not hard at all to make something a tensor. Think of it like geometric "assembly language".

In comparison, a matrix is rank 2 (and not all matrices represent tensors). That's it. No rank 3, rank 4, rank 1 (!!). So what does a matrix help you, really?

If you mean that the operations in tensorflow (and numpy before it) aren't beautiful or natural, I agree. It still works, though. If you want to stick to ascii and have no indices on names, you can't do much better (otherwise, use Cadabra[1]--which is great). For example, it was really difficult to write the stuff above without using indices and it's really not beautiful this way :(

More detail on https://medium.com/@quantumsteinke/whats-the-difference-betw...

See also http://singhal.info/ieee2001.pdf for a primer on information science, including its references, for vector spaces with an inner product that are usually used in ML. The latter are definitely geometry.

[1] https://cadabra.science/ (also in mogan or texmacs) - Einstein field equations also work there and are beautiful

Re: Tensors, the geometric tool that solved Einstein's relativity problem

#17

Post by cshimmin »

will-burner wrote: post #6
senderista wrote: post #2

If you have any linear algebra background, then the definition of a tensor is straightforward: given a vector space V over a field K (in physics, K = R or C ), a tensor T is a multilinear (i.e. linear in each argument) function from vectors and dual vectors in V to numbers in K . That's it! A type (p, q) tensor T takes p vectors and q dual vectors as arguments ( p+q is often called the rank of T but is ambiguous comp…

The definition may be simple, but it's not very concrete and I'd argue that makes it not strait forward. While examples of vector spaces can be very concrete (think R, R^2, R^30), I struggle to think of a concrete example of a multilinear function from vectors and dual vectors in V to numbers in K. On top of that when working with tensors, you don't usually use the definition os a multilinear function at least as far…

A simple example of a multilinear function is the inner (a.k.a dot) product : it takes a vector (b), and a dual vector (a^T), and returns a number. In tensor notation it's typically written δ_ij.

It's multilinear because it's linear in each of its arguments separately: = c and = c.

Another simple but less obvious example is a rotation (orthogonal) matrix. It takes a vector as an input, and returns a vector. But a vector itself can be thought of as a linear function that takes a dual vector and returns a number (via the inner product, above!). So, applying the rotation matrix to a vector is a sort of "currying" on the multilinear map, while the matrix alone can be considered a function that takes a vector and a dual vector, and returns a number.

In functional notation, you can consider your rotation matrix to be a function (V x V*) -> K, which can in turn be considered a function V -> (V* -> K), where V* is the dual space of V.

Re: Tensors, the geometric tool that solved Einstein's relativity problem

#18

Post by AnotherGoodName »

mitthrowaway2 wrote: post #11
AnotherGoodName wrote: post #9

Earlier quoted context omitted.

For those without a strong math background but more of a programmers background; You know the matrices you work with in 2D or 3D graphics environments that you can apply to vectors or even other matrices to more easily transform (rotate, translate, scale)? Well tensors are the generalisation of this concept. If you’ve noticed 2D games transformation matrices seem similar (although much simpler) to 3D games transforma…

Is there a difference between a 4x4 matrix and a 4x4 tensor?

A matrix is a subset of a tensor and the 4x4 matrix is absolutely a tensor. Tensors can be way more complex and do more but the 4x4 matrix you use in 3D operations is a great starting point.

Re: Tensors, the geometric tool that solved Einstein's relativity problem

#19

Post by jesuslop »

AnotherGoodName wrote: post #9
senderista wrote: post #2

If you have any linear algebra background, then the definition of a tensor is straightforward: given a vector space V over a field K (in physics, K = R or C ), a tensor T is a multilinear (i.e. linear in each argument) function from vectors and dual vectors in V to numbers in K . That's it! A type (p, q) tensor T takes p vectors and q dual vectors as arguments ( p+q is often called the rank of T but is ambiguous comp…

For those without a strong math background but more of a programmers background; You know the matrices you work with in 2D or 3D graphics environments that you can apply to vectors or even other matrices to more easily transform (rotate, translate, scale)? Well tensors are the generalisation of this concept. If you’ve noticed 2D games transformation matrices seem similar (although much simpler) to 3D games transforma…

To add a bit, kudos to root-parent boil-down. Programmers have already the good representation, and call it n-dimensional array, that being a list of lists of lists ... (repeat n times) ... of lists of numbers. The only nuisance is that what programmers call dimension, math people call it rank. It is the sizes of those nested lists what math people call dimensions. It's all set up for a comedy of errors. Also in math the rank is split to make explicit how much of the rank-many arguments are vectors and how many are dual vectors. You'd say something like this is a rank 7 tensor, 3 times covariant (3 vector arguments) and 4 times contravariant (4 dual vector arguments) summing 7 total arguments. I'm assuming a fixed base, so root-parent map determines a number array.

Re: Tensors, the geometric tool that solved Einstein's relativity problem

#20

Post by SkyBelow »

andrewla wrote: post #10
AnotherGoodName wrote: post #9

Earlier quoted context omitted.

For those without a strong math background but more of a programmers background; You know the matrices you work with in 2D or 3D graphics environments that you can apply to vectors or even other matrices to more easily transform (rotate, translate, scale)? Well tensors are the generalisation of this concept. If you’ve noticed 2D games transformation matrices seem similar (although much simpler) to 3D games transforma…

Slight disagree here -- matrices are enough for transformations in 2, 3, 4, and 100 dimensions. Tensors are not arrays with more rows and columns; they are higher dimensional objects -- more indices, not greater range of indices.

Is a tensor higher dimension, or is it the generalized form of the structure encompassing all of it (individual numbers (0 dimensions), vectors (1 dimension), matrixes (2 dimensions), and so on)? Kind of like how an n-sphere describes circles and spheres for n equal to 2 or 3.
Post reply on HN