Live data from Hacker News

Multiplying Matrices Without Multiplying

arxiv.org

91–100 of 124 posts

Re: Multiplying Matrices Without Multiplying

#91

Earlier quoted context omitted.

Yes, I do; your bootstrapping is dependent on a binary C++ compiler, which could be hiding something that isn't in the source code, but which propagates to newly bootstrapped compiler binaries which again pass it on to the next round of bootstrapping, ad infinitum . Basically, we can't be sure that you really have a C++ compiler. The source code can be verified to be a C++ compiler, but the binaries deviate in some w…

Sure, but you seem to have gone off on a wild tangent. How does that relate to matrix multiplication?

[deleted]

Re: Multiplying Matrices Without Multiplying

#92

Earlier quoted context omitted.

Yes, I do; your bootstrapping is dependent on a binary C++ compiler, which could be hiding something that isn't in the source code, but which propagates to newly bootstrapped compiler binaries which again pass it on to the next round of bootstrapping, ad infinitum . Basically, we can't be sure that you really have a C++ compiler. The source code can be verified to be a C++ compiler, but the binaries deviate in some w…

Sure, but you seem to have gone off on a wild tangent. How does that relate to matrix multiplication?

By "you", you mean user qsort, right?

Re: Multiplying Matrices Without Multiplying

#93

Clever and logical. It reminds me of when John Carmack used a precomputed table of Sine values for fast lookup in Quake, rather than running the actual function on the CPU.

Long before Carmack, a precomputed table of cosine values has been used in a vacuum-tube based surface-to-air missile system I have been maintaining. The table consisted of only one element: 0.7, i.e. the cosine of any angle was considered 0.7. Somehow, it was good enough for intercepting jet fighters and saved a lot of vacuum tubes.

Wasn’t this also the basic JD of “computers” of old? Manually computing lookup tables for artillery, planet movement, etc

Re: Multiplying Matrices Without Multiplying

#94
post #31

Every element of a matrix multiplication is a dot-product of two vectors. The dot-product of two vectors quantifies their similarity -- in fact, we call it "dot-product similarity" in a nearest-neighbors context: If the dot product > 0, the vectors point in similar directions; if It's not too hard to imagine that it might be possible to learn representative K-means clusters of training vectors and then, at run-time,…

I spent some time thinking about this in college, the problem you run into is the sheer size of the lookup table, you will get higher complexity in the ability to search this table (or prep your input matrix to be searchable).

I'm going to try and remember the calculations I performed, but I remember just looking at permutations/combinations of 8bit integer matrices, 2x2 seems reasonable but 3x3 or 4x4 gets unreasonable from an algorithmic/memory perspective very quickly

Re: Multiplying Matrices Without Multiplying

#96
post #78

Earlier quoted context omitted.

Most people don't find arguing from formulas intuitive unless the formulas themselves are intuitive. If you truly believe they are, I'd be curious to know why.

There is an intuitive version of this. Volume in n dimensions is C*r^n (C is some constant) and surface is the first derivative, leading to a ratio of n/r (the C constant cancels out). Hmm... Maybe not that intuitive

But the former formula already tells you that most of the volume is near the high range of r: that which was to be shown.

The surface area to volume concept adds nothing.

Because the volume of a sphere is proportional to r cubed, you know there is much more volume between r in [0.9, 1.0] than in the same sized interval of r [0.0, 0.1].

You can find the break-even point almost in your head. At what r value is half the volume of a R = 1.0 sphere below that value? Why, that's just the cube root of 1/2 ~= 0.794. So almost half the volume is within 20% of the radius from the surface.

That's far from the claim that almost all the volume is near the surface: half is not almost all, and 20% isn't all that near. However, you can see how it gets nearer and nearer for higher dimensions.

For a ten dimensional sphere, the tenth root of 1/2 is ~ 0.933. So over half the volume of a ten dimensional sphere is within the 7% depth.

Re: Multiplying Matrices Without Multiplying

#97

Earlier quoted context omitted.

Yes, I do; your bootstrapping is dependent on a binary C++ compiler, which could be hiding something that isn't in the source code, but which propagates to newly bootstrapped compiler binaries which again pass it on to the next round of bootstrapping, ad infinitum . Basically, we can't be sure that you really have a C++ compiler. The source code can be verified to be a C++ compiler, but the binaries deviate in some w…

Sure, but you seem to have gone off on a wild tangent. How does that relate to matrix multiplication?

I believe he’s getting at the idea of nested levels of corruption, with no visibility into where that corruption happens (because recompilation of the compiler/ML produces a fresh inscrutable binary).

I suppose the better term in this context is error propogation

Re: Multiplying Matrices Without Multiplying

#98
post #97

Earlier quoted context omitted.

Sure, but you seem to have gone off on a wild tangent. How does that relate to matrix multiplication?

I believe he’s getting at the idea of nested levels of corruption, with no visibility into where that corruption happens (because recompilation of the compiler/ML produces a fresh inscrutable binary). I suppose the better term in this context is error propogation

It relates in the following way. Suppose you have some framework for calculation which depends on matrix multiplications. You use that framework to develop some approximation for matrix multiplications which are faster. Then to make that framework faster, you substitute that back into the framework. That is a sort of bootstrapping which user qsort has related to compiler bootstrapping.

The goal in compiler boostrapping is to reach a fixed point, in a single iteration. When the compiler binary compiles itself, what should come out is a faithful replica of that compiler binary. Having done compiler bootstrapping, I know about hair-pulling problems that occur when you don't have a fixed point. E.g. a bug in the optimization causes the compiler to subtly miscompile itself. The resulting compiler then severely miscompiles itself; e.g. fails to recompile itself entirely, or produces something that no longer runs at all. (And that's actually a lucky case; a compiler compiling itself is basically a dog food test. When that fails, you've caught a bug early. What keeps you awake at night is the compiler compiling itself perfectly fine, but rarely miscompiling other code in the wild.)

Anyway, this sort of malfunction is exactly what will happen if you take a working, self-hosted compiler and then replace some important chunk of it with some hokey machine-learning approximation. It's unlikely you can do that without breaking the fixed point.

I don't see how you can not have fixed point issues in the matrix scenario. Maybe it will stabilize if you iterate on it. You take the machine learning framework with approximate matrix multiplications in it and run the process for figuring out those matrix multiplications. You may get some different values in it, which you can substitute back and try again. Will that converge? If it does, how do you know the whole thing is still valid? (What definition of valid applies?)

Re: Multiplying Matrices Without Multiplying

#99

Earlier quoted context omitted.

Ah that's useful to know that it was conventional up until then, thanks. It was my first exposure, personally :)

I am old enough to remember printed books in the library that contained pre-computed tables of logarithms, sines and other common functions.

the probably were computed by mechanical means, but earlier ones were computed by hand.

One of Gauss’s ‘hobbies’ was to find errors in such tables.

There even were books containing tables of products of integers. You can buy a reproduction of one today: https://www.amazon.com/Calculating-Products-Thousand-Applica...

Re: Multiplying Matrices Without Multiplying

#100
post #90

Lots of discussion here about higher dimensional matrices. The argument is that the dot product is an operation on two arguments, therefore any multidimensional matrix multiplication can be broken down into multiple two dimensional operations. The image offerred is that two vectors define a plane so any two dimensional operation is valid. But how about curved space? Suppose I have two vectorsin curved space-they look…

> But how about curved space? Suppose I have two vectorsin curved space-they look like two droopy arrows.

No, they don't. In math, curved spaces are modeled through so-called manifolds which locally look like ℝ^n. In particular, at any point p of the manifold there's a tangent space, i.e. a linear space (higher-dimensional plane) tangent to the manifold at p. Vectors at p are just vectors in that linear space. So they are "straight" not "droopy".

On the tangent space of each point p you can now define an inner product g(p). The resulting family of inner products g is called a (Riemannian) metric on the manifold[0] and describes how lengths (of vectors) and angles (between vectors) can be measured at each point.

> Currently, we define the curvature of space with gravity or the deflection of light past massive objects. Is there a way to measure the curvature of space locally?

Yes, there is. In fact, the curvature[1] of a (Riemannian) manifold is a purely local quantity – it's basically the second derivative of the metric g, so it describes how the notion of length changes (more precisely: how the change in length changes) as you go from a point p to neighboring points.

There are other ways to express what curvature is, e.g. by locally parallelly transporting[2] a vector along a closed curve (and making that curve smaller and smaller) which basically measures how the notion of straight lines changes locally. (Though, since a line being "straight" means "locally length minimizing" this brings us back to the notion of length and, thus, the metric.)

Alternatively, if the manifold has dimension 2, there's a particularly simple way of looking at and interpreting curvature, see [3].

In any case, curvature being used to model gravity is entirely separate from that idea.

[0]: Provided this inner product "varies smoothly" as you move from p to a neighboring point q.

[1]: https://en.wikipedia.org/wiki/Riemann_curvature_tensor

[2]: https://en.wikipedia.org/wiki/Parallel_transport

[3]: https://en.wikipedia.org/wiki/Gaussian_curvature

Post reply on HN