Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Matrix Multiplication Ugly?

mathenchant.wordpress.com

Post reply on HN
Print view View original

91–100 of 113 posts

Re: Is Matrix Multiplication Ugly?

#91

Post by srean »

almostgotcaught wrote: post #90
srean wrote: post #89

Earlier quoted context omitted.

You are changing goal posts now. Your absolute claim was > it's pathetic to pass off someone else's insights as your own. To which my point was citations are made when there is an expectation of originality. By now Feynman's anecdotes are folklore and folks wisdom. OK let's go by your standards. Cooley Tukey's FFT algorithm was "discovered" by them in around 1965. How often do they get a citation when FFT is used, es…

> if you are an aspiring mathematician, and you do not have a sense of a good taste or mathematical beauty, you probably will probably not have a good time Lol I have a PhD from a T10 and 15 published papers. I'm pretty sure I don't need your advice on "taste" or "beauty".

Yup. PhD is a start.

My condolences though, for being in a line of work where you don't perceive beauty.

Re: Is Matrix Multiplication Ugly?

#92

Post by robot-wrangler »

btown wrote: post #88
sdenton4 wrote: post #30

Earlier quoted context omitted.

A somewhat more beautiful matmul for neural networks is given by the Monarch paper: https://arxiv.org/abs/2204.00595 Generally, low-rank and block-diagonal matrices are both great strategies for producing expressive matmuls with fewer parameters. We can view the FFT as a particularly deft example of factorizing one big matmul into a number of block-diagonal matmuls, greatly reducing the overall number of multiplicati…

https://hazyresearch.stanford.edu/blog/2023-12-11-truly-subq... is an approachable overview of the Monarch approach, for those interested! There's a lot of opportunity here. Just because matrix multiplication makes for a beautiful mathematical building block, and a very reasonable one to build high-level ML logic on, doesn't mean it needs to be computed the same way, and in the same order, that we learned in linear a…

> doesn't mean it needs to be computed the same way, and in the same order, that we learned in linear algebra courses.

I think this touches on something fundamental. As a stand-alone operation matmul is ugly because it's arbitrary. In other words.. if the goal was just to entangle values, there's a bunch of ways to do it, so why this particular way landing on ae+bg etc? You kind of need algebra/geometry to justify matmul this way, which makes it obviously useful, but now it's still ugly, exactly because you had to invoke this other stuff.

Compare that situation to algebra and geometry themselves, which in a real sense don't need each other. Or to things like logic, sets, categories, processes, numbers, knots, games, etc where you can build up piles of stuff based on it in a whole rich universe before you need to appeal to much that is "outside". And in those universes operations would be defined mostly in ways that were more like "natural" or "necessary" without anything feeling arbitrary.

Traditional matmul is beautiful in the sense of "connections across and between", where all the particulars do become necessary. For those that prefer a certain amount of abstract perfection / platonism / etc or those with a taste for foundations though, it's understandable if it's not that appealing. This is related to, but not the same as the pure vs applied split.

Re: Is Matrix Multiplication Ugly?

#93

Post by pbhjpbhj »

More like watching a weaving machine than watching a person hammer nails imo. Maybe like an old-time mill, with several machines if you think in terms of actual processing on an accelerator?

There's a wooden weaving machine at a heritage museum near me that gives me the same 'taste' in my brain as thinking about 'matrix' processing in a TPU or whatever.

Re: Is Matrix Multiplication Ugly?

#94

Post by inglor_cz »

almostgotcaught wrote: post #44
chris_wot wrote: post #38

Earlier quoted context omitted.

That’s the opinion of one person. There are many mathematicians who find beauty in the maths they are studying.

> There are many mathematicians who find beauty Lol did you think this was clever? You just literally reiterated exactly what I said. See, if you had said "there are many pianists that find beauty in math" - you know like how many mathematicians find beauty in piano concertos - then you'd have me.

Pianists don't find beauty in written maths, but mathematicians don't usually find beauty in sheet music either. It is the performance, accesible to our senses, that can convey beauty even to amateurs.

Accidentally - in the parts of maths where the concepts can be visualized, such as fractal theory, non-mathematicians seem to love what they see.

Re: Is Matrix Multiplication Ugly?

#96

Post by almostgotcaught »

srean wrote: post #91
almostgotcaught wrote: post #90

Earlier quoted context omitted.

> if you are an aspiring mathematician, and you do not have a sense of a good taste or mathematical beauty, you probably will probably not have a good time Lol I have a PhD from a T10 and 15 published papers. I'm pretty sure I don't need your advice on "taste" or "beauty".

Yup. PhD is a start. My condolences though, for being in a line of work where you don't perceive beauty.

I'm a researcher in FAANG - I'll keep in mind your condolences when I get my yearly RSU re-grant lololol.

Re: Is Matrix Multiplication Ugly?

#97

Post by almostgotcaught »

inglor_cz wrote: post #94
almostgotcaught wrote: post #44

Earlier quoted context omitted.

> There are many mathematicians who find beauty Lol did you think this was clever? You just literally reiterated exactly what I said. See, if you had said "there are many pianists that find beauty in math" - you know like how many mathematicians find beauty in piano concertos - then you'd have me.

Pianists don't find beauty in written maths, but mathematicians don't usually find beauty in sheet music either. It is the performance, accesible to our senses, that can convey beauty even to amateurs. Accidentally - in the parts of maths where the concepts can be visualized, such as fractal theory, non-mathematicians seem to love what they see.

> written maths

Absolutely no one when they're navel gazing on this topic is discussing the aesthetics of notation.

> seem to love what they see.

Nor visualizations

Re: Is Matrix Multiplication Ugly?

#98

Post by inglor_cz »

almostgotcaught wrote: post #97
inglor_cz wrote: post #94

Earlier quoted context omitted.

Pianists don't find beauty in written maths, but mathematicians don't usually find beauty in sheet music either. It is the performance, accesible to our senses, that can convey beauty even to amateurs. Accidentally - in the parts of maths where the concepts can be visualized, such as fractal theory, non-mathematicians seem to love what they see.

> written maths Absolutely no one when they're navel gazing on this topic is discussing the aesthetics of notation . > seem to love what they see. Nor visualizations

It was you who compared music to maths.

People in general perceive music as "what is being played" vs. mathematics "what is being written on a page". This is the common concept, but it is incomplete. Music has its boring parts (notation), so does maths, but the general public is prone to confuse maths as a whole with its "sheet music".

"when they're navel gazing"

Maybe they're just thinking?

Re: Is Matrix Multiplication Ugly?

#99

Post by Joker_vD »

janalsncm wrote: post #18

> But matrix multiplication, to which our civilization is now devoting so many of its marginal resources, has all the elegance of a man hammering a nail into a board. Elegance is a silly critique. Imagine instead we were spending trillions on floral bouquets, calligraphy, and porcelain tea sets. I would argue that would be a bad allocation of resources. What matters to me is whether it solves the problems we have. No…

> Imagine instead we were spending trillions on floral bouquets, calligraphy, and porcelain tea sets. I would argue that would be a bad allocation of resources.

And I would argue it wouldn't. So? It's a value call.

> What matters to me is whether it solves the problems we have.

Again, what is and is not a problem is a value call. "Lacking tools to surveil and control the population" and "having population that demands its share of economic output" arguably are problems for someone which AI probably could solve. "The planet is literally on fire" is another problem (for, arguably, much bigger number of someones) and pouring terawatts of energy into chips that, coincidentally, do AI-related matrix multiplications, won't solve that problem.

Re: Is Matrix Multiplication Ugly?

#100

Post by jamespropp »

veqq wrote: post #8
jamespropp wrote: post #3

Do you disagree with my take or think I’m missing Witt’s point? I’d be happy to hear from people who disagree with me.

> sends the pair (x, y) to the pair (−x, y) I know linear algebra, but this part seems profoundly unclear. What does "send" mean? Following with different examples in 2 by 2 notation only makes it worse. It seems like you're changing referents constantly.

I’ve updated this passage. Let me know if the new version is clearer.
Post reply on HN