Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Matrix Multiplication with Half the Multiplications

github.com

Post reply on HN
Print view View original

81–85 of 85 posts

Re: Show HN: Matrix Multiplication with Half the Multiplications

#81

Post by SJC_Hacker »

bee_rider wrote: post #41
OJFord wrote: post #20

Earlier quoted context omitted.

LLM hype and this submission in particular keep making me think of a lecturer I had for Topics in Large Dimensional Data Processing , circa 2016: as I recall he was enthusiastically adamant that the most important thing, breakthroughs etc., in years/decades to come was going to be faster matrix operations. Anyway, I'm pretty sure I recognise FIP (not FFIP of course) from that course. I wish I could remember his name,…

Maybe I’m joking, but: our society is just a vehicle for economics at this point, our economy is built around science, our science has mostly been turned into observations about engineering, some time ago we changed all of engineering into differential equations, and differential equations can be solved by discretizing them and doing linear algebra, and most of linear algebra can be done with matrix multiplications (…

For life sciences need grad students / postdocs to do the grunt work of pipetting, dissected, plating etc. And whatever the equivalent is in chemistry (titration/GC/mass transfer I guess)?

But those tools created by engineers are pretty darn important, and allow plenty of experiments/observations to be performed that were previously out of reach.

Re: Show HN: Matrix Multiplication with Half the Multiplications

#82

Post by SJC_Hacker »

p1esk wrote: post #60
emacs28 wrote: post #50

Earlier quoted context omitted.

For everyone discussing the reduced accuracy/numerical stability of the algorithms in floating-point, this is true. But note that the application of the algorithms in the work is explored for fixed-point MM/quantized integer NN inference, not floating-point MM/inference. Hence, there is no reduction in accuracy for that application of it compared to using conventional fixed-point MM.

Inference world is gradually switching from INT formats to FP formats. FP8 is already supported in modern hardware, and FP4 support is coming. In my experiments I get better perplexity in language models with FP4 than with INT4.

How is FP fundamentally different than integers? I've done FPGA programming and it just seems like the programmer has to decide where/when to do the shifting based on the expected range of the data. I'm not sure how this is "natively supported" in hardware.

Re: Show HN: Matrix Multiplication with Half the Multiplications

#83

Post by SJC_Hacker »

pclmulqdq wrote: post #56
abetusk wrote: post #52

Earlier quoted context omitted.

I'm no expert but I suspect this is wrong. To me, this is like saying you don't need to worry about integer overflow because your operations are only working on fixed integers. Really? You don't care if you multiply or add two large numbers and they spill over? The more appropriate answer, I suspect, is that the numerical precision and stability sacrifices are more than adequate for normal usage. If I'm wrong about t…

In hardware, you control your integer widths completely, so if you add two 32-bit ints to a 33-bit int, there is no chance of overflow. The same goes for multiplications, etc.

Yeah with shifts you can guarantee no overflow, but you have to decide under what circumstances is avoiding overflow/clipping worth the loss of precision.

Fixed point typically requires alot more programming, but sometimes its worth it if you know what the data ranges are.

Re: Show HN: Matrix Multiplication with Half the Multiplications

#84

Post by p1esk »

SJC_Hacker wrote: post #82
p1esk wrote: post #60

Earlier quoted context omitted.

Inference world is gradually switching from INT formats to FP formats. FP8 is already supported in modern hardware, and FP4 support is coming. In my experiments I get better perplexity in language models with FP4 than with INT4.

How is FP fundamentally different than integers? I've done FPGA programming and it just seems like the programmer has to decide where/when to do the shifting based on the expected range of the data. I'm not sure how this is "natively supported" in hardware.

If you have designed FPUs you should know that FP computation involves a lot more additional operations than just shifting (e.g. rounding, subnormals, and special value handling). That’s why, for example, CPUs use different hardware blocks for INT vs FP computation.

But that’s not the point. The point is, this particular method to speed up matmul is not suitable for FP.

Re: Show HN: Matrix Multiplication with Half the Multiplications

#85

Post by emacs28 »

adastra22 wrote: post #38
halflings wrote: post #2

This looks pretty cool! What's the catch? e.g. why isn't this already implemented in accelerators, is it really just a forgotten algorithm, or this has some implications on the cost of building the accelerator or else?

I’ve only glanced at it so someone correct me if I’m wrong, but IIUC this is not a replacement for matrix multiplication but rather an approximation that only gives decent-ish results for the types of linear systems you see in AI/ML. But for that use case it is totally fine?

It produces identical/bit-equivalent results as conventional/naive matrix multiplication for integer/fixed-point data types
Post reply on HN