Making floating point math highly efficient for AI hardware
1–10 of 33 posts
Re: Making floating point math highly efficient for AI hardware
#2Not sure why this isn't getting more votes, but it's a good avenue of research and the authors should be commended. That said, this approach to optimizing floating point implementations has a lot of history at Imagination Technologies, ARM and similar low-power inferencing chipsets providers. I especially like the Synopsys ASIP Design [0] tool which leverages the open-source (although not yet IEEE ratified) LISA 2.0 Architecture Design Language [1] to iterate on these design issues.
Interesting times...
[0] https://www.synopsys.com/dw/ipdir.php?ds=asip-designer [1] https://en.wikipedia.org/wiki/LISA_(Language_for_Instruction...
Re: Making floating point math highly efficient for AI hardware
#3I find it interesting that they were able to find improvements even on hardware that is presumably optimized for IEEE-754 floating point numbers.
Re: Making floating point math highly efficient for AI hardware
#4Here's the bottom line for anyone who doesn't want to read the whole article.
> Using a commercially available 28-nanometer ASIC process technology, we have profiled (8, 1, 5, 5, 7) log ELMA as 0.96x the power of int8/32 multiply-add for a standalone processing element (PE).
> Extended to 16 bits this method uses 0.59x the power and 0.68x the area of IEEE 754 half-precision FMA
In other words, interesting but not earth shattering. Great to see people working in this area though!
Re: Making floating point math highly efficient for AI hardware
#5Wow! It's kind of a wierd feeling to see some research I worked on get some traction in the real world!! The ELMA lookup problem for 32 bit could be fixed by using the posit standard, which just has "simple" adders for the section past the golomb encoded section, though you may have to worry about spending transistors on the barrel shifter.
Re: Making floating point math highly efficient for AI hardware
#6Here's the bottom line for anyone who doesn't want to read the whole article. > Using a commercially available 28-nanometer ASIC process technology, we have profiled (8, 1, 5, 5, 7) log ELMA as 0.96x the power of int8/32 multiply-add for a standalone processing element (PE). > Extended to 16 bits this method uses 0.59x the power and 0.68x the area of IEEE 754 half-precision FMA In other words, interesting but not ear…
That's for the actual number crunching but the real power cost is often in bandwidth (as discussed earlier in the op). If you can reliably use lower precision stuff for training, you get 4x the flops for a halving of the bandwidth costs due to matrix mult being O(n^2)
Re: Making floating point math highly efficient for AI hardware
#7Wow! It's kind of a wierd feeling to see some research I worked on get some traction in the real world!! The ELMA lookup problem for 32 bit could be fixed by using the posit standard, which just has "simple" adders for the section past the golomb encoded section, though you may have to worry about spending transistors on the barrel shifter.
The ELMA LUT problem is in the log -> linear approximation to perform sums in the linear domain. This avoids the issue that LNS implementations have had in the past, which is in trying to keep the sum in the log domain, requiring an even bigger LUT or piecewise approximation of the sum and difference non-linear functions.
This is independent of any kind of posit or other encoding issue (i.e. it has nothing to do with posits).
(I'm the author)
Re: Making floating point math highly efficient for AI hardware
#8Wow! It's kind of a wierd feeling to see some research I worked on get some traction in the real world!! The ELMA lookup problem for 32 bit could be fixed by using the posit standard, which just has "simple" adders for the section past the golomb encoded section, though you may have to worry about spending transistors on the barrel shifter.
The ELMA LUT problem is in the log -> linear approximation to perform sums in the linear domain. This avoids the issue that LNS implementations have had in the past, which is in trying to keep the sum in the log domain, requiring an even bigger LUT or piecewise approximation of the sum and difference non-linear functions. This is independent of any kind of posit or other encoding issue (i.e. it has nothing to do with…
Thanks for your work!! (And citing us ofc)
Do you think there might be an analytic trick that you could use for higher size ELMA numbers that yields semiaccurate results for machine learning purposes? Although to be honest I still think with a kuslich FMA and an extra operation for fused exponent add (softmax e.g.) you can cover most things you'll need 32 bits for with 8
Re: Making floating point math highly efficient for AI hardware
#9Here's the bottom line for anyone who doesn't want to read the whole article. > Using a commercially available 28-nanometer ASIC process technology, we have profiled (8, 1, 5, 5, 7) log ELMA as 0.96x the power of int8/32 multiply-add for a standalone processing element (PE). > Extended to 16 bits this method uses 0.59x the power and 0.68x the area of IEEE 754 half-precision FMA In other words, interesting but not ear…
At least 69% more multiply-add flops at the same power iso-process is nothing to sneeze at (we're largely power/heat bound at this point), and unlike normal floating point (IEEE or posit or whatever), multiplication, division/inverse and square root are more or less free power, area and latency-wise. This is not a pure LNS or pure floating point because it is a hybrid of "linear" floating point (FP being itself hybrid log/linear, but the significand is linear) and LNS log representations for the summation.
Latency is also a lot less than IEEE or posit floating point FMA (not in the paper, but the results were only at 500 MHz because the float FMA couldn't meet timing closure at 750 MHz or higher in a single cycle, and the paper had to be pretty short with a deadline, so couldn't explore the whole frontier and show 1 cycle vs 2 cycle vs N cycle pipelined implementations).
The floating point tapering trick applied on top of this can help with the primary chip power problem, which is moving bits around, so you can solve more problems with a smaller word size because your encoding matches your data distribution better. Posits are a partial but not complete answer to this problem if you are willing to spend more area/energy on the encoding/decoding (I have a short mention about a learned encoding on this matter).
A floating point implementation that is more efficient than typical integer math but in which one can still do lots of interesting work is very useful too (providing an alternative for cases where you are tempted to use a wider bit width fixed point representation for dynamic range, or a 16+ bit floating point format).
Re: Making floating point math highly efficient for AI hardware
#10A bit off-topic, but I remember some studies about 'under-powered' ASICs, ie. running with 'lower-than-required' voltage and just letting the chip fail sometimes. I guess the outcome was that you can run with 0.1x power and get 0.9x of correctness. Usually chips are designed so that they never fail and that requires using substantially more energy than is needed in the average case. If the application is probabilistic or noisy in general, additional 'computation noise' could be allowed for better energy efficiency.