For those interested the general area I saw a good talk about representing and manipulating floating point numbers in Julia at CSAIL last week by Jiahao Chen. The code with some good documentation is on his github.
Making floating point math highly efficient for AI hardware
21–30 of 33 posts
Re: Making floating point math highly efficient for AI hardware
#22Here's the bottom line for anyone who doesn't want to read the whole article. > Using a commercially available 28-nanometer ASIC process technology, we have profiled (8, 1, 5, 5, 7) log ELMA as 0.96x the power of int8/32 multiply-add for a standalone processing element (PE). > Extended to 16 bits this method uses 0.59x the power and 0.68x the area of IEEE 754 half-precision FMA In other words, interesting but not ear…
They don't show a comparison to bfloat16 PEs/FMA. IEEE half precision uses a larger mantissa than bfloat16, and the cost of multiplication is proportionate to the square of the mantissa size. I'd expect much lower gains relatively to bfloat16
Re: Making floating point math highly efficient for AI hardware
#23I find it interesting that they were able to find improvements even on hardware that is presumably optimized for IEEE-754 floating point numbers.
It is a trade-of : they find improvements by losing precision where they believe it is not useful for their use case.
Re: Making floating point math highly efficient for AI hardware
#24Earlier quoted context omitted.
Wait, 0.59x isn't Earth shattering? That's almost half the power, and at only 2/3 the area. Those are _huge_ differences at data center scale!
Only a fraction of the total power goes the actual ALU's, even on an ML chip, so the actual top line impact is probably small. Not that it's bad, just that this is a fairly complex change for the amount of power saved. Plus this requires (unproven) changes on the modeling side which isn't desirable.
Not sure I understand the "Plus [...]" part: this is new research, so obviously no one is going to implement this at scale until there's been some time for people to go over the approach and either confirm the maths is solid, or find problems with it. But that is universally true for any new low level design, I assume we all understand that "it's still in peer review" implies "so now it needs to be put to the test", not "and now we all use it without question" =)
Re: Making floating point math highly efficient for AI hardware
#25Earlier quoted context omitted.
I've thought of that, but the problem is that it needs to linearly interpolate between the more accurate values, and depending upon how finely grained the linear interpolation is, you would need a pretty big fixed point multiplier to do that interpolation accurately. If you didn't want to interpolate with an accurate slope, and just use a linear interpolation with a slope of 1 (using the approximations 2^x ~= 1+x and…
> It would be a lot harder to know what the problem is in 8 bits when everything is under question if you don't know what the outcome should be. I might have a solution for that : I work on methods to both quantify the impact of your precision on the result and locate the sections of your code that introduced the significant numerical errors (as long as your numeric representation respects the IEEE standard). However…
None of the representations considered in the paper (log or linear posit or log posit) respect the IEEE standard, deliberately so :)
Re: Making floating point math highly efficient for AI hardware
#26Earlier quoted context omitted.
Wait, 0.59x isn't Earth shattering? That's almost half the power, and at only 2/3 the area. Those are _huge_ differences at data center scale!
Only a fraction of the total power goes the actual ALU's, even on an ML chip, so the actual top line impact is probably small. Not that it's bad, just that this is a fairly complex change for the amount of power saved. Plus this requires (unproven) changes on the modeling side which isn't desirable.
The Kulisch accumulator and entropy coding of the floating point words (tapering) address this particular issue.
They allow you to get away with much smaller word sizes while preserving dynamic range (and precision!) than would otherwise be the case. This is what the "word size"/tapering discussion in the blog. This is the thing that makes 8 bit floating point work in this case with just a drop in replacement via round to nearest even. You have to change significantly more to get 8 bit FP to work without either the Kulisch accumulator or entropy coding, as you have to make much different tradeoffs between precision and dynamic range.
"Users of floating point are seldom concerned simultaneously with with loss of accuracy and with overflow" (or underflow for that matter) [1]
The paper and blog post consider 4-5 different things/techniques, not all of which need be combined and some of which can be considered completely independently. The paper is a little bit gimmicky in that I combine all of them together, but that need not be the case.
(log significand fraction map (LNS), posit/Huffman/other entropy encoding, Kulisch accumulation, ELMA hybrid log/linear multiply-add as a replacement for pure log domain)
[1] Morris, Tapered floating point: a new floating point representation (1971) https://ieeexplore.ieee.org/abstract/document/1671767
Re: Making floating point math highly efficient for AI hardware
#27Earlier quoted context omitted.
> It would be a lot harder to know what the problem is in 8 bits when everything is under question if you don't know what the outcome should be. I might have a solution for that : I work on methods to both quantify the impact of your precision on the result and locate the sections of your code that introduced the significant numerical errors (as long as your numeric representation respects the IEEE standard). However…
None of the representations considered in the paper (log or linear posit or log posit) respect the IEEE standard, deliberately so :)
You drop denormals and change the distribution but do you keep the 0,5 ULP (round to nearest) garantee from the IEEE standard ?
And are your rounding errors exact numbers in your representation (can you build Error Free Transforms) ?
Re: Making floating point math highly efficient for AI hardware
#28A bit off-topic, but I remember some studies about 'under-powered' ASICs, ie. running with 'lower-than-required' voltage and just letting the chip fail sometimes. I guess the outcome was that you can run with 0.1x power and get 0.9x of correctness. Usually chips are designed so that they never fail and that requires using substantially more energy than is needed in the average case. If the application is probabilisti…
That sounds awful for verification, debugging, reproducibility and safety-critical systems. Imagine this in a self-driving car. Scary.
Re: Making floating point math highly efficient for AI hardware
#29Earlier quoted context omitted.
None of the representations considered in the paper (log or linear posit or log posit) respect the IEEE standard, deliberately so :)
You drop denormals and change the distribution but do you keep the 0,5 ULP (round to nearest) garantee from the IEEE standard ? And are your rounding errors exact numbers in your representation (can you build Error Free Transforms) ?
For (linear) posit, what the "last place" is varies. Versus a fixed-size significand, there is no 0.5 ulp guarantee. If you are in the regime of full precision, then there is a 0.5 ulp guarantee. The rounding also becomes logarithmic rather than linear in some domains (towards 0 and +/- inf), in which case it is 0.5 ulp log scale rather than linear, when the exponent scale is not 0.
For my log numbers under ELMA (with or without posit-ing), the sum of 2 numbers alone cannot be analyzed in a simple ulp framework I think, given the hybrid log/linear nature. Two numbers summed are both approximated in the linear domain (to 0.5 ulp linear domain, assuming alpha >= frac + 1), then summed exactly, but conversion back to the log domain when done is approximate, to 0.5 ulp in the log domain. But the result is of course not necessarily 0.5 ulp in the log domain. Multiplication, division and square root are always the exact answer however (no rounding). The sum of two log numbers could of course also be done via traditional LNS summation, in which case there is Kulisch accumulation throws another wrench in the issue. Summation of many log domain numbers via ELMA will usually be way more accurate than 0.5 (log domain) ulp rounding via LNS traditional summation techniques, because the compounding of error is minimized, especially when you are summing numbers of different (or slightly different) magnitudes. Kulisch accumulation for linear numbers is of course exact, so the sum of any set of numbers rounded back to traditional floating point is accurate to 0.5 ulp.
Re: Making floating point math highly efficient for AI hardware
#30Earlier quoted context omitted.
None of the representations considered in the paper (log or linear posit or log posit) respect the IEEE standard, deliberately so :)
You drop denormals and change the distribution but do you keep the 0,5 ULP (round to nearest) garantee from the IEEE standard ? And are your rounding errors exact numbers in your representation (can you build Error Free Transforms) ?
The rounding errors for linear posit are exact numbers (excepting division), assuming higher precision. The rounding errors for LNS add/sub are not exact numbers in the representation in the general case. 2 and sqrt(2) are represented exactly, but (2 + sqrt(2)) is not.