Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bfloat16 – Hardware Numerics Definition [pdf]

software.intel.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#11

Post by marcyb5st »

masklinn wrote: post #10
marcyb5st wrote: post #4

Earlier quoted context omitted.

Google's TPUs use them. But it has been for a year. I don't agree with the "new" or "Intel's" in the title.

And TPU uses them because Tensorflow uses them, it's been present since the first public commit: https://github.com/tensorflow/tensorflow/blob/f41959ccb2d9d4...

Thank you! I didn't know this. I thought they introduced them shortly after announcing TPU v1 in the 2016 (or 2017, can't remember) Google I/O.

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#12

Post by scottlegrand2 »

staticfloat wrote: post #5
rbanffy wrote: post #3

Earlier quoted context omitted.

To me it looks like a clever optimization. Same range as FP32, but half the size and less precise and can be converted back and forth by truncating and concatenating zeros. Is anyone else using it?

Google uses it on their TPUs [0]. If you're interested in how it would effect the numerical stability of an algorithm you want to use, there is a Julia package that makes prototyping linear algebra over this datatype pretty straightforward [1]. [0] https://cloud.google.com/tpu/docs/system-architecture [1] https://github.com/JuliaComputing/BFloat16s.jl

And Facebook is taking this even further. And while all these things are very cool, do not let ASIC designers claim they are barriers to entry for GPUs and CPUs. Whatever variants of this precision potpourri catch on are but a generation away from incarnation in general processors IMO...

https://code.fb.com/ai-research/floating-point-math/

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#13

Post by Tuna-Fish »

jokoon wrote: post #6

7 bit mantissa doesn't sound like a lot, 2^-7 is 0.0078125... shouldn't there be at least 9 bits for the mantissa?

With the bonus bit, that's 2.4 significant digits of precision. Plenty enough for ML, especially if you use a F32 as the accumulator.

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#14

Post by espadrine »

masklinn wrote: post #9
jokoon wrote: post #6

7 bit mantissa doesn't sound like a lot, 2^-7 is 0.0078125... shouldn't there be at least 9 bits for the mantissa?

Note that 7 bits stored means 8 bit mantissa. And bfloat16 is new but not that new: Tensorflow had it 3 years back[0]. Apparently range is more useful than precision for machine learning, which would be why they went 8/8 instead of IEEE's 11/5 FP16. [0] https://github.com/tensorflow/tensorflow/blob/f41959ccb2d9d4...

My understanding is that the optimization steps of neural networks (typically gradient descent + backpropagation) act a bit like Lloyd's algorithm. Neuron weights will push each other into place. Often, what matters is how they compare to each other, such that the error is minimized; reaching the theoretically-perfect weight is less important.

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#15

Post by wolfgke »

kps wrote: post #8
jokoon wrote: post #6

7 bit mantissa doesn't sound like a lot, 2^-7 is 0.0078125... shouldn't there be at least 9 bits for the mantissa?

There's an existing standardized 16-bit float with 10 bits mantissa¹, that graphics people are fond of. This one is for machine learning [ drink ]. ¹ https://en.wikipedia.org/wiki/Half-precision_floating-point_...

Have a look at section 1.1 and Figure 1-1 of the Intel Whitepaper. There this topic is discussed.

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#16

Post by twtw »

masklinn wrote: post #10
marcyb5st wrote: post #4

Earlier quoted context omitted.

Google's TPUs use them. But it has been for a year. I don't agree with the "new" or "Intel's" in the title.

And TPU uses them because Tensorflow uses them, it's been present since the first public commit: https://github.com/tensorflow/tensorflow/blob/f41959ccb2d9d4...

I would be extremely surprised if the motivation for putting bfloat16 in tensorflow was not the TPU. That first public commit was ~1.5 years before TPUv2 was announced at I/O, so it was almost certainly already in development.

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#17

Post by vrv »

twtw wrote: post #16
masklinn wrote: post #10

Earlier quoted context omitted.

And TPU uses them because Tensorflow uses them, it's been present since the first public commit: https://github.com/tensorflow/tensorflow/blob/f41959ccb2d9d4...

I would be extremely surprised if the motivation for putting bfloat16 in tensorflow was not the TPU. That first public commit was ~1.5 years before TPUv2 was announced at I/O, so it was almost certainly already in development.

bfloat16 was first in DistBelief, so it actually predates TensorFlow and TPUs (I worked on both systems). IIRC the motivation was more about minimizing parameter exchange bandwidth for large-scale CPU clusters rather than minimizing memory bandwidth within accelerators, but the idea generalized.

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#19

Post by stephencanon »

ndesaulniers wrote: post #18

I'll just leave this here. :-X Bfloat16 is much better for conversions from f32. https://github.com/tensorflow/tensorflow/blob/master/tensorf...

It’s definitely conceptually simpler, but both conversions are a single fully-pipelined operation on any CPU made in the past 5 years, and can be folded into the arithmetic operation on custom HW. In practice the cost of conversion isn’t really an issue; the win with bfloat16 is the added dynamic range.
Post reply on HN