Bfloat16 – Hardware Numerics Definition [pdf] • Hacker News Board

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#21

Post by shaklee3 » Sat, Nov 17, 2018, 11:45 PM UTC

rbanffy wrote: post #3

bleke wrote: post #2

Sorry for not in topic, did Intel calculate bonuses on hn karma (more officially impact)? I see this bf16 multiple times and it like authors dying for Christmas bonus.

To me it looks like a clever optimization. Same range as FP32, but half the size and less precise and can be converted back and forth by truncating and concatenating zeros. Is anyone else using it?

Why is it clever to change the mantissa and exponent size? I thought the clever ones were the nervana flexpoint which seemed at least partially novel. And it's interesting Intel isn't pushing that format given nervana's asic had it.

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#22

Post by Tuna-Fish » Sun, Nov 18, 2018, 4:07 PM UTC

noobermin wrote: post #20

I'm a computational scientist. Do ML problems not deal with problems that are sensitive input precision? Is it too naive to say if they don't, does one really need ML for said problem over just plain old fitting and stats?

These are not used for data but for the computation of the internal coefficients. Said coefficients would be stored as F32:s, and the job of modifying them involves computing a lot of multiplications, none of which need to be that precise.

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#23

Post by noobermin » Sun, Nov 18, 2018, 8:51 PM UTC

Tuna-Fish wrote: post #22

noobermin wrote: post #20

I'm a computational scientist. Do ML problems not deal with problems that are sensitive input precision? Is it too naive to say if they don't, does one really need ML for said problem over just plain old fitting and stats?

These are not used for data but for the computation of the internal coefficients. Said coefficients would be stored as F32:s, and the job of modifying them involves computing a lot of multiplications, none of which need to be that precise.

Wouldn't conversion between the steps lead to round off?

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#24

Post by ncmncm » Sun, Nov 18, 2018, 10:43 PM UTC

As I understand it, the shortened mantissa matters less because these numbers never appear alone. The total number of bits, of all the numbers in the set, provide enough precision to make fine distinctions between states of the network.

Even for regular calculations, it is unfortunate that the conventional split reserved only five bits of exponent. A single extra bit there would make it much more useful, and the loss to the mantissa would be an easy tradeoff.

Re: Bfloat16 – Hardware Numerics Definition [pdf]

#25

Post by Tuna-Fish » Mon, Nov 19, 2018, 11:53 AM UTC

noobermin wrote: post #23

Tuna-Fish wrote: post #22

Earlier quoted context omitted.

These are not used for data but for the computation of the internal coefficients. Said coefficients would be stored as F32:s, and the job of modifying them involves computing a lot of multiplications, none of which need to be that precise.

Wouldn't conversion between the steps lead to round off?

Conversion from Bfloat16 to f32 is just extending the mantissa with zeroes.

Basically, what they are computing is:

f32 acc = C_a0 * C_b0 + Ca1 * Cb1 + Ca2 * Cb2 ... with very many coefficients, all of which are Bfloat16. The precision of the coefficients is not that important, but they can be of substantially different magnitude, so the coefficients can use few bits in the mantissa but the accumulator needs to be wider.