61–70 of 138 posts
Numerical details: https://software.intel.com/sites/default/files/managed/40/8b...
Support for bfloat16 is already present in MKL-DNN (https://github.com/intel/mkl-dnn)
Disclaimer: I work for Intel
They're not really "grafting" it, they're implementing it as a first-class data type.
Something about Google being able to influence features in consumer grade CPUs rubs me the wrong way.
Earlier quoted context omitted.
Taking my 7980xe as an example: When it runs non-avx512 loads, I currently have it set to run at 4.1 GHz (all-core). When running avx-512 heavy loads, it instead runs at 3.6 GHz -- and tends to get much hotter (70-80C instead of 50-60C). 3.6 GHz is a mild overclock; Silicon Lottery reports 100% can achieve that speed for avx512 loads.[1] Running programs doing the same thing (eg, Hamiltonian Monte Carlo where the lik…
Yeah, people tend to completely exaggerate the impact of throttling from AVX512. It's only an issue when you do short bursts of AVX512 and the rest is not AVX512. If you do math and your math can be done in AVX512, even with throttling it's going to be substantially faster. That it runs hotter doesn't concern me at all. Intel's claimed safe Tjunction is something like 105C. EEs tend to take the published component sp…
So is 105C just a very conservative number, compensating for the probe location, or are there process specific things which brings it down to 105C?
Earlier quoted context omitted.
That's not true, as I have implemented both, in FPGA. The INRA implementation missed key optimizations in the adder and multiplier.
The INRIA paper was indeed my reference. It seems like a huge mistake if they missed key optimizations, but I'm happy to take your word for it. Are there write-ups that go in detail about these mistakes? It's the kind of somebody-is-wrong-on-the-Internet topic that would result in flaming blog posts. :-)
I mean, I helped design the posit spec and the twos complements treatment is something not even John Gustafson understands... The key insight is that the hidden bit is -2 for negative numbers (instead of 1 as it is for positive numbers). It's kind of nonobvious and I happened upon it by accident one night while fooling around with circuit diagrams. If people really get serious about it I'm sure though that it will get rediscovered by EDA folks smarter than I.
Earlier quoted context omitted.
Yeah, people tend to completely exaggerate the impact of throttling from AVX512. It's only an issue when you do short bursts of AVX512 and the rest is not AVX512. If you do math and your math can be done in AVX512, even with throttling it's going to be substantially faster. That it runs hotter doesn't concern me at all. Intel's claimed safe Tjunction is something like 105C. EEs tend to take the published component sp…
Maximum Tjunction for an STM32F303 (just happened to have datasheet open) is 150C, as is most other ICs I've seen. So is 105C just a very conservative number, compensating for the probe location, or are there process specific things which brings it down to 105C?
I wonder why they chose it over facebook’s 8 bit posit: https://code.fb.com/ai-research/floating-point-math/
bfloat16 sounds like it could be supported with minimal changes to existing floating point units, maybe with just some improved microcode. FB's approach on the other side requires entirely redesigned and separate execution units. That's harder to justify, that silicon will remain dark for non-DL usage.
I imagine it still keeps most of the performance benefits since it's eliminating around 2/3rds of the longest binary component.
Does bfloat16 have any other uses than deep learning?
For weights during training, 7 bits of mantissa also seems a bit low - it's common for weights to adjust much less than 1% during a single batch of training, which this couldn't represent.
I think this is more a "we want something which is faster but is compatible with existing code written for fp32's".