Intel Prepares to Graft Google’s Bfloat16 onto Processors • Hacker News Board

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#51

Post by _Nat_ » Sun, Jul 21, 2019, 4:17 AM UTC

liquidify wrote: post #33

Could we get a Bfloat32?

Hard to infer humor here sometimes -- were you kidding?

Just in case, since I can see someone else being serious about this, I think the gist is that neural-networks tend to be fairly approximate things such that we're not particularly concerned with having a lot of precision in many cases. This use-case wouldn't seem to demand a 32-bit variant too often.

But.. if you want it anyway...

Higher-bit extensions would seem to be floating-point values that favor range-over-precision more than typical floating-point numerics with the same bit-count.

If we take that to an extreme, we can talk about ranges over infinities and infinitesimals -- this is, much like the hyperreal-number system [1].

And ya know what's funny?

Some guy's been pushing for such a primitive numeric data type [2] since the early-2000's [3]!

[1]: https://en.wikipedia.org/wiki/Hyperreal_number

[2]: http://wwwinfo.deis.unical.it/yaro/EMSS_Sergeyev.pdf

[3]: https://patents.google.com/patent/US7860914

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#52

Post by sandGorgon » Sun, Jul 21, 2019, 4:25 AM UTC

does anyone know what is the story on the software side - CUDA is basically industry standard now.

Tensorflow OpenCL support bug [1] has been open for FOUR years now (with the discussion devolving into an Intel PlaidML flame war).

AMD OpenCL is now ROCm ?

At the end of the day, I cant run ANY accelerated workloads using Intel graphics or AMD ....because there's simply no software support anywhere.

OTOH, if you have a nVidia stack... boom. you get accelerated python https://developer.nvidia.com/how-to-cuda-python

Are you running containerized workloads on kubernetes ? it has BAKED-IN support for nvidia-docker (https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus...)

Is anything going to change ?

[1] https://github.com/tensorflow/tensorflow/issues/22

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#53

Post by blipblap » Sun, Jul 21, 2019, 4:36 AM UTC

Is there a principled way to not overfit? Or is the best we can do not overtraining or reducing the precision of the calculations?

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#54

Post by zamadatix » Sun, Jul 21, 2019, 4:39 AM UTC

sandGorgon wrote: post #52

does anyone know what is the story on the software side - CUDA is basically industry standard now. Tensorflow OpenCL support bug [1] has been open for FOUR years now (with the discussion devolving into an Intel PlaidML flame war). AMD OpenCL is now ROCm ? At the end of the day, I cant run ANY accelerated workloads using Intel graphics or AMD ....because there's simply no software support anywhere. OTOH, if you have a…

Does AMD/Intel have any hardware worth switching for or is it just an ideological question?

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#55

Post by kristianp » Sun, Jul 21, 2019, 5:01 AM UTC

exsf0859 wrote: post #50

m0zg wrote: post #46

Why "graft"? It's not something that's foreign to them. This promises to essentially double the performance of Intel chips on an increasingly important workload, and also simplify the modeling work, because the models don't experience any accuracy drop when simply converted to bfloat16, unlike with quantization, where it's model dependent and finicky AF. I'd much rather do fp16 or bfloat16 at inference time, without…

I read this article as saying, "hey we can emulate bfloat16 pretty well in software on top of our existing hardware features". That's what "graft" and "minimal impact" mean to me. Intel (for better or worse) takes a very experiment-results-driven approach to choosing which features to implement in hardware. So this result -- that software emulation of a feature works almost as well as a hardware implementation would…

The article links to another saying that bfloat16 is coming to Xeons. Also the title of the current says it too. There's nothing implying less likely if you read the articles.

https://www.nextplatform.com/2018/12/16/intel-unfolds-roadma...

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#56

Post by shaklee3 » Sun, Jul 21, 2019, 5:06 AM UTC

choppaface wrote: post #29

I met Naveen Rao after Intel bought Nervana. He seemed pretty adamant about getting stuff shipped fast. In contrast, the Xeon folks own all the politics and seem to want the transition to be very gradual. Plus the Phi folks get phased out. They had done a Nervana trial at Facebook but then flaked on other trials. Clearly Intel is trying to desperately manage their books. Having Nervana and friends on a Xeon chip coul…

They had an assembler where one person wrote kernels that were faster than cublas in a lot of cases. Afaik, nobody ever released anything else with that assembler, and Nvidia caught up to that performance quickly. In talking with the cublas devs, it seemed more that maxas kernels were highly tuned for specific sizes, whereas cublas/cudnn had to be more general.

Nowadays it's really a moot point with Nvidia's Cutlass being open source.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#57

Post by ndesaulniers » Sun, Jul 21, 2019, 5:07 AM UTC

sandGorgon wrote: post #52

does anyone know what is the story on the software side - CUDA is basically industry standard now. Tensorflow OpenCL support bug [1] has been open for FOUR years now (with the discussion devolving into an Intel PlaidML flame war). AMD OpenCL is now ROCm ? At the end of the day, I cant run ANY accelerated workloads using Intel graphics or AMD ....because there's simply no software support anywhere. OTOH, if you have a…

Is Tensorflow+OpenCL a feature that a lot of folks are demanding? shrugs

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#58

Post by ndesaulniers » Sun, Jul 21, 2019, 5:10 AM UTC

liquidify wrote: post #33

Could we get a Bfloat32?

You already have that; it's more formally called IEEE 754 single precision. Requires double the memory bandwidth (bad) for extra precision that back propagation would have corrected for anyways.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#59

Post by krick » Sun, Jul 21, 2019, 5:17 AM UTC

That's surprising, I wouldn't guess it is something they would do so easily, especially considering Intel processors isn't something you generally use to train NNs.

But I'd be rather glad if they implemented unums already.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#60

Post by sdenton4 » Sun, Jul 21, 2019, 5:18 AM UTC

ndesaulniers wrote: post #57

sandGorgon wrote: post #52

does anyone know what is the story on the software side - CUDA is basically industry standard now. Tensorflow OpenCL support bug [1] has been open for FOUR years now (with the discussion devolving into an Intel PlaidML flame war). AMD OpenCL is now ROCm ? At the end of the day, I cant run ANY accelerated workloads using Intel graphics or AMD ....because there's simply no software support anywhere. OTOH, if you have a…

Is Tensorflow+OpenCL a feature that a lot of folks are demanding? shrugs

Yes, in the sense that it creates actual competition, and will presumably mean that datacenter cards for ML will lose Nvidia's $5-10k markup.