Ohh intel is awesome
Intel Prepares to Graft Google’s Bfloat16 onto Processors
91–100 of 138 posts
Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors
#92Is there a principled way to not overfit? Or is the best we can do not overtraining or reducing the precision of the calculations?
regularization, dropout, early stopping etc.
Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors
#93> At this point, Intel doesn’t have bfloat16 implemented in any of its processors, so they used current AVX512 vector hardware present in its existing processor to emulate the format and the requisite operations. According to the researchers, this resulted in “only a very slight performance tax.” Why implement bfloat if you get just slightly less performance emulating it with AVX512, which already exists? Maybe it’s…
The reason to use float16 isn’t to make individual operations faster, its to fit more numbers in vector registers. For example, AVX512 vector registers are 512 bits. That’s 8 64-bit floats, 16 32-bit ones, or 32 16-bit ones.
=> A vector multiply using bfloat16 may not be much faster than one using float32, but it will do more multiplications.
Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors
#94Earlier quoted context omitted.
A V100 GPU gets 900GB/s of memory bandwidth. I am less of a CPU expert but afaict you'll be lucky to get much more than 10% of that out of a CPU. This is going to make a huge difference that Intel can't make up with bigger execution units. bfloat helps with this because the data is half as large. But of course if you're doing Nvidia you're probably already doing (IEEE) fp16.
That sounded a bit low. This https://en.wikichip.org/wiki/intel/microarchitectures/cooper... says "Higher bandwidth (174.84 GiB/s, up from 119.209 GiB/s)" I don't know if memory bandwidth matters for this type of job, though.
Is that intel ARK published spec sheet bandwidth, or actual usable bandwidth? There is a difference.
I've found I get about 75-80% of the advertised bandwidth both from my real app (TLS crypto) and a toy memory copy benchmark using AVX256 instructions. The toy memory copy benchmark is how I realized that my bottleneck was actually memory bandwidth and not CPU horsepower on Broadwell based servers.
Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors
#95Earlier quoted context omitted.
That sounded a bit low. This https://en.wikichip.org/wiki/intel/microarchitectures/cooper... says "Higher bandwidth (174.84 GiB/s, up from 119.209 GiB/s)" I don't know if memory bandwidth matters for this type of job, though.
Is that intel ARK published spec sheet bandwidth, or actual usable bandwidth? There is a difference. I've found I get about 75-80% of the advertised bandwidth both from my real app (TLS crypto) and a toy memory copy benchmark using AVX256 instructions. The toy memory copy benchmark is how I realized that my bottleneck was actually memory bandwidth and not CPU horsepower on Broadwell based servers.
I haven't a clue. I just got it off the link quoted. It's a good question and when there's a difference, you know what marketing will say.
To make a stab, I suppose it might depend on whether all requests are coming from a single memory bank or spread evenly across all memory banks, assuming fully populated (again from the link "Octa-channel (up from hexa-channel)")
Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors
#96Earlier quoted context omitted.
Taking my 7980xe as an example: When it runs non-avx512 loads, I currently have it set to run at 4.1 GHz (all-core). When running avx-512 heavy loads, it instead runs at 3.6 GHz -- and tends to get much hotter (70-80C instead of 50-60C). 3.6 GHz is a mild overclock; Silicon Lottery reports 100% can achieve that speed for avx512 loads.[1] Running programs doing the same thing (eg, Hamiltonian Monte Carlo where the lik…
What are you using to vectorize avx512 for HMC? Do you have a lot of element wise ops on big arrays? When running Stan (NUTS/HMC) on Xeon Phi, telling Eigen to use avx512 provided a noticeable speed up but I didn't look at the assembly to be sure.
I've been using Julia. I've been working on a front end meant to help specify vectorized models and their gradients. It is alpha-quality software (far from production ready), but here is the github:
https://github.com/chriselrod/ProbabilityModels.jl
In the example I give there, the logdensity and gradient evaluation was about 25x faster than Stan, and sampling was about 20x faster. A simulation fitting many data sets for my dissertation took about 9 hours. 20x is the difference between running overnight, and taking a week.
If I understand correctly, one problem Stan has is that it uses a var datatype for its arrays, which interleaves the values (Scalar) with pointers (vi_). https://github.com/stan-dev/math/blob/master/stan/math/rev/c...
This interleaving is going to cause problems to an autovectorizer. To get a SIMD vector of the scalars, you'd probably have to load two vectors, and then blend them.
Even with arrays of doubles, I found Eigen's fixed size arrays to get about 3-8x worse performance than my Julia library (3-8x worse than my Julia library for Mx32 * 32xN, for combinations of M and N = (3,...,32) ): https://bayeswatch.org/2019/06/06/small-matrix-multiplicatio...
I compiled the Eigen benchmarks with: g++ -O3 -fno-signed-zeros -fno-trapping-math -fassociative-math -march=native -mprefer-vector-width=512 -shared -fPIC -I/usr/include/eigen3 eigen_mul.cpp -o libeigenmul.so
How did you tell Eigen to use avx512? At the time, I was getting errors when specifying -DEIGEN_ENABLE_AVX512. http://eigen.tuxfamily.org/bz/show_bug.cgi?id=1705
Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors
#97Earlier quoted context omitted.
It's not really first class if you have to CVT BF16 to FP32 if you want to use FMA.
FMA on BF16 uses FP32 for the accumulator for accuracy reasons, but you can have instructions for "Rm = Rm + low(Rn) * low(Rp)" and "Rm = Rm + high(Rn) * high(Rp)", and they can be faster than FP32 "Rm=Rm+Rn*Rp" because there are fewer bits in the mantissas. Also, converting BF16 to FP32 and back is just a vector shuffle that sticks/drops 16 extra mantissa bits at the end of each FP16 value, so it's cheaper than othe…
That's a good analysis. I particularly like the idea that if the overhead is low it gets hidden in the memory BW savings.
Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors
#98Earlier quoted context omitted.
I read this article as saying, "hey we can emulate bfloat16 pretty well in software on top of our existing hardware features". That's what "graft" and "minimal impact" mean to me. Intel (for better or worse) takes a very experiment-results-driven approach to choosing which features to implement in hardware. So this result -- that software emulation of a feature works almost as well as a hardware implementation would…
The article links to another saying that bfloat16 is coming to Xeons. Also the title of the current says it too. There's nothing implying less likely if you read the articles. https://www.nextplatform.com/2018/12/16/intel-unfolds-roadma...
Ouch, you're right! I misunderstood.
Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors
#99> At this point, Intel doesn’t have bfloat16 implemented in any of its processors, so they used current AVX512 vector hardware present in its existing processor to emulate the format and the requisite operations. According to the researchers, this resulted in “only a very slight performance tax.” Why implement bfloat if you get just slightly less performance emulating it with AVX512, which already exists? Maybe it’s…
The reason to use float16 isn’t to make individual operations faster, its to fit more numbers in vector registers. For example, AVX512 vector registers are 512 bits. That’s 8 64-bit floats, 16 32-bit ones, or 32 16-bit ones. => A vector multiply using bfloat16 may not be much faster than one using float32, but it will do more multiplications.
> The reason to use float16 isn’t to make individual operations faster, its to fit more numbers vector registers. For example, AVX512 vector registers are 512 bits. That’s 8 64-bit floats, 16 32-bit ones, or 32 16-bit ones.
I'm unclear on the advantage you are trying to explain. If both AVX and bfloat are SIMD instructions that cannot be the reason implementing bfloat is better. I'm expecting something like "bfloat16 is more specialised so it can have larger registers" or something?
[Edit]
Sorry re-reading your comment, I think you are trying to say that. The key part being:
> fit more numbers vector registers
(vs AVX i assume), so adding bfloat16 would provide more registers vs AVX with similar gate usage due to greater specialization?
Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors
#100This is super exciting! Brings a bit of competition to NVIDIA for ML-related tasks, while being more "open" (to some extent) than the TPU ASICs (because you won't have single-cloud lock-in). In any case, good to see Intel finally waking up.
Is Intel planning TPU's or GPU's?
I don't see how GPU with AVX512 can compete with TPU's or GPU's, BF16 or not.