81–90 of 138 posts
I desperately want Chinese companies finally begin producing and designing general purpose CPUs, GPUS, and other types of accelerators. Current situation is terrible duo- and mono-polies, slow pace of innovations, and low reliability. We need more players.
It may well be a stupid q but I really don't know, and always assumed they would be [-1..+1] and that fixed point would suffice. Clearly not.
Earlier quoted context omitted.
They had an assembler where one person wrote kernels that were faster than cublas in a lot of cases. Afaik, nobody ever released anything else with that assembler, and Nvidia caught up to that performance quickly. In talking with the cublas devs, it seemed more that maxas kernels were highly tuned for specific sizes, whereas cublas/cudnn had to be more general. Nowadays it's really a moot point with Nvidia's Cutlass…
True story-- the Nervana Maxwell stuff didn't go very far-- but it was noteworthy because they had both that small win as well as their own hardware platform. One other thought about the Nervana-Xeon convergence is that the support for more memory (thru DDR, Optane, or even just mmap'ed NVME) will be a big win for modeling and large-minibatch SGD. For example, the minibatch fetching could be pushed to the hardware /…
They're not really "grafting" it, they're implementing it as a first-class data type.
It's not really first class if you have to CVT BF16 to FP32 if you want to use FMA.
Also, converting BF16 to FP32 and back is just a vector shuffle that sticks/drops 16 extra mantissa bits at the end of each FP16 value, so it's cheaper than other floating point conversions. This means that even if you occasionally have to escape to FP32, the overhead is low and you keep all of the memory bandwidth benefits.
Earlier quoted context omitted.
True story-- the Nervana Maxwell stuff didn't go very far-- but it was noteworthy because they had both that small win as well as their own hardware platform. One other thought about the Nervana-Xeon convergence is that the support for more memory (thru DDR, Optane, or even just mmap'ed NVME) will be a big win for modeling and large-minibatch SGD. For example, the minibatch fetching could be pushed to the hardware /…
A V100 GPU gets 900GB/s of memory bandwidth. I am less of a CPU expert but afaict you'll be lucky to get much more than 10% of that out of a CPU. This is going to make a huge difference that Intel can't make up with bigger execution units. bfloat helps with this because the data is half as large. But of course if you're doing Nvidia you're probably already doing (IEEE) fp16.
When GPUs kickstarted deep learning research in 2012, people had already studied shallow models on mapreduce for a decade or so. Once NVME / Optane and modern CPUs get 1-10TB of useful “memory” in the hands of grad students, there should be another wave of new research. To date, my experience has been that 1TB of “memory” is only commonly available in industry.
> At this point, Intel doesn’t have bfloat16 implemented in any of its processors, so they used current AVX512 vector hardware present in its existing processor to emulate the format and the requisite operations. According to the researchers, this resulted in “only a very slight performance tax.” Why implement bfloat if you get just slightly less performance emulating it with AVX512, which already exists? Maybe it’s…
They did not specify what the tax was relative to. Maybe they meant relative to 32-bit float? AVX512 is expensive. I believe if you have an AVX512-heavy workload it can cause the processor to throttle.
does anyone know what is the story on the software side - CUDA is basically industry standard now. Tensorflow OpenCL support bug [1] has been open for FOUR years now (with the discussion devolving into an Intel PlaidML flame war). AMD OpenCL is now ROCm ? At the end of the day, I cant run ANY accelerated workloads using Intel graphics or AMD ....because there's simply no software support anywhere. OTOH, if you have a…
You can, there’s an Intel framework: OpenVINO that is targeting Intel hardware, processors and HD video cards, you can convert TF graph to Vino and use their inference server, that mimics TF serving.
There’s TF ROCm port by AMD as well.
does anyone know what is the story on the software side - CUDA is basically industry standard now. Tensorflow OpenCL support bug [1] has been open for FOUR years now (with the discussion devolving into an Intel PlaidML flame war). AMD OpenCL is now ROCm ? At the end of the day, I cant run ANY accelerated workloads using Intel graphics or AMD ....because there's simply no software support anywhere. OTOH, if you have a…
As it stands, CUDA is the only API with a decent implementation available on the three major desktop OSes. It's ridiculous.
Earlier quoted context omitted.
It isn't really very good for machine learning - machine learning doesn't need 8 bits of exponent. For weights during training, 7 bits of mantissa also seems a bit low - it's common for weights to adjust much less than 1% during a single batch of training, which this couldn't represent. I think this is more a "we want something which is faster but is compatible with existing code written for fp32's".
Bfloat is specifically designed for ML. It is the native type in Google's TPUs. It is quite good at ML; most models that work with fp32 work with bfloat with no adjustments; that's in contrast to IEEE fp16. You're right that the mantissa is small. The trick is that you always accumulate into fp32 and then truncate down to 16 bits at the end. You'd do this for any 16-bit floating type. Source: I work on this at Google…
In particular the latter describes a generic framework that can be used to generate a lot of different number systems. Could hardware implement this, allowing us to compose and choose the number system by just setting some simple flags?
I am sorry for Intel. Perhaps John Gustafson’s 16 bit posits or unums would have made a better choice.