Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intel Prepares to Graft Google’s Bfloat16 onto Processors

nextplatform.com

Post reply on HN
Print view View original

121–130 of 138 posts

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#121

Post by nkurz »

simonebrunozzi wrote: post #118
simonebrunozzi wrote: post #40

English is not my first language. I have never heard the term "Graft", even if I consider myself quite literate in English. So here you go, for everybody else in my situation: Graft, as understood in American English, is a form of political corruption, being the unscrupulous use of a politician's authority for personal gain. Edit: by the way, I really couldn't fit the term with the article. And realized I was probabl…

I really don't get it: I was donwovoted (currently at -4) on the parent comment, because I was confused by a word and tried to clarify that it meant. Does that really deserve downvoting? If so, explain it to me please, because I just can't see why.

No, you don't deserve it. In its non-horticultural sense, "graft" is a particularly tricky word, because it's familiar to most native English speakers, but can mean almost opposing things in England and America. So people often think they understand what is being said, while actually misunderstanding each other. Here's an article on the topic: https://separatedbyacommonlanguage.blogspot.com/2012/01/graf....

On the other hand, many of the readers of this blog are native English speakers, and to most of us the metaphorical meaning of the title was clear. A very useful function of voting is to re-order the comments on the page. This is a useful comment, but only for a small subset of readers. As such, it's perfectly reasonable that it would appear after the other more technical comments.

Which is to say that while you don't deserve to be downvoted, the comment arguably does. I personally upvoted it, because of my personal experience with intercontinental miscommunication involving the word "graft", but I can see why others would want to prioritize other comments. Not because it's a bad comment --- I'm sure a few people found it really helpful --- but because it's a meta-comment on a technical site that helps only a small portion of the audience.

So you should proudly keep making helpful comments like this, and not take it at all personally when they are downvoted to the bottom of the page. This is a case where you should feel confident that you did the right thing, despite the apparent feedback.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#122

Post by cbhl »

amelius wrote: post #83
choppaface wrote: post #71

Earlier quoted context omitted.

True story-- the Nervana Maxwell stuff didn't go very far-- but it was noteworthy because they had both that small win as well as their own hardware platform. One other thought about the Nervana-Xeon convergence is that the support for more memory (thru DDR, Optane, or even just mmap'ed NVME) will be a big win for modeling and large-minibatch SGD. For example, the minibatch fetching could be pushed to the hardware /…

Can't deep learning be done using 16 bit fixed point instead?

If you're using fixed point, wouldn't you just as easily substitute 16-bit integers instead?

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#123

Post by simonebrunozzi »

nkurz wrote: post #121
simonebrunozzi wrote: post #118

Earlier quoted context omitted.

I really don't get it: I was donwovoted (currently at -4) on the parent comment, because I was confused by a word and tried to clarify that it meant. Does that really deserve downvoting? If so, explain it to me please, because I just can't see why.

No, you don't deserve it. In its non-horticultural sense, "graft" is a particularly tricky word, because it's familiar to most native English speakers, but can mean almost opposing things in England and America. So people often think they understand what is being said, while actually misunderstanding each other. Here's an article on the topic: https://separatedbyacommonlanguage.blogspot.com/2012/01/graf... . On the o…

Thanks for sharing your point of view, I appreciate it.

In this situation I would personally not downvote, but rather upvote the other more technical comments, which provides a similar result, without penalizing the commenter.

Your interpretation might be right, but a downvote will often be interpreted as "I didn't contribute to the conversation". The fact that "graft" is confusing, and that I am trying to shed light on it, is a way for me to try contributing, and therefore, in my view, shouldn't be penalized.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#124

Post by shaklee3 »

tempguy9999 wrote: post #79
jlebar wrote: post #76

Earlier quoted context omitted.

A V100 GPU gets 900GB/s of memory bandwidth. I am less of a CPU expert but afaict you'll be lucky to get much more than 10% of that out of a CPU. This is going to make a huge difference that Intel can't make up with bigger execution units. bfloat helps with this because the data is half as large. But of course if you're doing Nvidia you're probably already doing (IEEE) fp16.

That sounded a bit low. This https://en.wikichip.org/wiki/intel/microarchitectures/cooper... says "Higher bandwidth (174.84 GiB/s, up from 119.209 GiB/s)" I don't know if memory bandwidth matters for this type of job, though.

Intel is dominated by how many cores/threads are accessing simultaneously. So with many scientific libraries you can get about 80% of the throughput. And the V100 will not give you 900GBps. That's the theoretical, but nominally it's about 750GBps.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#125

Post by shaklee3 »

jlebar wrote: post #74
shaklee3 wrote: post #56

Earlier quoted context omitted.

They had an assembler where one person wrote kernels that were faster than cublas in a lot of cases. Afaik, nobody ever released anything else with that assembler, and Nvidia caught up to that performance quickly. In talking with the cublas devs, it seemed more that maxas kernels were highly tuned for specific sizes, whereas cublas/cudnn had to be more general. Nowadays it's really a moot point with Nvidia's Cutlass…

Cutlass doesn't match the performance of cublass, even on nvidia's benchmarks. Hand assembly is still alive and well!

That's somewhat nitpicking. Cutlass is 90-95% of cublas in many cases, and there are things Cutlass can do that cublas can't. Cublas will always be marginally faster, but it won't matter in most cases, especially if that means saving a kernel call.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#126

Post by drewm1980 »

krick wrote: post #59

That's surprising, I wouldn't guess it is something they would do so easily, especially considering Intel processors isn't something you generally use to train NNs. But I'd be rather glad if they implemented unums already.

I absolutely train on Intel from time to time at work. If your model fits, cpu is a dream. The driver never breaks, they never crash and lock up your display...

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#127

Post by drewm1980 »

tempguy9999 wrote: post #82

Newbie question: what is the typical and extreme values (excluding +/-infinity, and are these used too?) that can occur in training/running of NNs? Also, what level of accuracy is needed? It may well be a stupid q but I really don't know, and always assumed they would be [-1..+1] and that fixed point would suffice. Clearly not.

Nope, no standardization at that level.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#128

Post by marmaduke »

celrod wrote: post #96
marmaduke wrote: post #72

Earlier quoted context omitted.

What are you using to vectorize avx512 for HMC? Do you have a lot of element wise ops on big arrays? When running Stan (NUTS/HMC) on Xeon Phi, telling Eigen to use avx512 provided a noticeable speed up but I didn't look at the assembly to be sure.

I've been using Julia. I've been working on a front end meant to help specify vectorized models and their gradients. It is alpha-quality software (far from production ready), but here is the github: https://github.com/chriselrod/ProbabilityModels.jl In the example I give there, the logdensity and gradient evaluation was about 25x faster than Stan, and sampling was about 20x faster. A simulation fitting many data sets…

> here is the github: https://github.com/chriselrod/ProbabilityModels.jl. In the example I give there, the logdensity and gradient evaluation was about 25x faster than Stan, and sampling was about 20x faster.

that looks pretty cool, though I don't yet know enough Julia to understand all of it. The speedups make sense given that Stan's compiler/math lib doesn't do much in the way of smart data layout. I would still keep in mind that the metric worth using for benchmarking is the number of effective samples per second, and this also depends on the HMC variant you use.

> Eigen's fixed size arrays to get about 3-8x worse performance than my Julia library

seems unsurprising that Julia can specialize a lot better than verbose C++ templating, no? (still, good job, very worth checking out)

> I was getting errors when specifying -DEIGEN_ENABLE_AVX512

I used this flag with Eigen 3.3.1, I think, on GCC 6 or 7. This was for Xeon Phi, so I tried to use icc but despite supporting C++11 it doesn't handle Stan or Eigen's template metaprogramming.

This is all the more reason to use Julia, but my graduate student days are long past..

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#129

Post by celrod »

marmaduke wrote: post #128
celrod wrote: post #96

Earlier quoted context omitted.

I've been using Julia. I've been working on a front end meant to help specify vectorized models and their gradients. It is alpha-quality software (far from production ready), but here is the github: https://github.com/chriselrod/ProbabilityModels.jl In the example I give there, the logdensity and gradient evaluation was about 25x faster than Stan, and sampling was about 20x faster. A simulation fitting many data sets…

> here is the github: https://github.com/chriselrod/ProbabilityModels.jl . In the example I give there, the logdensity and gradient evaluation was about 25x faster than Stan, and sampling was about 20x faster. that looks pretty cool, though I don't yet know enough Julia to understand all of it. The speedups make sense given that Stan's compiler/math lib doesn't do much in the way of smart data layout. I would still k…

> I would still keep in mind that the metric worth using for benchmarking is the number of effective samples per second, and this also depends on the HMC variant you use.

I was getting similar effective sample sizes/sample size in both after switching to a diagonal mass matrix, like Stan uses, from the dense mass matrix DynamicHMC.jl uses by default (the HMC backed library I'm using).

Given how common it is for folks to run Stan over night or for a week to study prior sensitivity, internal coverage, type I and II errors, etc, via Monte Carlo, I think a focus on speed is worth while.

> seems unsurprising that Julia can specialize a lot better than verbose C++ templating, no? (still, good job, very worth checking out)

The C++ library Blaze did a lot better than Eigen, but still not as well. But yes, Julia'a meta-programming is much easier to work with. Julia expressions are Julia objects that you can manipulate like anything else, so I can write all the functions I want describing how to generate matmul kernels as a function of matrix size and CPU Info, and how to loop over them.

That approach feels much more straightforward. I haven't looked at the code bases of Eigen or Blaze, nor am I that familiar with template meta-programming. But I'd guess they define matmul recursively for arbitrary fixed sizes, and then have some templates defined for specific sizes (the kernels) -- or ideally have some clever way of generating the kernels from there.

Regardless, I agree that this is much easier in Julia. Aggressive specialization is also better aligned with Julia's compilation model in general, because methods get compiled just before they're used. Defining a million possible specializations doesn't have the cost of compiling a million specializations.

> This is all the more reason to use Julia, but my graduate student days are long past..

I'm defending this week, and next Monday will be my first day in an industry job. They expressed openness to Julia, but my biggest fear is that they'll renege so that I'll only be able to work on or use Julia in my spare time at home.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#130

Post by jlebar »

shaklee3 wrote: post #124
tempguy9999 wrote: post #79

Earlier quoted context omitted.

That sounded a bit low. This https://en.wikichip.org/wiki/intel/microarchitectures/cooper... says "Higher bandwidth (174.84 GiB/s, up from 119.209 GiB/s)" I don't know if memory bandwidth matters for this type of job, though.

Intel is dominated by how many cores/threads are accessing simultaneously. So with many scientific libraries you can get about 80% of the throughput. And the V100 will not give you 900GBps. That's the theoretical, but nominally it's about 750GBps.

My project, XLA, will get you quite close to the nominal 900GB/s. :)
Post reply on HN