Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intel Prepares to Graft Google’s Bfloat16 onto Processors

nextplatform.com

Post reply on HN
Print view View original

131–138 of 138 posts

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#131

Post by burfog »

Veedrac wrote: post #120
burfog wrote: post #119

Earlier quoted context omitted.

Sure, I can see that being the case. My point is that the Xeon should support denormals. The Bfloat16 format does allow for denormals. Intel's implementation mangles them, changing them to 0.0 on both input and output.

But then you need either new denormal-accepting instructions or, worse, a new global state bit enabling bfloat16 denormals, all to support use cases probably over two orders of magnitude less common than ML. What's the compelling reason to bother? Note that you need to support the denormal-disabled case because you'll want compatibility with Nervana.

Intel did add a global state bit. It just isn't useful because you can't modify it.

Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#132

Post by Veedrac »

burfog wrote: post #131
Veedrac wrote: post #120

Earlier quoted context omitted.

But then you need either new denormal-accepting instructions or, worse, a new global state bit enabling bfloat16 denormals, all to support use cases probably over two orders of magnitude less common than ML. What's the compelling reason to bother? Note that you need to support the denormal-disabled case because you'll want compatibility with Nervana.

Intel did add a global state bit. It just isn't useful because you can't modify it. Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output.

> Intel did add a global state bit. It just isn't useful because you can't modify it.

You mean the CPUID bit? That's free. Toggling denormals isn't.

> Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output.

Nervana isn't discontinued according to their website[1], and bitwise compatibility does matter, certainly more than denormals do.

[1] https://www.intel.ai/ai-at-ces/

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#133

Post by burfog »

Veedrac wrote: post #132
burfog wrote: post #131

Earlier quoted context omitted.

Intel did add a global state bit. It just isn't useful because you can't modify it. Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output.

> Intel did add a global state bit. It just isn't useful because you can't modify it. You mean the CPUID bit? That's free. Toggling denormals isn't. > Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output. Nervana isn't discontinued according to their website[1], and bitwise compatibility does matter, certainly more than de…

I mean the bit to toggle denormals, not the one to identify support for the opcodes.

Denormals are far more important than bitwise compatibility. To be clear, you would still be able to load a Nervana-produced number into a processor that supports denormals, and the other way would work too. You'd just avoid mangling numbers that are near zero.

If you still think denormals don't matter, seriously do what I suggested: draw it out on graph paper. They matter.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#134

Post by marmaduke »

celrod wrote: post #129
marmaduke wrote: post #128

Earlier quoted context omitted.

> here is the github: https://github.com/chriselrod/ProbabilityModels.jl . In the example I give there, the logdensity and gradient evaluation was about 25x faster than Stan, and sampling was about 20x faster. that looks pretty cool, though I don't yet know enough Julia to understand all of it. The speedups make sense given that Stan's compiler/math lib doesn't do much in the way of smart data layout. I would still k…

> I would still keep in mind that the metric worth using for benchmarking is the number of effective samples per second, and this also depends on the HMC variant you use. I was getting similar effective sample sizes/sample size in both after switching to a diagonal mass matrix, like Stan uses, from the dense mass matrix DynamicHMC.jl uses by default (the HMC backed library I'm using). Given how common it is for folks…

> switching to a diagonal mass matrix

That's an interesting comment. We've always used Stan's default of a diagonal, but I think we'd benefit from mixed metrics, which doesn't seem possible in Stan, but looks somewhat doable in some of the HMC libs in Julia.

> Given how common it is for folks to run Stan over night or for a week to study prior sensitivity

Yes, we changed the walltime on our Slurm cluster to support Stan Jobs running up to a week long, have used multiple million core hours on this. Stan still isn't so shabby but it's a hard problem.

> I'm defending this week, and next Monday will be my first day in an industry job.

Good luck and congrats on the job. You'll probably have to bite your tongue and look for opportunities where Julia's advanced compilation model (as you described well above) is going to more than pay for the cost of deployment/extra language etc.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#135

Post by Veedrac »

burfog wrote: post #133
Veedrac wrote: post #132

Earlier quoted context omitted.

> Intel did add a global state bit. It just isn't useful because you can't modify it. You mean the CPUID bit? That's free. Toggling denormals isn't. > Nervana is discontinued, isn't it? Compatibility doesn't matter. It's pretty compatible anyway, as long as you aren't demanding bit-identical output. Nervana isn't discontinued according to their website[1], and bitwise compatibility does matter, certainly more than de…

I mean the bit to toggle denormals, not the one to identify support for the opcodes. Denormals are far more important than bitwise compatibility. To be clear, you would still be able to load a Nervana-produced number into a processor that supports denormals, and the other way would work too. You'd just avoid mangling numbers that are near zero. If you still think denormals don't matter, seriously do what I suggested:…

> I mean the bit to toggle denormals, not the one to identify support for the opcodes.

bfloat16 doesn't handle denormals, why is there a bit to toggle it? What's it called (so I can CTRL-f for it)?

> If you still think denormals don't matter, seriously do what I suggested: draw it out on graph paper. They matter.

No, I get how denormals work, I know what you're pointing at. But ML genuinely doesn't care, neural nets don't give a damn about mathematical purity[1]. In contrast compatibility matters because ML doesn't give you any guarantee that it's not depending on the behaviour at small values, and minor differences in rounding does cause issues. For example, Leela Chess Zero had difficulties with reproducibility because different GPUs round floats differently.

[1] Fun but relevant aside: https://openai.com/blog/nonlinear-computation-in-linear-netw...

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#136

Post by shaklee3 »

jlebar wrote: post #130
shaklee3 wrote: post #124

Earlier quoted context omitted.

Intel is dominated by how many cores/threads are accessing simultaneously. So with many scientific libraries you can get about 80% of the throughput. And the V100 will not give you 900GBps. That's the theoretical, but nominally it's about 750GBps.

My project, XLA, will get you quite close to the nominal 900GB/s. :)

Do you have more information?

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#137

Post by burfog »

Veedrac wrote: post #135
burfog wrote: post #133

Earlier quoted context omitted.

I mean the bit to toggle denormals, not the one to identify support for the opcodes. Denormals are far more important than bitwise compatibility. To be clear, you would still be able to load a Nervana-produced number into a processor that supports denormals, and the other way would work too. You'd just avoid mangling numbers that are near zero. If you still think denormals don't matter, seriously do what I suggested:…

> I mean the bit to toggle denormals, not the one to identify support for the opcodes. bfloat16 doesn't handle denormals, why is there a bit to toggle it? What's it called (so I can CTRL-f for it)? > If you still think denormals don't matter, seriously do what I suggested: draw it out on graph paper. They matter. No, I get how denormals work, I know what you're pointing at. But ML genuinely doesn't care, neural nets…

It's two bits, DAZ and FTZ. (seems like "denormals are zero" and "flush to zero")

Bfloat16 obviously can handle denormals. The encoding is possible. There would be no need to handle the issue if the encoding did not exist.

As hex, these would be denormal: 0x0001 to 0x0080, and 0x8001 to 0x8080. It's the same as plain old 32-bit IEEE, with half the bits lopped off.

ML is all about difficulties with reproducibility. I don't see a reason to get upset about denormals when a 3D-printed turtle can be confused with a rifle.

Re: Intel Prepares to Graft Google’s Bfloat16 onto Processors

#138

Post by Veedrac »

burfog wrote: post #137
Veedrac wrote: post #135

Earlier quoted context omitted.

> I mean the bit to toggle denormals, not the one to identify support for the opcodes. bfloat16 doesn't handle denormals, why is there a bit to toggle it? What's it called (so I can CTRL-f for it)? > If you still think denormals don't matter, seriously do what I suggested: draw it out on graph paper. They matter. No, I get how denormals work, I know what you're pointing at. But ML genuinely doesn't care, neural nets…

It's two bits, DAZ and FTZ. (seems like "denormals are zero" and "flush to zero") Bfloat16 obviously can handle denormals. The encoding is possible. There would be no need to handle the issue if the encoding did not exist. As hex, these would be denormal: 0x0001 to 0x0080, and 0x8001 to 0x8080. It's the same as plain old 32-bit IEEE, with half the bits lopped off. ML is all about difficulties with reproducibility. I…

> It's two bits, DAZ and FTZ. (seems like "denormals are zero" and "flush to zero")

You mean the standard ones for normal floats? You certainly wouldn't want to reuse that for bfloat16s.

> ML is all about difficulties with reproducibility. I don't see a reason to get upset about denormals when a 3D-printed turtle can be confused with a rifle.

These are different things, despite the similarity in terminology.

Post reply on HN