Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elementary Functions and Not Following IEEE754 Floating-Point Standard (2020)

hlsl.co.uk

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Elementary Functions and Not Following IEEE754 Floating-Point Standard (2020)

#21

Post by dzaima »

For 32-bit float single-operand ops it's simple enough to rely on a brute-force check. For 64-bit floats, though, while the goal of "sin(x) in library A really should match sin(x) in library B" is nice, it essentially ends up as "sin(x) in library A really should match... ...well, there's no option other than library A if I want sin(x) to not take multiple microseconds of bigint logic".

Though, a potentially-useful note is that for two-argument functions is that a correctly-rounded implementation means that it's possible to specialize certain constant operands to a much better implementations while preserving the same result (log(2,x), log(10,x), pow(x, 0.5), pow(x, 2), pow(x, 3), etc; floor(log(int,x)) being potentially especially useful if an int log isn't available).

Re: Elementary Functions and Not Following IEEE754 Floating-Point Standard (2020)

#22

Post by stephencanon »

lifthrasiir wrote: post #4

> All the standard Maths libraries which claim to be IEEE complaint I have tested are not compliant with this constraint. It is possible to read the standard in the way that they still remain compliant. The standard, as of IEEE 754-2019, does not require recommended operations to be implemented in accordance to the standard in my understanding; implementations are merely recommended to ("should") define recommended o…

Right; IEEE 754 (2019) _recommends_ that correctly rounded transcendental functions be provided, but does not require it. The next revision of the standard may require that a subset of correctly-rounded transcendental functions be provided (this is under active discussion), but would still not require how they are bound in the language (i.e. a C stdlib might provide both `pow` and `cr_pow` or similar), and might not require that all functions be correctly rounded over their entire range (e.g. languages might require only that `cr_sin` be correctly rounded on [-π,π]).

Re: Elementary Functions and Not Following IEEE754 Floating-Point Standard (2020)

#23

Post by alkonaut »

saagarjha wrote: post #12
alkonaut wrote: post #11

Earlier quoted context omitted.

I think you mean double precision?

No.

Can you elaborate? Outside of AI workloads, almost all computations on GPUs are single precision. Double precision is pretty rare, and smaller precision is mostly useless outside AI (and obviously irrelevant for trig precision)

Re: Elementary Functions and Not Following IEEE754 Floating-Point Standard (2020)

#24

Post by saagarjha »

alkonaut wrote: post #23
saagarjha wrote: post #12

Earlier quoted context omitted.

No.

Can you elaborate? Outside of AI workloads, almost all computations on GPUs are single precision. Double precision is pretty rare, and smaller precision is mostly useless outside AI (and obviously irrelevant for trig precision)

Unfortunately almost all computations on GPUs are AI workloads.

Re: Elementary Functions and Not Following IEEE754 Floating-Point Standard (2020)

#25

Post by pclmulqdq »

saagarjha wrote: post #24
alkonaut wrote: post #23

Earlier quoted context omitted.

Can you elaborate? Outside of AI workloads, almost all computations on GPUs are single precision. Double precision is pretty rare, and smaller precision is mostly useless outside AI (and obviously irrelevant for trig precision)

Unfortunately almost all computations on GPUs are AI workloads.

Uh, what?

The main usage of GPUs is graphics processing, and it's not even close. That is what Graphics Processing Units are built for. AI is probably the main use of datacenter GPUs today, but even that isn't "almost all" in comparison to the HPC work out there.

Re: Elementary Functions and Not Following IEEE754 Floating-Point Standard (2020)

#26

Post by dzaima »

AlotOfReading wrote: post #20
zokier wrote: post #8

Earlier quoted context omitted.

That Zimmermann paper is far more useful than the article. Notably LLVM libm is correctly rounded for most single precision ops. Notable omission are crlibm/rlibm/core-math etc libs which claim to be more correct, but I suppose we can already be pretty confident about them.

I've found that LLVM is slightly worse than GCC on reproducibility once you get past the basic issues in libm. For example, LLVM will incorrectly round sqrt on ppc64el in some unusual circumstances: https://github.com/J-Montgomery/rfloat/blob/8a58367db32807c8...

...on -ffast-math. Of course you'll have arbitrary behavior of (at least) a couple ULPs on -ffast-math, but it'll be faster (hopefully)! That's, like, the flag's whole idea.

Re: Elementary Functions and Not Following IEEE754 Floating-Point Standard (2020)

#27

Post by AlotOfReading »

dzaima wrote: post #26
AlotOfReading wrote: post #20

Earlier quoted context omitted.

I've found that LLVM is slightly worse than GCC on reproducibility once you get past the basic issues in libm. For example, LLVM will incorrectly round sqrt on ppc64el in some unusual circumstances: https://github.com/J-Montgomery/rfloat/blob/8a58367db32807c8...

...on -ffast-math. Of course you'll have arbitrary behavior of (at least) a couple ULPs on -ffast-math, but it'll be faster (hopefully)! That's, like, the flag's whole idea.

I didn't say it was a bug for exactly that reason, but as the comment explains this doesn't affect any platform or compiler combination except ppc64 LLVM, or even most usages of sqrt for that target.

LLVM has lots of small reproducibility issues like this that GCC doesn't, but also much better documentation around its limitations. The point of this library is to eliminate as many of those as possible without performance costs.

Re: Elementary Functions and Not Following IEEE754 Floating-Point Standard (2020)

#28

Post by dzaima »

AlotOfReading wrote: post #27
dzaima wrote: post #26

Earlier quoted context omitted.

...on -ffast-math. Of course you'll have arbitrary behavior of (at least) a couple ULPs on -ffast-math, but it'll be faster (hopefully)! That's, like, the flag's whole idea.

I didn't say it was a bug for exactly that reason, but as the comment explains this doesn't affect any platform or compiler combination except ppc64 LLVM, or even most usages of sqrt for that target. LLVM has lots of small reproducibility issues like this that GCC doesn't, but also much better documentation around its limitations. The point of this library is to eliminate as many of those as possible without performa…

"doesn't affect other things" is in no way ever whatsoever a reason to do anything in any way related to thinking, believing, or hoping that it might, would, or should affect nothing, especially around compiler optimizations (fun sentence to write, and one I violate myself for some things, but trivially true regardless). I'd be curious to hear about actual issues though.

The ppc64 case looks like llvm very intentionally not using the existing square root instruction, instead emitting a sequence of manual operations that supposedly run faster. And it's entirely in its right to do so, and it should affect no correct code (not that it's even really possible to write "correct code" under -ffast-math).

Re: Elementary Functions and Not Following IEEE754 Floating-Point Standard (2020)

#29

Post by dzaima »

AlotOfReading wrote: post #20
zokier wrote: post #8

Earlier quoted context omitted.

That Zimmermann paper is far more useful than the article. Notably LLVM libm is correctly rounded for most single precision ops. Notable omission are crlibm/rlibm/core-math etc libs which claim to be more correct, but I suppose we can already be pretty confident about them.

I've found that LLVM is slightly worse than GCC on reproducibility once you get past the basic issues in libm. For example, LLVM will incorrectly round sqrt on ppc64el in some unusual circumstances: https://github.com/J-Montgomery/rfloat/blob/8a58367db32807c8...

On the ppc64 thing, some ways to avoid it: https://godbolt.org/z/fx88rYz5v

Re: Elementary Functions and Not Following IEEE754 Floating-Point Standard (2020)

#30

Post by AlotOfReading »

dzaima wrote: post #28
AlotOfReading wrote: post #27

Earlier quoted context omitted.

I didn't say it was a bug for exactly that reason, but as the comment explains this doesn't affect any platform or compiler combination except ppc64 LLVM, or even most usages of sqrt for that target. LLVM has lots of small reproducibility issues like this that GCC doesn't, but also much better documentation around its limitations. The point of this library is to eliminate as many of those as possible without performa…

"doesn't affect other things" is in no way ever whatsoever a reason to do anything in any way related to thinking, believing, or hoping that it might, would, or should affect nothing, especially around compiler optimizations (fun sentence to write, and one I violate myself for some things, but trivially true regardless). I'd be curious to hear about actual issues though. The ppc64 case looks like llvm very intentiona…

Note that this isn't even a case where a single sqrt() call is non-reproducible, but only sequences of inlined sqrt calls within an unrolled loop and I agree with you.

Some broader context is probably warranted though. This originated out of a discussion with the authors of P3375 [0] about the actual performance costs of reproducibility. I suspected that existing compilers could already do it without a language change and no runtime cost using inline assembly magic. This library was the experiment to see if I was full of it.

There were only a few minor limitations it found. One was this issue, which happens "outside" what the library is attempting to do (though potentially still fixable as your godbolt link demonstrated). Another was that Clang and GCC have slightly different interpretations of "creative" register constraints. Clang's interpretation is closer to GCC's docs than GCC itself, but produces worse code.

Otherwise, this gives you reproducibility right up to the deeper compiler limitations like NaN propagation at essentially no performance cost. I wasn't able to find any "real" cases where it's not reproducible, only incredibly specific situations like this one across all 3 major compilers and even the minor ones I tried.

[0] https://isocpp.org/files/papers/P3375R2.html

Post reply on HN