-funsafe-math-optimizations always makes me laugh. Of course I want fun and safe math optimisations
Personally I’m a fan of Kotlin’s “fun factory()”
Beware of fast-math
81–90 of 111 posts
Re: Beware of fast-math
#82Earlier quoted context omitted.
Knowing said teacher ;) I wonder if he’d still say the same thing now… It’s good practice to have to use single precision (or even half-precision!) now and then in order to be forced to deal with precision issues. Yes, use doubles if you really need them and aren’t trying to learn. But they’re often a lot more than 2x more expensive, and they might not be necessary at all. I’ve heard people who develop commercial ren…
Perhaps you were in the same lecture as me, when I asked the lead developer on Big Hero 6 why they didn't just use doubles to solve their precision woes, and he informed me that they literally couldn't afford to use doubles at that scale.
You know, that is actually ringing a bell, I think I might have indeed. Above I was thinking of someone else who works on a certain renderer made in New Zealand, but it’s true that many studios using doubles either sparingly or not at all. That might be getting even more true as GPUs blend into production…
Re: Beware of fast-math
#83"-fno-math-errno" and "-fno-signed-zeros" can be turned on without any problems. I got a four times speedup on functions with no loss in accuracy.
Unless, of course, you have some algorithm that depends on signed zeros. Which is basically the same with all the optimizations the article complains about.
I'd suggest -ffp-contract=fast is a good idea for 99% of code. It's only going to break things where very specific effort has gone in to the numerical analysis, and likely the authors of such things are sufficiently fp-savy to tell you not to do the thing.
Re: Beware of fast-math
#84"-fno-math-errno" and "-fno-signed-zeros" can be turned on without any problems. I got a four times speedup on functions with no loss in accuracy.
Unless, of course, you have some algorithm that depends on signed zeros. Which is basically the same with all the optimizations the article complains about. I'd suggest -ffp-contract=fast is a good idea for 99% of code. It's only going to break things where very specific effort has gone in to the numerical analysis, and likely the authors of such things are sufficiently fp-savy to tell you not to do the thing.
Is there any algorithm that depends on signed zeros? I'm not aware of any.
Re: Beware of fast-math
#85The LLVM IR is more expressive than clang is for expressing fast-math: it supports making an operation use fast-math optimization on a per operation basis ( https://llvm.org/docs/LangRef.html#fastmath ).
Do you know what happens when you have ops with different flags? e.g. if you have (a + b) + c, where one + allows reassoc but one doesn't?
(a+b)+c has two ops in LLVM: addition is a binop, meaning it has two "arguments", thus (a+b) and adding "c" are separate instructions. You can't directly add three or more values.
Re: Beware of fast-math
#86Earlier quoted context omitted.
When we were debating whether WebAssembly should support subnormal numbers (i.e. be IEEE compliant), some people often cited these mythical subnormal slowdowns. So Dan Gohman ran some benchmarks and the scary-sounding slowdowns amounted to something like less than 1% (i.e. noise) for almost all benchmarks. Interestingly, one benchmark did not converge correctly with FTZ (i.e. no subnormals) and actually ran 3x more i…
> When we were debating whether WebAssembly should support subnormal numbers (i.e. be IEEE compliant), some people often cited these mythical subnormal slowdowns. So Dan Gohman ran some benchmarks and the scary-sounding slowdowns amounted to something like less than 1% (i.e. noise) for almost all benchmarks. Interestingly, one benchmark did not converge correctly with FTZ (i.e. no subnormals) and actually ran 3x more…
This is strange, so it is likely that it might be more of a Java problem than a CPU problem.
I do not know how Java handles this, but maybe it actually enables exceptions for underflow which invoke some handler.
Otherwise I cannot see how you can obtain such a huge slowdown, unless your code consists entirely of back-to-back operations with denormals and of nothing else.
I am not sure what you mean by "state variables", but if they are pushed into the denormal range, they should be changed to double, not float.
If you push double variables into the denormals range, then it is likely that the algorithm must be modified, because this should not happen.
Underflows, i.e. denormals, are difficult to avoid when using float variables, which can be mandatory in DSP algorithms for audio or video, but outside the arrays processed with SIMD instructions at maximum speed, the scalar variables can be double, which should never underflow in most correct algorithms.
For computations run on CPUs, not GPUs, only very seldom there can be reasons to use a scalar float variable. Normally float should be used only for arrays.
Re: Beware of fast-math
#87It looks like -fassociative-math is "safe" in the sense that it can not be used to get UB in working code? That's a good property to make it easier to use in the right context.
See the one footnote: you can re-associate a list of 2046 numbers such that they sum to _any_ floating point number between 0 and 2^970. https://discourse.julialang.org/t/array-ordering-and-naive-s...
Floating point math is fun:
def re_add(a,b,c,d,e):
return (a+b+c+d+e) == (2 * (c+a+b+d+e))
print(re_add(1e17, -1e17, 3, 2, 1))Re: Beware of fast-math
#88Earlier quoted context omitted.
> When we were debating whether WebAssembly should support subnormal numbers (i.e. be IEEE compliant), some people often cited these mythical subnormal slowdowns. So Dan Gohman ran some benchmarks and the scary-sounding slowdowns amounted to something like less than 1% (i.e. noise) for almost all benchmarks. Interestingly, one benchmark did not converge correctly with FTZ (i.e. no subnormals) and actually ran 3x more…
This is strange, so it is likely that it might be more of a Java problem than a CPU problem. I do not know how Java handles this, but maybe it actually enables exceptions for underflow which invoke some handler. Otherwise I cannot see how you can obtain such a huge slowdown, unless your code consists entirely of back-to-back operations with denormals and of nothing else. I am not sure what you mean by "state variable…
entirely of back-to-back operations with denormals
In the context of sound, I could see this happening with an exponentially decaying envelope generator (or an IIR filter).
Re: Beware of fast-math
#89Earlier quoted context omitted.
> When we were debating whether WebAssembly should support subnormal numbers (i.e. be IEEE compliant), some people often cited these mythical subnormal slowdowns. So Dan Gohman ran some benchmarks and the scary-sounding slowdowns amounted to something like less than 1% (i.e. noise) for almost all benchmarks. Interestingly, one benchmark did not converge correctly with FTZ (i.e. no subnormals) and actually ran 3x more…
This is strange, so it is likely that it might be more of a Java problem than a CPU problem. I do not know how Java handles this, but maybe it actually enables exceptions for underflow which invoke some handler. Otherwise I cannot see how you can obtain such a huge slowdown, unless your code consists entirely of back-to-back operations with denormals and of nothing else. I am not sure what you mean by "state variable…
> unless your code consists entirely of back-to-back operations with denormals and of nothing else.
Ending with the data being entirely in the denormal range is a common occurrence in some audio algorithms (and in there, intel CPUs dominate by such a large margin it's not even funny) ; if that happens at the beginning of your signal processing pipeline you're in for a rough time
Re: Beware of fast-math
#90Earlier quoted context omitted.
Perhaps you were in the same lecture as me, when I asked the lead developer on Big Hero 6 why they didn't just use doubles to solve their precision woes, and he informed me that they literally couldn't afford to use doubles at that scale.
You know, that is actually ringing a bell, I think I might have indeed. Above I was thinking of someone else who works on a certain renderer made in New Zealand, but it’s true that many studios using doubles either sparingly or not at all. That might be getting even more true as GPUs blend into production…
I worked on a certain hopping lamp renderer for more than 11 eleven years. I can confirm that probably 99+% of the floating point math in it was in single precision.
And to this day, typing out the 'f' suffix on single precision literals is muscle memory for me after having had Steven Parker for my Ph.D. advisor.