Earlier quoted context omitted.
don't forget libiberty which is linked in using -liberty (and freedom for all)
strangely I'm not aware of a libibre.
oh, that's called -lmojito
Std: Clamp generates less efficient assembly than std:min(max,std:max(min,v))
121–130 of 152 posts
Re: Std: Clamp generates less efficient assembly than std:min(max,std:max(min,v))
#122Earlier quoted context omitted.
Compilers often under-generate conditional instructions. They implicitly assume (correctly) that most branches you write are 90/10 (ie very predictable), not 50/50. The branches that actually are 50/50 suffer from being treated as being 90/10.
It's hard to predict statically which branches will be dynamically unpredictable. A seasoned hardware architect once told me that Intel went all-in on predication for Itanium, under the assumption that a Sufficiently Smart Compiler could figure it out, and then discovered to their horror that their compiler team's best efforts were not Sufficiently Smart. He implied that this was why Intel pushed to get a profile-gui…
The compiler doesn't do much of the predicting, it's done by the CPU in runtime.
Re: Std: Clamp generates less efficient assembly than std:min(max,std:max(min,v))
#123Earlier quoted context omitted.
Why do you say almost never? Don’t let the name scare you; all floating point math is inaccurate. Fast math is only slightly less accurate, I think typically it’s a 1 or maybe 2 LSB difference. At least in CUDA it is, and I think many (most?) people & situations can tolerate 22 bits of mantissa compared to 23, and many (most?) people/situations aren’t paying attention to inf/nan/exception issues at all. I deal with a…
Nah, you don't deal with floats. You do machine learning which just happens to use floats. I do both numerical computing and machine learning. And oh boy are you wrong! People who deal with actual numerical computing know that the statement "fast math is only slightly less accurate" is absurd. Fast math is unbounded in its inaccuracy! It can reorder your computations so that something that used to sum to 1 now sums t…
I only do numeric computation, I don’t work in machine learning. Sorry your assumptions are incorrect, maybe it’s best not to assume or attack. I didn’t exactly advise using fast math either, I asked for reasoning and pointed out that most casual uses of float aren’t highly sensitive to precision.
It’s easy to have wrong sums and catastrophic cancellation without fast math, and it’s relatively rare for fast math to cause those issues when an underlying issue didn’t already exist.
I’ve been working in some code that does a couple of quadratic solves and has high order intermediate terms, and I’ve tried using Kahan’s algorithm repeatedly to improve the precision of the discriminants, but it has never helped at all. On the other hand I’ve used a few other tricks that improve the precision enough that the fast math version is higher precision than the naive one without fast math. I get to have my cake and eat it too.
Fast math is a tradeoff. Of course it’s a good idea to know what it does and what the risks of using it are, but at least in terms of the accuracy of fast math in CUDA, it’s not an opinion whether the accuracy is relatively close to slow math, it’s reasonably well documented. You can see for yourself that most fast math ops are in the single digit ulps of rounding error. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index....
Re: Std: Clamp generates less efficient assembly than std:min(max,std:max(min,v))
#124Earlier quoted context omitted.
From §25.8.9 Bounded value [alg.clamp]: > 2 Preconditions: `bool(comp(proj(hi), proj(lo)))` is false. For the first form, type `T` meets the Cpp17LessThanComparable requirements (Table 26). > 3 Returns: `lo` if `bool(comp(proj(v), proj(lo)))` is true, `hi` if `bool(comp(proj(hi), proj(v)))` is true, otherwise `v`. > 4 [Note: If NaN is avoided, `T` can be a floating-point type. — end note] From Table 26: > `
Does that mean NaN is undefined behavior for clamp?
My interpretation is that yes, passing NaN is undefined behavior. Strict weak ordering is defined in 25.8 Sorting and related operations [alg.sorting]:
> 4 The term strict refers to the requirement of an irreflexive relation (`!comp(x, x)` for all `x`), and the term weak to requirements that are not as strong as those for a total ordering, but stronger than those for a partial ordering. If we define `equiv(a, b)` as `!comp(a, b) && !comp(b, a)`, then the requirements are that `comp` and `equiv` both be transitive relations:
> 4.1 `comp(a, b) && comp(b, c)` implies `comp(a, c)`
> 4.2 `equiv(a, b) && equiv(b, c)` implies `equiv(a, c)`
NaN breaks these relations, because `equiv(42.0, NaN)` and `equiv(NaN, 3.14)` are both true, which would imply `equiv(42.0, 3.14)` is also true. But clearly that's not true, so floating point numbers do not satisfy the strict weak ordering requirement.
The standard doesn't explicitly say that NaN is undefined behavior. But it does not define the behavior for when NaN is used with `std::clamp()`, which I think by definition means it's undefined behavior.
Re: Std: Clamp generates less efficient assembly than std:min(max,std:max(min,v))
#125Earlier quoted context omitted.
Nah, you don't deal with floats. You do machine learning which just happens to use floats. I do both numerical computing and machine learning. And oh boy are you wrong! People who deal with actual numerical computing know that the statement "fast math is only slightly less accurate" is absurd. Fast math is unbounded in its inaccuracy! It can reorder your computations so that something that used to sum to 1 now sums t…
+1. I'm years away from fp-analysis, but do the transcendental expansions even converge in the presence of fast-math? No `sin()`, no `cos()`, no `exp()`, ...
Well there are library implementations of fast-math trancendentals that offer bounded error, and a million different fast sine approximation algorithms, so, yes? This is why you shouldn’t listen to FUD. The corner cases are indeed frustrating for a few people, but most never hit them, and the world doesn’t suddenly break when fast math is enabled. I am paid to do some FP analysis, btw.
Re: Std: Clamp generates less efficient assembly than std:min(max,std:max(min,v))
#126Earlier quoted context omitted.
It's hard to predict statically which branches will be dynamically unpredictable. A seasoned hardware architect once told me that Intel went all-in on predication for Itanium, under the assumption that a Sufficiently Smart Compiler could figure it out, and then discovered to their horror that their compiler team's best efforts were not Sufficiently Smart. He implied that this was why Intel pushed to get a profile-gui…
The compiler doesn't do much of the predicting, it's done by the CPU in runtime.
Not prediction, predication: https://en.wikipedia.org/wiki/Predication_(computer_architec...
By avoiding conditional branches and essentially masking out some instructions, you can avoid stalls and mis-predictions and keep the pipeline full.
Actually I think @IainIreland mis-remembers what the seasoned architect told him about Itanium. While Itanium did support predicated instructions, the problematic static scheduling was actually because Itanium was a VLIW machine: https://en.wikipedia.org/wiki/VLIW .
TL;DR: dynamic scheduling on superscalar out-of-order processors with vector units works great and the transistor overhead got increasingly cheap, but static scheduling stayed really hard.
Re: Std: Clamp generates less efficient assembly than std:min(max,std:max(min,v))
#127Earlier quoted context omitted.
> I’d have to agree that disabling denorms globally is pretty bad, and yet, for audio processing, this is an option that most DAWs either implement silently, or offer users the choice, because denormals are inevitable in reverb tails and on most Intel processors they slow things by orders of magnitude.
I would think for audio, there’s no audible difference between a denorm and a flushed zero. Are there cases where denorms are important to audio?
They are important in the negative sense: Intel processors are appalling at handling them, and they can break realtime code because of this.
My DAW uses both "denormals are zero" and "flush denormals to zero" to try to avoid them; it also offers a "DC Bias" option where extremely small values are added to samples to avoid denormals.
Re: Std: Clamp generates less efficient assembly than std:min(max,std:max(min,v))
#128Earlier quoted context omitted.
> The results of your program will vary depending on the exact make of your compiler and other random attributes of your compile environment, which can wreak havoc if you have code that absolutely wants bit-identical results. This doesn't matter for everybody, but there are some domains where this can be a non-starter (e.g., multiplayer game code). This already shouldn't be assumed, because even the same code, compil…
> Often this comes down to acceptable implementation defined behavior, I believe this is "always" rather than often when it comes to the actual operations defined by the FP standard. gcc does play it fast and loose (as -ffast-math is not yet enabled by default, and FMA on the other hand is), but this is technically illegal and at least can be easily configured to be in standards-compliant mode. I think the bigger pro…
I'm surprised by this, regarding sqrt. The standard stipulates correct rounding for simple arithmetic, including sqrt ever since 754 1985.
Unless of course we are talking about the 80 bit format.
If that's not the case, would be interested to know where they differ.
Unfortunately for the transcendental function the accuracy still hasn't been pinned down, especially since that's still an ongoing research problem.
There's been some great strides in figuring out the worst cases for binary floating point up to doubles so hopefully an upcoming standard will stipulate 0.5 ULP for transcendentals. But decimal floating point still has a long way to go.
Re: Std: Clamp generates less efficient assembly than std:min(max,std:max(min,v))
#129Earlier quoted context omitted.
Thank you, great points. I’d have to agree that disabling denorms globally is pretty bad, even if (or maybe especially if) caring about denorms is rare. > Fast-math can cause hard range guarantees to fail. Maybe you’ve got code that you can prove that, even with rounding error, the result will still be >= 0. Floats do this too, it’s pretty routine to bump into epsilon out-of-range issues without fast-math. Most peopl…
> I’d have to agree that disabling denorms globally is pretty bad, and yet, for audio processing, this is an option that most DAWs either implement silently, or offer users the choice, because denormals are inevitable in reverb tails and on most Intel processors they slow things by orders of magnitude.
For game development we had them off as well because of performance issues. Most stuff calculates around 0 so it was pretty common to trigger denorms.
The slowing down on Intel platforms has always frustrated me because denorms provide nice smoothing around 0.
At the same time it was nice only having to consider normal floating point when trying to get more accuracy out of calculations, etc.
Re: Std: Clamp generates less efficient assembly than std:min(max,std:max(min,v))
#130Earlier quoted context omitted.
> Often this comes down to acceptable implementation defined behavior, I believe this is "always" rather than often when it comes to the actual operations defined by the FP standard. gcc does play it fast and loose (as -ffast-math is not yet enabled by default, and FMA on the other hand is), but this is technically illegal and at least can be easily configured to be in standards-compliant mode. I think the bigger pro…
I'm surprised by this, regarding sqrt. The standard stipulates correct rounding for simple arithmetic, including sqrt ever since 754 1985. Unless of course we are talking about the 80 bit format. If that's not the case, would be interested to know where they differ. Unfortunately for the transcendental function the accuracy still hasn't been pinned down, especially since that's still an ongoing research problem. Ther…
Because compilers can and have implemented sqrt in terms of rsqrt which is .. fun to work with. This also on SSE.