111–120 of 152 posts
Earlier quoted context omitted.
What you really should enable is the fun and safe math optimizations, with -funsafe-math-optimizations.
I know almost nothing about compiler flags but I got a laugh out of this even though I still don't know if you're joking or not. Edit: Just read it again and now I understand the joke. Haha
Earlier quoted context omitted.
Why do you say almost never? Don’t let the name scare you; all floating point math is inaccurate. Fast math is only slightly less accurate, I think typically it’s a 1 or maybe 2 LSB difference. At least in CUDA it is, and I think many (most?) people & situations can tolerate 22 bits of mantissa compared to 23, and many (most?) people/situations aren’t paying attention to inf/nan/exception issues at all. I deal with a…
Nah, you don't deal with floats. You do machine learning which just happens to use floats. I do both numerical computing and machine learning. And oh boy are you wrong! People who deal with actual numerical computing know that the statement "fast math is only slightly less accurate" is absurd. Fast math is unbounded in its inaccuracy! It can reorder your computations so that something that used to sum to 1 now sums t…
Yes, and it could very well be that the correct answer is actually 0 and not 1.
Unless you write your code to explicitly account for fp associativity effects, in which case you don't need generic forum advice about fast-math.
Earlier quoted context omitted.
Why do you say almost never? Don’t let the name scare you; all floating point math is inaccurate. Fast math is only slightly less accurate, I think typically it’s a 1 or maybe 2 LSB difference. At least in CUDA it is, and I think many (most?) people & situations can tolerate 22 bits of mantissa compared to 23, and many (most?) people/situations aren’t paying attention to inf/nan/exception issues at all. I deal with a…
Nah, you don't deal with floats. You do machine learning which just happens to use floats. I do both numerical computing and machine learning. And oh boy are you wrong! People who deal with actual numerical computing know that the statement "fast math is only slightly less accurate" is absurd. Fast math is unbounded in its inaccuracy! It can reorder your computations so that something that used to sum to 1 now sums t…
Earlier quoted context omitted.
That was what Reagan said about the Soviet Union, not what was said in the Soviet Union. Correct me if I'm wrong.
https://en.wikipedia.org/wiki/Trust,_but_verify > Trust, but verify (Russian: доверяй, но проверяй, tr. doveryay, no proveryay, IPA: [dəvʲɪˈrʲæj no prəvʲɪˈrʲæj]) is a Russian proverb, which is rhyming in Russian. The phrase became internationally known in English after Suzanne Massie, a scholar of Russian history, taught it to Ronald Reagan, then president of the United States, the latter of whom used it on several o…
Earlier quoted context omitted.
> The results of your program will vary depending on the exact make of your compiler and other random attributes of your compile environment, which can wreak havoc if you have code that absolutely wants bit-identical results. This doesn't matter for everybody, but there are some domains where this can be a non-starter (e.g., multiplayer game code). This already shouldn't be assumed, because even the same code, compil…
This advice is out-of-date. All CPU hardware nowadays conforms to IEEE 754 semantics for binary32 and binary64. (I think all the GPUs now have non-denormal-flushing modes, but my GPU knowledge is less deep). All compilers will have a floating-point mode that preserves IEEE 754 semantics assuming that FP exceptions are unobservable and rounding mode is the default, and this is usually the default (icc/icx is unusual i…
Earlier quoted context omitted.
> The results of your program will vary depending on the exact make of your compiler and other random attributes of your compile environment, which can wreak havoc if you have code that absolutely wants bit-identical results. This doesn't matter for everybody, but there are some domains where this can be a non-starter (e.g., multiplayer game code). This already shouldn't be assumed, because even the same code, compil…
This advice is out-of-date. All CPU hardware nowadays conforms to IEEE 754 semantics for binary32 and binary64. (I think all the GPUs now have non-denormal-flushing modes, but my GPU knowledge is less deep). All compilers will have a floating-point mode that preserves IEEE 754 semantics assuming that FP exceptions are unobservable and rounding mode is the default, and this is usually the default (icc/icx is unusual i…
No, it's not. gcc itself still defaults to fp-contract=fast. Or at least does in all versions I have ever tried.
Earlier quoted context omitted.
-Ofast is one of those dangerous flags that you should probably be careful with. It is “contagious” and it can mess up code elsewhere in the program, because it changes processor flags. I would try a more specific flag like -ffinite-math-only.
finite-math-only is a footgun as well as it allows the compiler assume that NaNs do not exist. Which means all `isnan()` calls are just reduced to `false` so it’s difficult to program defensively. And if a NaN in fact occurs it’s naturally a one-way ticket to UB land.
Earlier quoted context omitted.
I know almost nothing about compiler flags but I got a laugh out of this even though I still don't know if you're joking or not. Edit: Just read it again and now I understand the joke. Haha
don't forget libiberty which is linked in using -liberty (and freedom for all)
Earlier quoted context omitted.
Thanks for sharing. I don't know if the C++ standard mandates one behavior or another, it really depends on how you want clamp to behave if the value is NaN. std::clamp returns NaN, while the reverse order returns the min value.
From §25.8.9 Bounded value [alg.clamp]: > 2 Preconditions: `bool(comp(proj(hi), proj(lo)))` is false. For the first form, type `T` meets the Cpp17LessThanComparable requirements (Table 26). > 3 Returns: `lo` if `bool(comp(proj(v), proj(lo)))` is true, `hi` if `bool(comp(proj(hi), proj(v)))` is true, otherwise `v`. > 4 [Note: If NaN is avoided, `T` can be a floating-point type. — end note] From Table 26: > `
If you benchmark these, you'll likely find the version with the jump edges out the one with the conditional instruction in practice.
Compilers often under-generate conditional instructions. They implicitly assume (correctly) that most branches you write are 90/10 (ie very predictable), not 50/50. The branches that actually are 50/50 suffer from being treated as being 90/10.
A seasoned hardware architect once told me that Intel went all-in on predication for Itanium, under the assumption that a Sufficiently Smart Compiler could figure it out, and then discovered to their horror that their compiler team's best efforts were not Sufficiently Smart. He implied that this was why Intel pushed to get a profile-guided optimization step added to the SPEC CPU benchmark, since profiling was the only way to get sufficiently accurate data.
I've never gone back to see whether the timeline checks out, but it's a good story.