Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of Fast-Math

simonbyrne.github.io

Post reply on HN
Print view View original

201–210 of 233 posts

Re: Beware of Fast-Math

#201

Post by foota »

pclmulqdq wrote: post #76
orlp wrote: post #4

I helped design an API for "algebraic operations" in Rust: https://github.com/rust-lang/rust/issues/136469 >, which are coming along nicely. These operations are 1. Localized, not a function-wide or program-wide flag. 2. Completely safe, -ffast-math includes assumptions such that there are no NaNs, and violating that is undefined behavior . So what do these algebraic operations do? Well, one by itself doesn't do much…

-ffast-math is actually something like 15 separate flags, and you can use them individually if you want. 3 of them are "no NaNs," "no infinities," and "no subnormals." Several of the other flags allow you to treat math as associative or distributive if you want that. The library has some merit, but the goal you've stated here is given to you with 5 compiler flags. The benefit of the library is choosing when these app…

There's probably a benefit to being able to choose which you want in different places though, right?

Re: Beware of Fast-Math

#202

Post by layer8 »

adgjlsfhk1 wrote: post #192
layer8 wrote: post #97

Earlier quoted context omitted.

It actually is commutative according to IEEE-754, except that in the case of a NaN result you might get a different NaN representation.

having multiple NaNs and no spec for how they should behave feels like such an unforced error to me

For mathematical use, NaN payloads shouldn’t matter, and behave identically (aside from quiet vs. signaling NaNs). It also doesn’t matter for equality comparison, because NaNs always compare unequal.

Re: Beware of Fast-Math

#203

Post by jmb99 »

hyghjiyhu wrote: post #104

One thing I wonder is what happens if you have an inline function in a header that is compiled with fast math by one translation unit and without in another.

I haven’t checked, but my assumption is that the output of each compilation unit will be different. The one definition rule doesn’t apply here (there’s still one definition), and there shouldn’t be a conflict if the functions are inlined in their respective compilation units so the linker shouldn’t complain.

Could be wrong but that’s my gut feeling.

Re: Beware of Fast-Math

#204

Post by alfiedotwtf »

adrian_b wrote: post #70
emn13 wrote: post #31

I get the feeling that the real problem here are the IEEE specs themselves. They include a huge bunch of restrictions that each individually aren't relevant to something like 99.9% of floating point code, and probably even in aggregate not a single one is relevant to a large majority of code segments out in the wild. That doesn't mean they're not important - but some of these features should have been locally opt-in,…

Not being able to auto-vectorize is not the fault of the IEEE standard, but the fault of those programming languages which do not have ways to express that the order of some operations is irrelevant, so they may be executed concurrently. Most popular programming languages have the defect that they impose a sequential semantics even where it is not needed. There have been programming languages without this defect, e.g…

> Most popular programming languages have the defect that they impose a sequential semantics even where it is not needed. There have been programming languages without this defect, e.g. Occam, but they have not become widespread.

We have memory ordering functions to let compilers know the atomic operation preference of the programmer… couldn’t we do the same for maths and in general a set of expressions?

Re: Beware of Fast-Math

#205

Post by jart »

smcameron wrote: post #82

One thing I did not see mentioned in the article, or in these comments (according to ctrl-f anyway) is the use of feenableexcept()[1] to track down the source of NaNs in your code. feenableexcept(FE_DIVBYZERO | FE_INVALID | FE_OVERFLOW); will cause your code to get a SIGFPE whenever a NaN crawls out from under a rock. Of course it doesn't work with fast-math enabled, but if you're unknowingly getting NaNs without fas…

Trapping math is the enlightened way to do things. I wrote an example in the cosmo repo of how to use it. https://github.com/jart/cosmopolitan/blob/master/examples/tr...

Re: Beware of Fast-Math

#206

Post by recursivecaveat »

DavidVoid wrote: post #200
cycomanic wrote: post #37

I think this article overstates the importance of the problems even for scientific software. In the scientific code I've written, noise processes are often orders of magnitude larger than what what is discussed here and I believe this applies to many (most?) simulations modelling the real world (i.e. Physics chemistry,..). At the same time enabling fast-math has often yielded a very significant (>10%) performance boo…

It matters for reproducibility between software versions, right? I work in audio software and we have some comparison tests that compare the audio output of a chain of audio effects with a previous result. If we make some small refactoring of the code and the compiler decides to re-organize the arithmetic operations then we might suddenly get a slightly different output. So of course we disable fast-math. One thing w…

Yeah that is the killer for me. I'm not particularly attached to IEEE semantics. Unfortunately the replacement is that your results can change between any two compiles, for nearly any reason. Even if you think you don't care about tiny precision variances: consider that if you ever score and rank things with an algorithm that involves floats, the resulting order can change.

Re: Beware of Fast-Math

#207

Post by kbolino »

kccqzy wrote: post #106
SonOfLilit wrote: post #103

Earlier quoted context omitted.

You can make money modeling buy/sell decisions in floats and then having the bank execute them, but if the bank models your account as a float and loses a cent here and there, it will be sued into bankruptcy.

You will not lose a cent here and there just by using float64, for the range of values that banks deal with. For added assurance, just round to the nearest cent after each operation.

You can round to the nearest .0078125 (1/128) but not to the nearest .01 (1/100) because that number cannot be represented in float64.

To round to the nearest cent, you would need to make cents your units (i.e. the quantity "1 dollar" would be represented as 100 instead of 1.0).

Re: Beware of Fast-Math

#208

Post by nurettin »

kbolino wrote: post #207
kccqzy wrote: post #106

Earlier quoted context omitted.

You will not lose a cent here and there just by using float64, for the range of values that banks deal with. For added assurance, just round to the nearest cent after each operation.

You can round to the nearest .0078125 (1/128) but not to the nearest .01 (1/100) because that number cannot be represented in float64. To round to the nearest cent, you would need to make cents your units (i.e. the quantity "1 dollar" would be represented as 100 instead of 1.0).

Within the general context of this discussion, "cannot be represented" is a red herring.

You don't need to have a representation of the exact number 0.1 if you can tolerate errors after the 7th decimal (and it turns out you can). And 0.1+0.1+0.1 does not have to be comparable with 0.3 using operator==. You have an is_close function for that. And accumulation is not an issue because you have rounding and fma for that.

Re: Beware of Fast-Math

#209

Post by eqvinox »

Measter wrote: post #183
Sharlin wrote: post #27

Earlier quoted context omitted.

Rust has some precedence for adding convenience newtypes with overloaded operators (eg. `Wrapping ´ for `I.wrapping_add(I)` etc). Such a wrapper isn't currently proposed AFAIK but there's no reason one couldn't be added in the future I believe.

For giggles, here's one I whipped up, along with an example use: https://godbolt.org/z/Eezj35dzc

Having done unspeakable things with the C preprocessor myself, this Rust macro soup truly warms my heart <3

Re: Beware of Fast-Math

#210

Post by adrian_b »

alfiedotwtf wrote: post #204
adrian_b wrote: post #70

Earlier quoted context omitted.

Not being able to auto-vectorize is not the fault of the IEEE standard, but the fault of those programming languages which do not have ways to express that the order of some operations is irrelevant, so they may be executed concurrently. Most popular programming languages have the defect that they impose a sequential semantics even where it is not needed. There have been programming languages without this defect, e.g…

> Most popular programming languages have the defect that they impose a sequential semantics even where it is not needed. There have been programming languages without this defect, e.g. Occam, but they have not become widespread. We have memory ordering functions to let compilers know the atomic operation preference of the programmer… couldn’t we do the same for maths and in general a set of expressions?

An example of programming language syntax that avoids to specify sequential execution where not needed is to specify that a sequence of expressions separated by semicolons must be executed sequentially, but a sequence of expressions separated by commas may be executed in any order or concurrently.

This is just a minor change from the syntax of the most popular programming languages, because they typically already specify that the order of evaluation of the expressions used for the arguments of a function, which are separated by commas, can be arbitrary.

Early in its history, the C language has been close to specifying this behavior for its comma operator, but unfortunately its designers have changed their mind and they have made the comma operator behave like a semicolon, in order to be able to use it inside for statement headers, where the semicolons have a different meaning. A much better solution for C, instead of making both comma and semicolon to have the same behavior, would have been to allow a block to appear in any place where an expression is expected, giving it the value of the last expression evaluated in the block.

Post reply on HN