Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of Fast-Math

simonbyrne.github.io

Post reply on HN
Print view View original

171–180 of 233 posts

Re: Beware of Fast-Math

#171

Post by osigurdson »

MobiusHorizons wrote: post #165
osigurdson wrote: post #134

Earlier quoted context omitted.

I meant fixed point decimal type (like C#) 128 bit. I don't understand why the parent commenter (top voted comment?) used unsigned integers to track individual cents. Why roll your own decimal type? Using arbitrary precision doesn't make sense if the data needs to be stored in a database (for most situations at least). Regardless, infinite precision is magical thinking anyway: try adding Pi to your bank account witho…

the C# decimal type is not fixed point, its a floating point implementation, but just uses a base 10 exponent instead of a base 2 one like IEE754 floats. Fixed point is a general technique that is commonly done with machine integers when the necessary precision is known at compile time. It is frequently used on embedded devices that don't have a floating point unit to avoid slow software based floating point implemen…

Good to know. In a scientific domain so haven't used it previously.

Re: Beware of Fast-Math

#172

Post by nulld3v »

chowells wrote: post #140
nulld3v wrote: post #116

Earlier quoted context omitted.

I'm curious where you got this idea from because it is trivially disprovable by typing 0.1 or 0.01 into any python or JS REPL?

Do you believe that the way the REPL prints a number is the way it's stored internally? If so, explaining this will be a fun exercise: $ python3 Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> a = 0.1 >>> a + a + a 0.30000000000000004 By way of explanation, the algorithm used to render a floating point number to text used in…

:facepalm: my bad, I completely missed the more rational intepretation of OP's comment...

I interpreted "directly representable" as "uniquely representable", all f64, though indeed this guarantee is lost once you perform any math.

Re: Beware of Fast-Math

#173

Post by jcranmer »

munch117 wrote: post #147
jcranmer wrote: post #129

Earlier quoted context omitted.

I've found fear of the use of floating-point in finance to be a good litmus test for how knowledgeable people are about floating-point. Because as far as I can tell, finance people almost exclusively uses (binary) floating-point [1], whereas a lot of floating-point FUD focuses on how disastrous it is for finance. And honestly, it's a bit baffling to me why so many people seem to think that floating-point is disastrou…

The answer is accounting. In accounting you want predictability and reproducibility more than anything, and you are prepared to throw away precision on that alter. If you're summing up the cost of items in a webshop, then you're in the domain of accounting. If the result appears to be off by a single cent because of a rounding subtlety, then you're in trouble, because even though no one should care about that single…

To get the right answers for accounting, all you have to do is pay attention to how you're doing rounding, which is no harder for floating-point than it is for fixed-point. Actually, it might be slightly easier for floating-point, since you're probably not as likely to skip over the part of the contract that tells you what the rounding rules you have to follow are.

Re: Beware of Fast-Math

#174

Post by pgwhalen »

munch117 wrote: post #152
pgwhalen wrote: post #146

Earlier quoted context omitted.

I agree overall but my take is that it shows more ignorance about the domain of finance (or a particular subdomain) than it does about floating-point ignorance. It’s really more of a concern in accounting, when monetary amounts are concrete and represent real money movement between distinct parties. A ton of financial software systems (HFT, trading in general) deal with money in a more abstract way in most of their c…

FP does not introduce imprecision. Quite the contrary: The continuous rounding (or truncation) triggered by using scaled integers is what introduces imprecision. Whereas exponent scaling in floating point ensures that all the bits in the mantissa are put to good use. It's a trade-off between precision and predictability. Floating point provides the former. Scaled integers provide the latter.

I was using imprecision in a more general and less mathematical sense than the way you’re interpreting it, but yes this is a good point about why FP is useful in many financial contexts, when the monetary amount is derived from some model.

Re: Beware of Fast-Math

#175

Post by munch117 »

jcranmer wrote: post #173
munch117 wrote: post #147

Earlier quoted context omitted.

The answer is accounting. In accounting you want predictability and reproducibility more than anything, and you are prepared to throw away precision on that alter. If you're summing up the cost of items in a webshop, then you're in the domain of accounting. If the result appears to be off by a single cent because of a rounding subtlety, then you're in trouble, because even though no one should care about that single…

To get the right answers for accounting, all you have to do is pay attention to how you're doing rounding, which is no harder for floating-point than it is for fixed-point. Actually, it might be slightly easier for floating-point, since you're probably not as likely to skip over the part of the contract that tells you what the rounding rules you have to follow are.

Agreed. To do accounting, you need to employ some kind of discipline to ensure that you get rounding right. So many people erroneously believe that such a discipline has to be based on fixed point or decimal floating point numbers. But binary floating point can work just fine.

Re: Beware of Fast-Math

#176

Post by dzaima »

emn13 wrote: post #31

I get the feeling that the real problem here are the IEEE specs themselves. They include a huge bunch of restrictions that each individually aren't relevant to something like 99.9% of floating point code, and probably even in aggregate not a single one is relevant to a large majority of code segments out in the wild. That doesn't mean they're not important - but some of these features should have been locally opt-in,…

The precise requirements of IEEE-754 may not be important for any given program, but as long as you want your numbers to have any form of well-defined semantics beyond "numbers exist, and here's a list of functions that do Something™ that may or may not be related to their name", any number format that's capable of (approximately) storing both 10^20 and 10^-20 in 64 bits is gonna have those drawbacks.

AFAIK GPU code is basically always written as scalar code acting on each "thing" separately, that's, as a whole, semantically looped over by the hardware, same way as multithreading would (i.e. no order guaranteed at all), so you physically cannot write code that'd need operation reordering to vectorize. You just can't write an equivalent to "for (each element in list) accumulator += element;" (or, well, you can, by writing that and running just one thread of it, but that's gonna be slower than even the non-vectorized CPU equivalent (assuming the driver respects IEEE-754)).

Re: Beware of Fast-Math

#177

Post by pasc1878 »

f33d5173 wrote: post #161
jksflkjl3jk3 wrote: post #113

Earlier quoted context omitted.

Floating point math shouldn't be that scary. The rules are well defined in standards, and for many domains are the only realistic option for performance reasons. I've spent most of my career writing trading systems that have executed 100's of billions of dollars worth of trades, and have never had any floating point related bugs. Using some kind of fixed point math would be entirely inappropriate for most HFT or scie…

It's the front of house/back of house distinction. Front of house should use fixed point, back of house should use floating point. Unless you're doing trading, you want really strict rules with regards to rounding and such, which are going to be easier to achieve with fixed point.

I don't think it is that clear. The split I think is between calculating settlement amounts which lead to real transfers of money and so should be fixed point whilst risk, pricing (thus trading) and valuation use models which need many calculations so need to be floating point.

Re: Beware of Fast-Math

#178

Post by kstrauser »

lo0dot0 wrote: post #68
goalieca wrote: post #57

Earlier quoted context omitted.

Floating point arithmetic is neither commutative or associative so you shouldn’t.

While it technically correct to say this it also gets the wrong point across because it leaves out the fact that ordering changes create only a small difference. Other examples where arithmetic is not commutative, e.g. matrix multiplication , can create much larger differences.

> ordering changes create only a small difference.

That can’t be assumed.

You can easily fall into a situation like:

  total = large_float_value
  for _ in range(1_000_000_000):
    total += .01
  assert total == large_float_value
Without knowing the specific situation, it’s impossible to say whether that’s a tolerably small difference.

Re: Beware of Fast-Math

#179

Post by leephillips »

This part was fascinating:

“The problem is how FTZ actually implemented on most hardware: it is not set per-instruction, but instead controlled by the floating point environment: more specifically, it is controlled by the floating point control register, which on most systems is set at the thread level: enabling FTZ will affect all other operations in the same thread.

“GCC with -funsafe-math-optimizations enables FTZ (and its close relation, denormals-are-zero, or DAZ), even when building shared libraries. That means simply loading a shared library can change the results in completely unrelated code, which is a fun debugging experience.”

Re: Beware of Fast-Math

#180

Post by ljosifov »

CamperBob2 wrote: post #169
usefulcat wrote: post #167

Earlier quoted context omitted.

It matters if the strategy is designed to do very different things depending on whether or not the offers are locked (when bid == ask, or spread is less than 0.01). In this example, I’m talking about securities that are priced in whole cents. If you represent prices as floats, then it’s possible that the spread appears to be less (or greater) than 0.01 when it’s actually not, due to the inability of floats to exactly…

But I'm still not understanding the real-world consequences. What will those be, exactly? Any good examples or case studies to look at?

I can imagine sth like: if (bid ask blah blah) { send order to buy 10 million of AAPL; }
Post reply on HN