Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of Fast-Math

simonbyrne.github.io

Post reply on HN
Print view View original

131–140 of 233 posts

Re: Beware of Fast-Math

#131

Post by BeetleB »

eddd-ddde wrote: post #115
jksflkjl3jk3 wrote: post #113

Earlier quoted context omitted.

Floating point math shouldn't be that scary. The rules are well defined in standards, and for many domains are the only realistic option for performance reasons. I've spent most of my career writing trading systems that have executed 100's of billions of dollars worth of trades, and have never had any floating point related bugs. Using some kind of fixed point math would be entirely inappropriate for most HFT or scie…

How do you handle the lack of commutativity? I've always wondered about the practical implications.

Nitpick: FP arithmetic is commutative. It's not associative.

Re: Beware of Fast-Math

#132

Post by jcranmer »

T0Bi wrote: post #130
jksflkjl3jk3 wrote: post #113

Earlier quoted context omitted.

Floating point math shouldn't be that scary. The rules are well defined in standards, and for many domains are the only realistic option for performance reasons. I've spent most of my career writing trading systems that have executed 100's of billions of dollars worth of trades, and have never had any floating point related bugs. Using some kind of fixed point math would be entirely inappropriate for most HFT or scie…

> Using some kind of fixed point math would be entirely inappropriate for most HFT or scientific computing applications. May I ask why? (generally curious)

For starters, it's giving up a lot of performance, since fixed-point isn't accelerated by hardware like floating-point is.

Re: Beware of Fast-Math

#133

Post by klysm »

simonw wrote: post #80
razighter777 wrote: post #65

The worst thing that strikes fear into me is seeing floating points used for real world currency. Dear god. So many things can go wrong. I always use unsigned integers counting number of cents. And if I gotta handle multiple currencies, then I'll use or make a wrapper class.

I've been having an interesting challenge relating to this recently. I'm trying to calculate costs for LLM usage, but the amounts of money involved are so tiny . Gemini 1.5 Flash 8B is $0.0375 per million tokens! Should I be running my accounting system on units of 10 billionths of a dollar?

Convert to money as late as possible

Re: Beware of Fast-Math

#134

Post by osigurdson »

MobiusHorizons wrote: post #105
osigurdson wrote: post #81

Earlier quoted context omitted.

Wouldn't it be better to use a decimal type?

This is what’s called a fixed point decimal type. If you need variable precision, then a decimal type might be a good idea, but fixed point removes a lot of potential foot guns if the constraints work for you.

I meant fixed point decimal type (like C#) 128 bit. I don't understand why the parent commenter (top voted comment?) used unsigned integers to track individual cents. Why roll your own decimal type?

Using arbitrary precision doesn't make sense if the data needs to be stored in a database (for most situations at least). Regardless, infinite precision is magical thinking anyway: try adding Pi to your bank account without loss of precision.

Re: Beware of Fast-Math

#135

Post by sfn42 »

londons_explore wrote: post #43
cycomanic wrote: post #37

I think this article overstates the importance of the problems even for scientific software. In the scientific code I've written, noise processes are often orders of magnitude larger than what what is discussed here and I believe this applies to many (most?) simulations modelling the real world (i.e. Physics chemistry,..). At the same time enabling fast-math has often yielded a very significant (>10%) performance boo…

It would be nice if there was some syntax for "math order matters, this is the order I want it done in". Then all other math will be fast-math, except where annotated.

I thought most languages have this? If you simply write a formula operations are ordered according to the language specifiction. If you want different ordering you use parentheses.

Not sure how that interacts with this fast math thing, I don't use C

Re: Beware of Fast-Math

#136

Post by djrj477dhsnv »

kolbe wrote: post #119
jksflkjl3jk3 wrote: post #113

Earlier quoted context omitted.

Floating point math shouldn't be that scary. The rules are well defined in standards, and for many domains are the only realistic option for performance reasons. I've spent most of my career writing trading systems that have executed 100's of billions of dollars worth of trades, and have never had any floating point related bugs. Using some kind of fixed point math would be entirely inappropriate for most HFT or scie…

All your price field messages are sent to the exchange and back via fixed point, so you are using fixed point for at least some of the process (unless you're targeting those few crypto exchanges that use fp prices). If you need to be extremely fast (like fpga fast), you don't waste compute transforming their fixed point representation into floating.

Sure, string encodings are used for most APIs and ultra HFT may pattern match on the raw bytes, but for regular HFT if you're doing much math, it's going to be floating point math.

Re: Beware of Fast-Math

#137

Post by gamescr »

usefulcat wrote: post #122
jksflkjl3jk3 wrote: post #113

Earlier quoted context omitted.

Floating point math shouldn't be that scary. The rules are well defined in standards, and for many domains are the only realistic option for performance reasons. I've spent most of my career writing trading systems that have executed 100's of billions of dollars worth of trades, and have never had any floating point related bugs. Using some kind of fixed point math would be entirely inappropriate for most HFT or scie…

You can certainly make trading systems that work using floating point, but there are just so many fewer edge cases to consider when using fixed point. With fixed point and at least 2 decimal places, 10.01 + 0.01 is always exactly equal to 10.02. But with FP you may end up with something like 10.0199999999, and then you have to be extra careful anywhere you convert that to a string that it doesn't get truncated to 10.…

I work on game engines and the problem with floats isn't on small values like 10.01 but on large ones like 400,010.01 that's when the precision wildly varies.

Re: Beware of Fast-Math

#138

Post by usefulcat »

phendrenad2 wrote: post #128
jksflkjl3jk3 wrote: post #113

Earlier quoted context omitted.

Floating point math shouldn't be that scary. The rules are well defined in standards, and for many domains are the only realistic option for performance reasons. I've spent most of my career writing trading systems that have executed 100's of billions of dollars worth of trades, and have never had any floating point related bugs. Using some kind of fixed point math would be entirely inappropriate for most HFT or scie…

I'm wondering if trading systems would run into the same issues as a bank or scientific calculation. You might not be making as many repeated calculations, and might not care if things are "off" by a tiny amount, because you're trading between money and securities, and the "loss" is part of your overhead. If a bank lost $0.01 after every 1 million transactions it would be a minor scandal.

Personally, I would be more concerned about something like determining whether the spread is more than a penny. Something like:

    if (ask - bid > 0.01) {
        // etc
    }
With floating point, I have to think about the following questions: * What if the constant 0.01 is actually slightly greater than mathematical 0.01? * What if the constant 0.01 is actually slightly less than mathematical 0.01? * What if ask - bid is actually slightly greater than the mathematical result? * What if ask - bid is actually slightly less than the mathematical result?

With floating point, that seemingly obvious code is anything but. With fixed point, you have none of those problems.

Granted, this only works for things that are priced in specific denominations (typically hundredths, thousandths, or ten thousandths), which is most securities.

Re: Beware of Fast-Math

#140

Post by chowells »

nulld3v wrote: post #116
dataangel wrote: post #91

Earlier quoted context omitted.

you can't accurately represent 10 cents with floats, 0.1 is not directly representable. same with 1 cent, 0.01. Seems like if you do and significant math on prices you should run into rounding issues pretty quickly?

I'm curious where you got this idea from because it is trivially disprovable by typing 0.1 or 0.01 into any python or JS REPL?

Do you believe that the way the REPL prints a number is the way it's stored internally? If so, explaining this will be a fun exercise:

    $ python3
    Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> a = 0.1
    >>> a + a + a
    0.30000000000000004
By way of explanation, the algorithm used to render a floating point number to text used in most languages these days is to find the shortest string representation that will parse back to an identical bit pattern. This has the direct effect of causing a REPL to print what you typed in. (Well, within certain ranges of "reasonable" inputs.) But this doesn't mean that the language stores what you typed in - just an approximation of it.
Post reply on HN