Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of Fast-Math

simonbyrne.github.io

Post reply on HN
Print view View original

161–170 of 233 posts

Re: Beware of Fast-Math

#161

Post by f33d5173 »

jksflkjl3jk3 wrote: post #113
razighter777 wrote: post #65

The worst thing that strikes fear into me is seeing floating points used for real world currency. Dear god. So many things can go wrong. I always use unsigned integers counting number of cents. And if I gotta handle multiple currencies, then I'll use or make a wrapper class.

Floating point math shouldn't be that scary. The rules are well defined in standards, and for many domains are the only realistic option for performance reasons. I've spent most of my career writing trading systems that have executed 100's of billions of dollars worth of trades, and have never had any floating point related bugs. Using some kind of fixed point math would be entirely inappropriate for most HFT or scie…

It's the front of house/back of house distinction. Front of house should use fixed point, back of house should use floating point. Unless you're doing trading, you want really strict rules with regards to rounding and such, which are going to be easier to achieve with fixed point.

Re: Beware of Fast-Math

#162

Post by osigurdson »

T0Bi wrote: post #130
jksflkjl3jk3 wrote: post #113

Earlier quoted context omitted.

Floating point math shouldn't be that scary. The rules are well defined in standards, and for many domains are the only realistic option for performance reasons. I've spent most of my career writing trading systems that have executed 100's of billions of dollars worth of trades, and have never had any floating point related bugs. Using some kind of fixed point math would be entirely inappropriate for most HFT or scie…

> Using some kind of fixed point math would be entirely inappropriate for most HFT or scientific computing applications. May I ask why? (generally curious)

Fundamentally there is uncertainty associated with any physical measurement which is usually proportional to the magnitude being measured. As long as floating point is << this uncertainty results are equally predictive. Floating point numbers bake these assumptions in.

Re: Beware of Fast-Math

#163

Post by jjmarr »

MobiusHorizons wrote: post #105
osigurdson wrote: post #81

Earlier quoted context omitted.

Wouldn't it be better to use a decimal type?

This is what’s called a fixed point decimal type. If you need variable precision, then a decimal type might be a good idea, but fixed point removes a lot of potential foot guns if the constraints work for you.

IEEE754 defines a floating point decimal type. What are your opinions on that?

Re: Beware of Fast-Math

#164

Post by anthk »

Athas wrote: post #157
T0Bi wrote: post #130

Earlier quoted context omitted.

> Using some kind of fixed point math would be entirely inappropriate for most HFT or scientific computing applications. May I ask why? (generally curious)

The problem with fixed point is in its, well, fixed point. You assign a fixed number of bits to the fractional part of the number. This gives you the same absolute precision everywhere, but the relative precision (distance to the next highest or lowest number) is worse for small numbers - which is a problem, because those tend to be pretty important. It's just overall a less efficient use of the bit encoding space (n…

With fixed point you either scale it up or use rationals.

Re: Beware of Fast-Math

#165

Post by MobiusHorizons »

osigurdson wrote: post #134
MobiusHorizons wrote: post #105

Earlier quoted context omitted.

This is what’s called a fixed point decimal type. If you need variable precision, then a decimal type might be a good idea, but fixed point removes a lot of potential foot guns if the constraints work for you.

I meant fixed point decimal type (like C#) 128 bit. I don't understand why the parent commenter (top voted comment?) used unsigned integers to track individual cents. Why roll your own decimal type? Using arbitrary precision doesn't make sense if the data needs to be stored in a database (for most situations at least). Regardless, infinite precision is magical thinking anyway: try adding Pi to your bank account witho…

the C# decimal type is not fixed point, its a floating point implementation, but just uses a base 10 exponent instead of a base 2 one like IEE754 floats.

Fixed point is a general technique that is commonly done with machine integers when the necessary precision is known at compile time. It is frequently used on embedded devices that don't have a floating point unit to avoid slow software based floating point implementations. Limiting the precision to $0.01 makes sense if you only do addition or subtraction. Precision of $0.001 (Tenths of a cent also called mils) may be necessary when calculating taxes or applying other percentages although this is typically called out in the relevant laws or regulations.

Re: Beware of Fast-Math

#166

Post by anthk »

outurnate wrote: post #84
simonw wrote: post #80

Earlier quoted context omitted.

I've been having an interesting challenge relating to this recently. I'm trying to calculate costs for LLM usage, but the amounts of money involved are so tiny . Gemini 1.5 Flash 8B is $0.0375 per million tokens! Should I be running my accounting system on units of 10 billionths of a dollar?

You're better off representing values as rationals; a ratio between two different numbers. For example, 0.0375 would be represented as 375 over 10000, or 3 over 80

From Forth, here's how I'd set the rationals:

    : gcd begin dup while tuck mod repeat drop ;
    : lcm 2dup \* abs -rot gcd / ;
    : reduce 2dup gcd tuck / >r / r> ;
    : q+ rot 2dup \* >r rot \* -rot \* + r> reduce ;
    : q- swap negate swap q+ ;
    : q\* rot \* >r  \* r> reduce ;
    : q/ >r \* swap r> \* swap reduce ;
Example: to compute 70 * 0.25 = 35/2

70 1 1 4 q* reduce .s 35 2 ok

On stack managing words like 2dup, rot and such, these are easily grasped under either Google/DDG or any Forth with the words "see" and/or "help".

as a hint, q- swaps the top two numbers in the stack, (which compose a rational), makes the last one negative and then turns back its position. And then it calls q+.

So, 2/5 - 3/2 = 2/5 + -3/2.

Re: Beware of Fast-Math

#167

Post by usefulcat »

CamperBob2 wrote: post #151
usefulcat wrote: post #138

Earlier quoted context omitted.

Personally, I would be more concerned about something like determining whether the spread is more than a penny. Something like: if (ask - bid > 0.01) { // etc } With floating point, I have to think about the following questions: * What if the constant 0.01 is actually slightly greater than mathematical 0.01? * What if the constant 0.01 is actually slightly less than mathematical 0.01? * What if ask - bid is actually…

So the spread is 0.0099999 instead of 0.01. When will that difference matter?

It matters if the strategy is designed to do very different things depending on whether or not the offers are locked (when bid == ask, or spread is less than 0.01).

In this example, I’m talking about securities that are priced in whole cents. If you represent prices as floats, then it’s possible that the spread appears to be less (or greater) than 0.01 when it’s actually not, due to the inability of floats to exactly represent most real numbers.

Re: Beware of Fast-Math

#168

Post by anthk »

chowells wrote: post #140
nulld3v wrote: post #116

Earlier quoted context omitted.

I'm curious where you got this idea from because it is trivially disprovable by typing 0.1 or 0.01 into any python or JS REPL?

Do you believe that the way the REPL prints a number is the way it's stored internally? If so, explaining this will be a fun exercise: $ python3 Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> a = 0.1 >>> a + a + a 0.30000000000000004 By way of explanation, the algorithm used to render a floating point number to text used in…

Oddly, tcl prints 0.30000000000000004 while jimtcl prints 0.3, while with 1/7 both crap out and round it to a simple 0.

Edit: Now it does it fine after inputting floats:

puts [ expr { 1.0/7.0 } ]

Eforth on top of Subleq, a very small and dumb virtual machine:

     1 f 7 f f/ f.
     0.143 ok
Still, using rationals where possible (and mod operations otherwise) gives a great 'precision', except for irrationals.

Re: Beware of Fast-Math

#169

Post by CamperBob2 »

usefulcat wrote: post #167
CamperBob2 wrote: post #151

Earlier quoted context omitted.

So the spread is 0.0099999 instead of 0.01. When will that difference matter?

It matters if the strategy is designed to do very different things depending on whether or not the offers are locked (when bid == ask, or spread is less than 0.01). In this example, I’m talking about securities that are priced in whole cents. If you represent prices as floats, then it’s possible that the spread appears to be less (or greater) than 0.01 when it’s actually not, due to the inability of floats to exactly…

But I'm still not understanding the real-world consequences. What will those be, exactly? Any good examples or case studies to look at?

Re: Beware of Fast-Math

#170

Post by emn13 »

adrian_b wrote: post #70
emn13 wrote: post #31

I get the feeling that the real problem here are the IEEE specs themselves. They include a huge bunch of restrictions that each individually aren't relevant to something like 99.9% of floating point code, and probably even in aggregate not a single one is relevant to a large majority of code segments out in the wild. That doesn't mean they're not important - but some of these features should have been locally opt-in,…

Not being able to auto-vectorize is not the fault of the IEEE standard, but the fault of those programming languages which do not have ways to express that the order of some operations is irrelevant, so they may be executed concurrently. Most popular programming languages have the defect that they impose a sequential semantics even where it is not needed. There have been programming languages without this defect, e.g…

If you care about absolute accuracy, I'm skeptical you want floats at all. I'm sure it depends on the use case.

Whether it's the standards fault or the languages fault for following the standard in terms of preventing auto-vectorization is splitting hairs; the whole point of the standard is to have predictable and usually fairly low-error ways of performing these operations, which only works when the order of operations is defined. That very aim is the problem; to the extent the stardard is harmless when ordering guarrantees don't exist you're essentially applying some of those tricky -ffast-math suboptimizations.

But to be clear in any case: there are obviously cases whereby order-of-operations is relevant enough and accuracy altering reorderings are not valid. It's just that those are rare enough that for many of these features I'd much prefer that to be the opt-in behavior, not opt-out. There's absolutely nothing wrong with having a classic IEEE 754 mode and I expect it's an essentialy feature in some niche corner cases.

However, given the obviously huge application of massively parallel processors and algorithms that accept rounding errors (or sometimes conversely overly precise results!), clearly most software is willing to generally accept rounding errors to be able to run efficiently on modern chips. It just so happens that none of the computer languages that rely on mapping floats to IEEE 754 floats in a straitforward fashion are any good at that, which is seems like its a bad trade off.

There could be multiple types of floats instead; or code-local flags that delineate special sections that need precise ordering; or perhaps even expressions that clarify how much error the user is willing to accept and then just let the compiler do some but not all transformations; and perhaps even other solutions.

Post reply on HN