Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of Fast-Math

simonbyrne.github.io

Post reply on HN
Print view View original

151–160 of 233 posts

Re: Beware of Fast-Math

#151

Post by CamperBob2 »

usefulcat wrote: post #138
phendrenad2 wrote: post #128

Earlier quoted context omitted.

I'm wondering if trading systems would run into the same issues as a bank or scientific calculation. You might not be making as many repeated calculations, and might not care if things are "off" by a tiny amount, because you're trading between money and securities, and the "loss" is part of your overhead. If a bank lost $0.01 after every 1 million transactions it would be a minor scandal.

Personally, I would be more concerned about something like determining whether the spread is more than a penny. Something like: if (ask - bid > 0.01) { // etc } With floating point, I have to think about the following questions: * What if the constant 0.01 is actually slightly greater than mathematical 0.01? * What if the constant 0.01 is actually slightly less than mathematical 0.01? * What if ask - bid is actually…

So the spread is 0.0099999 instead of 0.01. When will that difference matter?

Re: Beware of Fast-Math

#152

Post by munch117 »

pgwhalen wrote: post #146
jcranmer wrote: post #129

Earlier quoted context omitted.

I've found fear of the use of floating-point in finance to be a good litmus test for how knowledgeable people are about floating-point. Because as far as I can tell, finance people almost exclusively uses (binary) floating-point [1], whereas a lot of floating-point FUD focuses on how disastrous it is for finance. And honestly, it's a bit baffling to me why so many people seem to think that floating-point is disastrou…

I agree overall but my take is that it shows more ignorance about the domain of finance (or a particular subdomain) than it does about floating-point ignorance. It’s really more of a concern in accounting, when monetary amounts are concrete and represent real money movement between distinct parties. A ton of financial software systems (HFT, trading in general) deal with money in a more abstract way in most of their c…

FP does not introduce imprecision. Quite the contrary: The continuous rounding (or truncation) triggered by using scaled integers is what introduces imprecision. Whereas exponent scaling in floating point ensures that all the bits in the mantissa are put to good use.

It's a trade-off between precision and predictability. Floating point provides the former. Scaled integers provide the latter.

Re: Beware of Fast-Math

#153

Post by CamperBob2 »

dataangel wrote: post #91
nurettin wrote: post #83

Earlier quoted context omitted.

I inherited systems that trade real world money using f64. They work surprisingly well, and the errors and bugs are almost never due to rounding. Those that are also have easy fixes. So I'm always baffled by this "expert opinion" of using integers for cents. It is pretty much up there with "never use python pickle it is unsafe" and "never use http, even if the program will never leave the subnet".

you can't accurately represent 10 cents with floats, 0.1 is not directly representable. same with 1 cent, 0.01. Seems like if you do and significant math on prices you should run into rounding issues pretty quickly?

At the end of a long chain of calculations you're going to round to the nearest 0.01. It will be a LONG time before errors caused by double-precision floats cause you to gain or lose a penny.

Re: Beware of Fast-Math

#154

Post by Athas »

rendaw wrote: post #145
jcranmer wrote: post #132

Earlier quoted context omitted.

For starters, it's giving up a lot of performance, since fixed-point isn't accelerated by hardware like floating-point is.

Isn't fixed point just integer?

Yes, but you're not going to have efficient transcendental functions implemented in hardware.

Re: Beware of Fast-Math

#155

Post by osigurdson »

gamescr wrote: post #137
usefulcat wrote: post #122

Earlier quoted context omitted.

You can certainly make trading systems that work using floating point, but there are just so many fewer edge cases to consider when using fixed point. With fixed point and at least 2 decimal places, 10.01 + 0.01 is always exactly equal to 10.02. But with FP you may end up with something like 10.0199999999, and then you have to be extra careful anywhere you convert that to a string that it doesn't get truncated to 10.…

I work on game engines and the problem with floats isn't on small values like 10.01 but on large ones like 400,010.01 that's when the precision wildly varies.

The issue with floats is the mental model. The best way to think about them is like a ruler with many points clustered around 0 and exponentially fewer as the magnitude grows. Don't think of it like a real value - assume that there are hardly any values represented with perfect precision. Even "normalish" numbers like 10.1 are not in the set actually. When values are converted to strings, even in debuggers sometimes, they are often rounded which throws people off further ("hey, the value is exactly 10.1 - it is right there in the debugger"). What you can count on however is that integers are represented with perfect precision up to a point (e.g. 2^53 -1 for f64).

The other "metal model" issue is that associative operations in math. Adding a + (b + c) != (a + b) + c due to rounding. This is where fp-precise vs fp-fast comes in. Let's not talk about 80 bit registers (though that used to be another thing to think about).

Re: Beware of Fast-Math

#156

Post by boulos »

I've also come around to --ffast-math considered harmful. It's useful though to help find optimization opportunities, but in the modern (AVX2+) world, I think the risks outweigh the benefits.

I'm surprised by the take that FTZ is worse than reassociation. FTZ being environmental rather than per instruction is certainly unfortunate, but that's true of rounding modes generally in x86. And I would argue that most programs are unprepared to handle subnormals anyway.

By contrast, reassociation definitely allows more optimization, but it also prohibits you from specifying the order precisely:

> Allow re-association of operands in series of floating-point operations. This violates the ISO C and C++ language standard by possibly changing computation result.

I haven't followed standards work in forever, but I imagine that the introduction of std::fma, gets people most of the benefit. That combined with something akin to volatile (if it actually worked) would probably be good enough for most people. Known, numerically sensitive code paths would be carefully written, while the rest of the code base can effectively be "meh, don't care".

Re: Beware of Fast-Math

#157

Post by Athas »

T0Bi wrote: post #130
jksflkjl3jk3 wrote: post #113

Earlier quoted context omitted.

Floating point math shouldn't be that scary. The rules are well defined in standards, and for many domains are the only realistic option for performance reasons. I've spent most of my career writing trading systems that have executed 100's of billions of dollars worth of trades, and have never had any floating point related bugs. Using some kind of fixed point math would be entirely inappropriate for most HFT or scie…

> Using some kind of fixed point math would be entirely inappropriate for most HFT or scientific computing applications. May I ask why? (generally curious)

The problem with fixed point is in its, well, fixed point. You assign a fixed number of bits to the fractional part of the number. This gives you the same absolute precision everywhere, but the relative precision (distance to the next highest or lowest number) is worse for small numbers - which is a problem, because those tend to be pretty important. It's just overall a less efficient use of the bit encoding space (not just performance-wise, but also in the accuracy of the results you get back). Remember that fixed point does not mean absence of rounding errors, and if you use binary fixed point, you still cannot represent many decimal fractions such as 0.1.

Re: Beware of Fast-Math

#158

Post by Athas »

pie_flavor wrote: post #123
razighter777 wrote: post #65

The worst thing that strikes fear into me is seeing floating points used for real world currency. Dear god. So many things can go wrong. I always use unsigned integers counting number of cents. And if I gotta handle multiple currencies, then I'll use or make a wrapper class.

One of the things I always appreciate about the crypto community is that you do not have to ask what numeric type is being used for money, it is always 8-digit fixed-point. No floating-point rounding errors to be found anywhere.

How does this avoid rounding error? Division and multiplication and still result in nonrepresentable numbers, right?

Re: Beware of Fast-Math

#159

Post by roryirvine »

klysm wrote: post #133
simonw wrote: post #80

Earlier quoted context omitted.

I've been having an interesting challenge relating to this recently. I'm trying to calculate costs for LLM usage, but the amounts of money involved are so tiny . Gemini 1.5 Flash 8B is $0.0375 per million tokens! Should I be running my accounting system on units of 10 billionths of a dollar?

Convert to money as late as possible

This is surely the right answer: simply count the number of tokens used, and do the billing reconciliation as a separate step.

As an added benefit, it makes it much easier to deal with price changes.

Re: Beware of Fast-Math

#160

Post by rendaw »

Athas wrote: post #154
rendaw wrote: post #145

Earlier quoted context omitted.

Isn't fixed point just integer?

Yes, but you're not going to have efficient transcendental functions implemented in hardware.

Ah okay, fair enough. But what sort of transcendental functions would you use for HFT?

I guess I understood GGGGP's comment about using fixed point for interacting with currency to be about accounting. I'd expect floating point to be used for trading algorithms, but that's mostly statistics and I presume you'd switch back to fixed point before making trades etc.

Post reply on HN