Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of Fast-Math

simonbyrne.github.io

Post reply on HN
Print view View original

91–100 of 233 posts

Re: Beware of Fast-Math

#91

Post by dataangel »

nurettin wrote: post #83
razighter777 wrote: post #65

The worst thing that strikes fear into me is seeing floating points used for real world currency. Dear god. So many things can go wrong. I always use unsigned integers counting number of cents. And if I gotta handle multiple currencies, then I'll use or make a wrapper class.

I inherited systems that trade real world money using f64. They work surprisingly well, and the errors and bugs are almost never due to rounding. Those that are also have easy fixes. So I'm always baffled by this "expert opinion" of using integers for cents. It is pretty much up there with "never use python pickle it is unsafe" and "never use http, even if the program will never leave the subnet".

you can't accurately represent 10 cents with floats, 0.1 is not directly representable. same with 1 cent, 0.01. Seems like if you do and significant math on prices you should run into rounding issues pretty quickly?

Re: Beware of Fast-Math

#92

Post by scythe »

StefanKarpinski wrote: post #86
vanderZwan wrote: post #46

Earlier quoted context omitted.

> These tell the compiler to assume for optimization purposes that floats are associative and so on (ie. algebraic), even when in reality they aren't. I wonder if it is possible to add an additional constraint that guarantees the transformation has equal or fewer numerical rounding errors. E.g. for floating point doubles (0.2 + 0.1) - 0.1 results in 0.20000000000000004, so I would expect that transforming some (A + B…

Pretty sure that’s not possible. More accurate for some inputs will be less accurate for others. There’s a very tricky tension in float optimization that the most predictable operation structure is a fully skewed op tree, as in naive left-to-right summation, but this is the slowest and least accurate order of operations. Using a more balanced tree is faster and more accurate (great), but unfortunately which tree shap…

I think a restricted version might be possible to implement: only allow transformations if the transformed version has strictly fewer numerical rounding errors on some inputs. This will usually only mean canceling terms and collecting expressions like "x+x+x" into 3x.

In general, rules that allow fewer transformations are probably easier to understand and use. Trying to optimize everything is where you run into trouble.

Re: Beware of Fast-Math

#93

Post by scott_w »

razighter777 wrote: post #65

The worst thing that strikes fear into me is seeing floating points used for real world currency. Dear god. So many things can go wrong. I always use unsigned integers counting number of cents. And if I gotta handle multiple currencies, then I'll use or make a wrapper class.

For far too many years I had inherited a billing system that used floats for all calculations then rounded up or down. Also doing some calculations in JS and mirroring them on the Python backend, so “just switch to Decimal” wasn’t an easy change to make…

Re: Beware of Fast-Math

#94

Post by simonw »

outurnate wrote: post #84
simonw wrote: post #80

Earlier quoted context omitted.

I've been having an interesting challenge relating to this recently. I'm trying to calculate costs for LLM usage, but the amounts of money involved are so tiny . Gemini 1.5 Flash 8B is $0.0375 per million tokens! Should I be running my accounting system on units of 10 billionths of a dollar?

You're better off representing values as rationals; a ratio between two different numbers. For example, 0.0375 would be represented as 375 over 10000, or 3 over 80

Sounds hard to model in SQLite.

Re: Beware of Fast-Math

#95

Post by anthk »

AlotOfReading wrote: post #88
anthk wrote: post #3

Earlier quoted context omitted.

On Forth, there's the philosophy of the fixed point: https://www.forth.com/starting-forth/5-fixed-point-arithmeti... With 32 and 64 bit numbers, you can just scale decimals up. So, Torvalds was right. On dangerous contexts (uper-precise medical doses, FP has good reasons to exist, and I am not completely sure). Also, both Forth and Lisp internally suggest to use represented rationals before floating point numbers. Ev…

Floats are fixed point, just done in log space. The main change is that the designers dedicated a few bits to variable exponents, which introduces alignment and normalization steps before/after the operation. If you don't mix exponents, you can essentially treat it as identical to a lower precision fixed point system.

No, not even close. Scaling integers to mimic decimals under 32 and 64 bit can be much faster. And with 32 bit double numbers you can cover Plank numbers, so with 64 bit double numbers you can do any field.

Re: Beware of Fast-Math

#96

Post by scott_w »

simonw wrote: post #80
razighter777 wrote: post #65

The worst thing that strikes fear into me is seeing floating points used for real world currency. Dear god. So many things can go wrong. I always use unsigned integers counting number of cents. And if I gotta handle multiple currencies, then I'll use or make a wrapper class.

I've been having an interesting challenge relating to this recently. I'm trying to calculate costs for LLM usage, but the amounts of money involved are so tiny . Gemini 1.5 Flash 8B is $0.0375 per million tokens! Should I be running my accounting system on units of 10 billionths of a dollar?

Fixed point Decimal is your friend here. I’m guessing you buy tokens in increments of 1,000,000 so it isn’t too much of an issue to account for. You can then normalise in your accounting so 1,000,000 is just “1 unit,” or you can just account in increments of 1,000,000 but that does start looking weird (but might be necessary!)

Re: Beware of Fast-Math

#97

Post by layer8 »

eapriv wrote: post #77
goalieca wrote: post #57

Earlier quoted context omitted.

Floating point arithmetic is neither commutative or associative so you shouldn’t.

Why is it not commutative?

It actually is commutative according to IEEE-754, except that in the case of a NaN result you might get a different NaN representation.

Re: Beware of Fast-Math

#98

Post by layer8 »

goalieca wrote: post #57
Affric wrote: post #36

Earlier quoted context omitted.

How does IEEE 754 prevent auto-vectorisation?

Floating point arithmetic is neither commutative or associative so you shouldn’t.

IEEE-754 addition and multiplication is commutative. It isn't distributive, though.

Re: Beware of Fast-Math

#99

Post by layer8 »

JKCalhoun wrote: post #60

> Even compiler developers can't agree. > This is perhaps the single most frequent cause of fast-math-related StackOverflow questions and GitHub bug reports The second line above should settle the first.

The first line points out that it doesn't, even if one thinks that it should. Also, note the "perhaps".

Re: Beware of Fast-Math

#100

Post by layer8 »

razighter777 wrote: post #65

The worst thing that strikes fear into me is seeing floating points used for real world currency. Dear god. So many things can go wrong. I always use unsigned integers counting number of cents. And if I gotta handle multiple currencies, then I'll use or make a wrapper class.

Wait until you learn that Excel calculates everything using floating-point, and doesn't even fully observe IEEE 754.

https://learn.microsoft.com/en-us/office/troubleshoot/excel/...

(It nevertheless happens to work just fine for most of what Excel is used for.)

Post reply on HN