Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How many floating-point numbers are in the interval [0,1]?

lemire.me

Post reply on HN
Print view View original

81–90 of 158 posts

Re: How many floating-point numbers are in the interval [0,1]?

#81

Post by mark-r »

SamReidHughes wrote: post #76
strainer wrote: post #70

Earlier quoted context omitted.

I see your talking 64 bit floats there which is my habit too. Having only 24 bit mantissa float32s seem insufficient for producing practically uniform variates - the missing 2^24th can be spotted averaging just a billion or so of them. I dont know if it has been improved recently but last year Chrome's Math.random was only putting 32 bits into its float64 - the missing four-billionth can be suggested from the average…

Yeah, I really think a better way is to imagine generating a uniform real and rounding -- generating [0,1) instead of [0,1] is just sick. If you do rand() * k + n you can get [n, n+k] anyway, so it's not a good primitive to get a half-open interval with. You can generate an integer n in [1, 2^25], then take (n >> 1) * (1 / (float)(1 << 24)) to get a pretty good [0,1].

For a 64-bit floating point number, the difference between [0,1) and [0,1] is for all practical purposes nonexistent. The chances of getting 1.0 exactly are so low that you can't build a test to know if it's excluded or not.

For a 32-bit floating point number it's more of a problem, but your application would have to be extremely demanding to detect the difference.

Re: How many floating-point numbers are in the interval [0,1]?

#82

Post by moron4hire »

astrodust wrote: post #8

> Of all the float-pointing point numbers your computer can represent, a quarter of them lie in [0,1]. Many people think floating point numbers are magically precise. They're not. They're far more accurate at lower magnitudes and precision fades as you work with larger values.

What's hilarious is that this is what is meant by calling them "floating point". The counterpoint "fixed point", in comparison, does not necessarily imply integers.

Re: How many floating-point numbers are in the interval [0,1]?

#83

Post by nine_k »

rnhmjoj wrote: post #50
nine_k wrote: post #45

Earlier quoted context omitted.

If NaN is nonsense, what would you have 1/0 or acos(2) to produce? (An exception is the same NaN, only less convenientlying packaged.)

Ideally I would restrict all functions to their natural domain using refinement types. This ways 1/0 simply doesn't type check and you don't have to bother assigning it some nonsensical value. However I can see something like this becoming much more a hassle than a benefit when things get complicated enough.

I wonder how this could be done on hadware level.

You can look at the NaN value as an implementation of Option / Maybe, where it means None. It even behaves like it should, short-circuiting any normal operations with it to also produce NaN. Too bad Fortran and C lack sum types to surface that properly on the type level.

Re: How many floating-point numbers are in the interval [0,1]?

#84

Post by dbcurtis »

dlubarov wrote: post #60
nine_k wrote: post #45

Earlier quoted context omitted.

If NaN is nonsense, what would you have 1/0 or acos(2) to produce? (An exception is the same NaN, only less convenientlying packaged.)

> An exception is the same NaN, only less convenientlying packaged. I prefer an exception because I'd rather have my code fail fast, and immediately point me close to the source of the bug, rather than letting a NaN or Infinity propagate through my code base and cause some harder-to-debug problems down the line. That's the main thing I don't like about JavaScript. 1/0 is Infinity, Math.acos(2) is NaN, Object().foo is…

Exceptions are utterly painful for vectorized operations. Goodbye Tensorflow if you say goodbye to NaNs

Re: How many floating-point numbers are in the interval [0,1]?

#85

Post by dbcurtis »

sunfish wrote: post #58
jacobolus wrote: post #18

Earlier quoted context omitted.

John Gustafson’s new “sigmoid unum”/“posit” proposal is kind of interesting; see his recent talk: http://web.stanford.edu/class/ee380/Abstracts/170201.html https://news.ycombinator.com/item?id=13562164 It does a variable number of fraction bits, so that the values near 1 are even more densely represented than under the usual IEEE floats, while also providing greater dynamic range (but at reduced precision). I made a…

Out of curiosity, I started watching this talk, but right there in the introduction, the very first example is a dot product where supposedly IEEE 754 double precision gets the wrong answer; I stopped to check the result, and I got the correct answer with double precision (even without binary sum collapse). Then, he says the x87's results are nondeterministic due to being affected by cache, which is incorrect. Then,…

The Cray 1 was done before anyone realized the importance of denormals. And denormals are gruesome to implement on vector machines anyway.

Re: How many floating-point numbers are in the interval [0,1]?

#86

Post by greglindahl »

dbcurtis wrote: post #84
dlubarov wrote: post #60

Earlier quoted context omitted.

> An exception is the same NaN, only less convenientlying packaged. I prefer an exception because I'd rather have my code fail fast, and immediately point me close to the source of the bug, rather than letting a NaN or Infinity propagate through my code base and cause some harder-to-debug problems down the line. That's the main thing I don't like about JavaScript. 1/0 is Infinity, Math.acos(2) is NaN, Object().foo is…

Exceptions are utterly painful for vectorized operations. Goodbye Tensorflow if you say goodbye to NaNs

That's an excellent explanation for why scientific computing has been using vectors for a long time and has traditionally faulted when hitting the first NaN. Is there something about Tensorflow that makes propagating NaNs useful? And if so, why are you generalizing from Tensorflow to all "vectorized operations"?

Re: How many floating-point numbers are in the interval [0,1]?

#87

Post by greglindahl »

dbcurtis wrote: post #85
sunfish wrote: post #58

Earlier quoted context omitted.

Out of curiosity, I started watching this talk, but right there in the introduction, the very first example is a dot product where supposedly IEEE 754 double precision gets the wrong answer; I stopped to check the result, and I got the correct answer with double precision (even without binary sum collapse). Then, he says the x87's results are nondeterministic due to being affected by cache, which is incorrect. Then,…

The Cray 1 was done before anyone realized the importance of denormals. And denormals are gruesome to implement on vector machines anyway.

Only a subset of scientific codes benefit from denormals, and it does not appear that implementing them is that big of a deal, given that 4-stage pipelines can do it. The pipeline depth to memory is a lot bigger than 4.

Re: How many floating-point numbers are in the interval [0,1]?

#88

Post by modeless »

dnautics wrote: post #73
jfaucett wrote: post #13

Honestly, I think IEEE 754 floating point numbers are a pretty bad way of dealing with real numbers and just cause tons of headaches that every math library has to deal with. Even high level programmers aren't shielded from the NaN nonsense. It would be great if we had an underlying implementation we could ignore and that allowed us to think at a more mathematical level. I actually found something recently on this to…

so I'm the author of one of the first usable unum implementations. We're actually moving to something called sigmoid numbers which are even better. The lecture doesn't cover "valid mode" but that's more like the "unum". At the end of the video, there's a demonstration where I show some very interesting results concerning machine learning. https://www.youtube.com/watch?v=aP0Y1uAA-2Y You can try out sigmoid numbers in…

> very interesting results concerning machine learning

This is the big chance for alternative number formats. a whole lot of people are working on neural net ASICs these days and IEEE 754 is not a requirement. A new number format with a low power/low area hardware implementation could easily find adoption in this new area.

Re: How many floating-point numbers are in the interval [0,1]?

#90

Post by dbcurtis »

greglindahl wrote: post #87
dbcurtis wrote: post #85

Earlier quoted context omitted.

The Cray 1 was done before anyone realized the importance of denormals. And denormals are gruesome to implement on vector machines anyway.

Only a subset of scientific codes benefit from denormals, and it does not appear that implementing them is that big of a deal, given that 4-stage pipelines can do it. The pipeline depth to memory is a lot bigger than 4.

If you consider every iterative algorithm that solves systems of nonlinear equations a subset you cam ignore.... SPICE, linpack, fishpack.... sorry, denormals are essential to today's algorithms.

As to "no big deal".... show me your code.

Don't worry, I'll be able to understand it. After 20 years as a CPU designer I've learned how to understand bit-bashing.

Post reply on HN