31–40 of 53 posts
Earlier quoted context omitted.
Why don't they just change the current +/-inf to NaN, and then have the adjacent values be -inf or +inf. Or is that exception hard to implement?
here is the spec we drafted a few months ago on exactly what you suggest. https://github.com/interplanetary-robot/SigmoidNumbers/blob/...
> There are no “NaN” (not-a-number) bit representations with posits; instead, the calculation is interrupted, and the interrupt handler can be set to report the error and its cause, or invoke a workaround and continue computing, but posits do not make the logical error of assigning a number to something that is, by definition, not a number. This simplifies the hardware considerably. What a strange claim. Outputting a…
> This is not to say that NaNs are a good or bad method, but they're definitely not expensive to implement. I saw him explain that in float, there a too many bit representations that amount to "NaN". According to a Stack Overflow I found: "IEEE 754 standard defines 16,777,214 32-bit floating point values as NaNs, or 0.4% of all possible values." That's "expensive" in terms of losing bits of expressiveness that could…
Cost of a Interrupt: 100 ns to 1 microseconds (Quora) Sorry, that solution is simply not interesting for most implementations where floats are used. There are sensors which in realtime hammer out so many values, that not using NaNs means dropping part of your sensor values.
This is a classic case of re-inventing a optimal wheel (fine for racing) and not really looking at the use-cases (not fine for a lot of normal day to day driving).
Im sure it will bring the groupies on conferences.
Earlier quoted context omitted.
> This is not to say that NaNs are a good or bad method, but they're definitely not expensive to implement. I saw him explain that in float, there a too many bit representations that amount to "NaN". According to a Stack Overflow I found: "IEEE 754 standard defines 16,777,214 32-bit floating point values as NaNs, or 0.4% of all possible values." That's "expensive" in terms of losing bits of expressiveness that could…
Cost of a NaN Operation on chip- same as every other operation, as NaN just is handled and returned like another floating point value - always resulting in a new NaN value- thus a error invalidates all resulting wrong results. Cost of a Interrupt: 100 ns to 1 microseconds (Quora) Sorry, that solution is simply not interesting for most implementations where floats are used. There are sensors which in realtime hammer o…
That sounds like the time it takes to do a context switch to the OS. A math error interrupt doesn't need to do a context switch. The cost doesn't need to be any higher than a branch misprediction at 10-20 cycles.
I'll unpack just the first example that jumped out at me:
> Currently, half-precision (16-bit) IEEE floats are often used for this purpose, but 8-bit posits have the potential to be 2−4× faster. An important function for neural network training is a sigmoid function, a function f(x) that is asymptotically 0 as x → −∞ and asymptotically 1 as x → ∞. A common sigmoid function is 1/(1 + e−x) which is expensive to compute, easily requiring over a hundred clock cycles because of the math library call to evaluate exp(x), and because of the divide.
"have the potential to be" 2-4x faster? Sure. But until we see an implementation, 1-2x is much more likely (closer to the 1x end of the spectrum). Commodity hardware runs 32b IEEE float multiplications with 3-4 cycles latency and single-cycle throughput (or better). 16b can be made faster if designers care to. There's simply not much "faster" available for posits to inhabit (2-4x faster than 16b float would be faster than small integer arithmetic).
Evaluating a sigmoid function requires "over a hundred clock cycles" only in the most naive possible implementation sitting on top of a lousy math library. Using 32b floats on a current generation phone or laptop with a decent math library, a naive scalar implementation of the sigmoid function has a latency of less than 50 cycles. But latency doesn't matter at all in a machine learning context; we're interested only in throughput. On a machine with AVX-512, a single core can evaluate a sigmoid function with a throughput of about 1.25 cycles / input. In full-precision 32b floating-point (i.e. a relative error of ~10^-7). John's proposed posit implementation has a relative error of about 10^-1. If we target that error threshold, we can trivially go below 1 cycle/input in 32b or 16b float on a phone. So IEEE floats are at least two orders of magnitude faster than he claims. You need to go back more than 15 years for the numbers that the paper tosses around to even be plausible.
There are several other examples like this in the paper. I don't want to be too antagonistic, because posits are not a bad idea (actually, I think they're a pretty good format), but this paper is either ignorant of the state of the art, or more marketing than science.
> There are no “NaN” (not-a-number) bit representations with posits; instead, the calculation is interrupted, and the interrupt handler can be set to report the error and its cause, or invoke a workaround and continue computing, but posits do not make the logical error of assigning a number to something that is, by definition, not a number. This simplifies the hardware considerably. What a strange claim. Outputting a…
> If a programmer finds the need for NaN values, it indicates the program is not yet finished... These guys have never allreduced a timestep in a simulation where something can go aphysical. I mean, sure, one can communicate aphysical conditions using out-of-band means. But it is so pretty when it just rides along for free as a NaN result. Edit: I stand corrected having looked up authors. I still think having float e…
A closer analogue of NaN might be the x ("unknown") value in the three/four-valued logic of hardware simulation languages like Verilog. But the situation in Verilog, despite its challenges, is so much better since you don't have a hard discontinuity between "numbers" and "logical values". Numerical comparisons between unknowns can produce unknown logical values, and you have equations like true & unknown = unknown, true | unknown = true, ~unknown = unknown, etc, so you don't have the issue with NaNs where as soon as something crosses from the floating-point realm to the integer or logical realm, it has to commit to a definite value, with all the aforementioned issues.
All that said, NaNs were the right design choice given the constraints. But it's far from ideal.
Here's a C/C++ partial implementation of posits: https://github.com/libcg/bfp
bfp uses C++ classes which store the format hyperparameters as fields. If you'd rather a C/C++ library memory layout which reflects the bitwidth of the particular posit: https://github.com/Etaphase/FastSigmoids.jl contains both a julia library and a C/C++ library. Posits are implemented as a C type and a C++ class where the type and class sizes correspond to the bitwidth.
FastSigmoids uses float and double types which would be problematic in embedded environments.
Here's a C/C++ partial implementation of posits: https://github.com/libcg/bfp
Thanks! Can you elaborate on what's missing?
The next big TODOs are addition, substraction and rounding. Then advanced operations like sqrt, fma, etc. As part of the posit spec.
I've been working alone on that project, this being my first time working with floating point representation. I'm happy I could even get to that stage, and I hope someone can pick up my work and start contributing.
Earlier quoted context omitted.
bfp uses C++ classes which store the format hyperparameters as fields. If you'd rather a C/C++ library memory layout which reflects the bitwidth of the particular posit: https://github.com/Etaphase/FastSigmoids.jl contains both a julia library and a C/C++ library. Posits are implemented as a C type and a C++ class where the type and class sizes correspond to the bitwidth.
Dev here, I'm planning to transition to a C library with fixed size posit implementations. I need all the help I can get :) FastSigmoids uses float and double types which would be problematic in embedded environments.