Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

4-bit floating point FP4

johndcook.com

Post reply on HN
Print view View original

21–30 of 81 posts

Re: 4-bit floating point FP4

#21

Post by Dylan16807 »

sc0ttyd wrote: post #13

9 years ago, I shared this as an April Fools joke here on HN. It seems that life is imitating art. https://github.com/sdd/ieee754-rrp

> 9 years ago, I shared this as an April Fools joke here on HN.

That's fun.

> It seems that life is imitating art.

You didn't even beat wikipedia to the punch. They've had a nice page about minifloats using 6-8 bit sizes as examples for about 20 years.

The 4 bit section is newer, but it actually follows IEEE rules. Your joke formats forgot there's an implied 1 bit in the fraction. And how exponents work.

Re: 4-bit floating point FP4

#22

Post by omoikane »

pixelesque wrote: post #9
kimixa wrote: post #3

Earlier quoted context omitted.

Even the latest CPUs have a 2:1 fp64:fp32 performance ratio - plus the effects of 2x the data size in cache and bandwidth use mean you can often get greater than a 2x difference. If you're in a numeric heavy use case that's a massive difference. It's not some outdated "Ancient Lore" that causes languages that care about performance to default to fp32 :P

> Even the latest CPUs have a 2:1 fp64:fp32 performance ratio Not completely - for basic operations (and ignoring byte size for things like cache hit ratios and memory bandwidth) if you look at (say Agner Fog's optimisation PDFs of instruction latency) the basic SSE/AVX latency for basic add/sub/mult/div (yes, even divides these days), the latency between float and double is almost always the same on the most recent…

> the latency between float and double is almost always the same on the most recent AMD/Intel CPUs

If you are developing for ARM, some systems have hardware support for FP32 but use software emulation for FP64, with noticeable performance difference.

https://gcc.godbolt.org/z/7155YKTrK

Re: 4-bit floating point FP4

#23

Post by adgjlsfhk1 »

kimixa wrote: post #19
pixelesque wrote: post #9

Earlier quoted context omitted.

> Even the latest CPUs have a 2:1 fp64:fp32 performance ratio Not completely - for basic operations (and ignoring byte size for things like cache hit ratios and memory bandwidth) if you look at (say Agner Fog's optimisation PDFs of instruction latency) the basic SSE/AVX latency for basic add/sub/mult/div (yes, even divides these days), the latency between float and double is almost always the same on the most recent…

> ... if you look at (say Agner Fog's optimisation PDFs of instruction latency) ... That.... doesn't seem true? At least for most architectures I looked at? While true the latency for ADDPS and ADDPD are the same latency, using the zen4 example at least, the double variant only calculates 4 fp64 values compared to the single-precision's 8 fp32. Which was my point? If each double precision instruction processes a smal…

This depends largely on your operations. There is lots of performance critical code that doesn't vectorize smoothly, and for those operations, 64 bit is just as fast.

Re: 4-bit floating point FP4

#24

Post by ted_dunning »

ant6n wrote: post #4

> In ancient times, floating point numbers were stored in 32 bits. I thought in ancient times, floating point numbers used to be 80 bit. They lived in a funky mini stack on the coprocessor (x87). Then one day, somebody came along and standardized those 32 and 64 bit floats we still have today.

That is merely medieval times.

In ancient times, floats were all 60 bits and there was no single precision.

See page 3-15 of this https://caltss.computerhistory.org/archive/6400-cdc.pdf

Re: 4-bit floating point FP4

#25

Post by ant6n »

ted_dunning wrote: post #24
ant6n wrote: post #4

> In ancient times, floating point numbers were stored in 32 bits. I thought in ancient times, floating point numbers used to be 80 bit. They lived in a funky mini stack on the coprocessor (x87). Then one day, somebody came along and standardized those 32 and 64 bit floats we still have today.

That is merely medieval times. In ancient times, floats were all 60 bits and there was no single precision. See page 3-15 of this https://caltss.computerhistory.org/archive/6400-cdc.pdf

That written document is prehistoric.

Re: 4-bit floating point FP4

#27

Post by Figs »

> The notation ExMm denotes a format with x exponent bits and y mantissa bits.

Shouldn't that be m mantissa bits (not y) -- i.e. typo here -- or am I misunderstanding something?

Re: 4-bit floating point FP4

#28

Post by kimixa »

adgjlsfhk1 wrote: post #23
kimixa wrote: post #19

Earlier quoted context omitted.

> ... if you look at (say Agner Fog's optimisation PDFs of instruction latency) ... That.... doesn't seem true? At least for most architectures I looked at? While true the latency for ADDPS and ADDPD are the same latency, using the zen4 example at least, the double variant only calculates 4 fp64 values compared to the single-precision's 8 fp32. Which was my point? If each double precision instruction processes a smal…

This depends largely on your operations. There is lots of performance critical code that doesn't vectorize smoothly, and for those operations, 64 bit is just as fast.

Yes, if you're not FP ALU limited (which is likely the case if not vectorized), or data cache/bandwidth/thermally limited from the increased cost of fp64, then it doesn't matter - but as I said that's true for every performance aspect that "doesn't matter".

That doesn't mean that there are no situations where it does matter today - which is what I feel is implied by calling it "Ancient".

Re: 4-bit floating point FP4

#29

Post by nomel »

sc0ttyd wrote: post #13

9 years ago, I shared this as an April Fools joke here on HN. It seems that life is imitating art. https://github.com/sdd/ieee754-rrp

Lowest I've used is 8 bit floats for time delays, in embedded devices.
Post reply on HN