Live data from Hacker News

4-bit floating point FP4

johndcook.com

61–70 of 81 posts

Re: 4-bit floating point FP4

#61

> In ancient times, floating point numbers were stored in 32 bits. Then somewhere along the way 64 bits became standard. I think Cray doubles were 128 bits, and their singles were 64… which makes it seem like smaller floats are just a continuation of the eternal trend.

The earliest Cray models (starting with Cray-1 in 1976) had only 64-bit floating-point numbers. 128-bit numbers were a later addition and I do not think that they were implemented in hardware, but only in software. Very few computers, except some from IBM, have implemented FP128 in hardware, while software libraries for quadruple-precision or double-double-precision FP128 are widespread.

The Cray 64-bit format was a slight increase in size over the 60-bit floating-point numbers that had been used in the previous computers designed by Seymour Cray, at CDC.

Before IBM increased the size of a byte to 8 bits, which caused all numeric formats to use sizes that are multiple of 8-bits, in the computers with 6-bit bytes the typical floating-point number sizes were either 60-bit in the high-end models or 48-bit in cheaper models or 36-bit in the cheapest models.

Re: 4-bit floating point FP4

#62
post #57

Earlier quoted context omitted.

Barely any information. After surviving RELU that signed zero is probably getting added to another value and then oops the information is gone. It sounds a lot worse than properly spaced values.

sign = most important bit of information

If you were looking at the entire number line, sign would roughly be the most important part.

But you still have all the other numbers carrying sign info. This is only the sign of denormals and that's way less valuable. Outside of particular equations it ends up added to something else and disappearing entirely. It would be way better to cut it and have either half the smallest existing positive value or double the largest existing value as a replacement. Or many other options.

Re: 4-bit floating point FP4

#63
post #20

Earlier quoted context omitted.

> languages that care about performance to default to fp32 What do you mean by this? In C 1.0 is a double.

But the "float" typename is generally fp32 - if we assume the "most generically named type" is the "default". Though this is a bit of an inconsistency with C - the type name "double" surely implies it's double the expected baseline while, as you mentioned, constants and much of libm default to 'double'.

The C keywords "float" and "double" are based on the tradition established a decade earlier by IBM System/360 of calling FP32 as "single-precision" and FP64 as "double-precision".

This IBM convention has been inherited by the IBM programming languages FORTRAN IV and PL/I and from these 2 languages it has spread everywhere.

The C language has taken several keywords and operators from IBM PL/I, which was one of the three main inspiration sources for C (which were CPL/BCPL, PL/I and ALGOL 68).

So "float" and "double" are really inherited by C from PL/I.

A feature that is specific to C is that it has changed the default format for constants and for intermediate values to double-precision, instead of the single-precision that was the default in earlier programming languages.

This was done with the intention of protecting naive users from making mistakes, because if you compute with FP32 it is very easy to obtain erroneous results, unless you analyze very carefully the propagation of errors. Except in applications where errors matter very little, e.g. graphics and ML/AI, the use of FP32 is more suitable for experts, while bigger formats are recommended for normal users.

Re: 4-bit floating point FP4

#64
post #25

Earlier quoted context omitted.

That is merely medieval times. In ancient times, floats were all 60 bits and there was no single precision. See page 3-15 of this https://caltss.computerhistory.org/archive/6400-cdc.pdf

That written document is prehistoric.

By definition, a document that is written is historic, not prehistoric.

Prehistoric information could be preserved by an oral tradition, until it is recorded in some documents (like the Oral Histories at the Computer History Museum site).

Re: 4-bit floating point FP4

#65
post #4

> In ancient times, floating point numbers were stored in 32 bits. I thought in ancient times, floating point numbers used to be 80 bit. They lived in a funky mini stack on the coprocessor (x87). Then one day, somebody came along and standardized those 32 and 64 bit floats we still have today.

80 bits is just in the processor. Thats why you might a little bit different result, depending how you calculated first and maybe stored something in the RAM

Intel 8087, which has introduced in 1980 the 80-bit extended floating point format, could store and load 80-bit numbers, avoiding any alterations caused by conversions to less precise formats.

To be able to use the corresponding 8087 instructions, "long double" has been added to the C language, so to avoid extra roundings one had to use "long double" variables and one had to also be careful so that intermediate values used for the computing of an expression will not be spilled into the memory as "double".

However this became broken in some newer C compilers, where due to the deprecation of the x87 ISA "long double" was made synonymous to "double". Some better C compilers have chosen to implement "long double" as quadruple-precision instead of extended precision, which ensures that no precision is lost, but which may be slow on most computers, where no hardware support for FP128 exists.

Re: 4-bit floating point FP4

#66
post #4

> In ancient times, floating point numbers were stored in 32 bits. I thought in ancient times, floating point numbers used to be 80 bit. They lived in a funky mini stack on the coprocessor (x87). Then one day, somebody came along and standardized those 32 and 64 bit floats we still have today.

I was going to reply that just because intel did something funny doesn't mean that it was the beginning of the story. but it turns out that the release of the 8087 predates the ratification of IEEE floats by 2 years. in addition, the primary numeric designer for the 8087 was apparently Kahan, which means that they were both part of the same design process. of course there were other formats predating both of these

The Intel 8087 design team, with Kahan as their consultant, who was the author of most novel features, based on his experience with the design of the HP scientific calculators, have realized that instead of keeping their new much improved floating-point format as proprietary it would be much better to agree with the entire industry on a common floating-point standard.

So Intel has initiated the discussions for the future IEEE standard with many relevant companies, even before the launch of 8087. AMD was a company convinced immediately by Intel, so AMD was able to introduce a FP accelerator (Am9512) based on the 8087 FP formats, which were later adopted in IEEE 754, also in 1980 and a few months before the launch of Intel 8087. So in 1980 there already were 2 implementations of the future IEEE 754 standard. Am9512 was licensed to Intel and Intel made it using the 8232 part number (it was used in 8080/8085/Z80 systems).

Unlike AMD, the traditional computer companies agreed that a FP standard is needed to solve the mess of many incompatible FP formats, but they thought that the Kahan-Intel proposal would be too expensive for them, so they came with a couple of counter-proposals, based on the tradition of giving priority to implementation costs over usefulness for computer users.

Fortunately the Intel negotiators eventually succeeded to convince the others to adopt the Intel proposal, by explaining how the new features can be implemented at an acceptable cost.

The story of IEEE 754 is one of the rare stories in standardization where it was chosen to do what is best for customers, not what is best for vendors.

Like the use of encryption in communications, the use of the IEEE standard has been under continuous attacks during its history, coming from each new generation of logic designers, who think that they are smarter than their predecessors, and who are lazy to implement properly some features of the standard, despite the fact that older designs have demonstrated that they can in fact be implemented efficiently, but the newbies think that they should take the easy path and implement inefficiently some features of the standard, because supposedly the users will not care about that.

Re: 4-bit floating point FP4

#67

> In ancient times, floating point numbers were stored in 32 bits. This was true only for cheap computers, typically after the mid sixties. Most of the earliest computers with vacuum tubes used longer floating-point number formats, e.g. 48-bit, 60-bit or even weird sizes like 57-bit. The 32-bit size has never been acceptable in scientific computing with complex computations where rounding errors accumulate. The early…

You are totally correct but I need you to recognize that "in ancient times" includes the 1990s.

I am...very sorry to be the one delivering this news. It was not a pleasant realization for me, either.

Re: 4-bit floating point FP4

#68
post #48
post #46

Earlier quoted context omitted.

OTOH, it seems quite plausible that the most important numbers to represent are: +0 -0 +1 -1 +inf -inf

Why waste a slot on -0?

You need it if you want the idea of total ordering over the extended Reals. There's +/- infinity--an affine closure, not projective (point at infinity)--so to make that math work you need to give 0 a sign.

Re: 4-bit floating point FP4

#69
post #16
post #4

> In ancient times, floating point numbers were stored in 32 bits. I thought in ancient times, floating point numbers used to be 80 bit. They lived in a funky mini stack on the coprocessor (x87). Then one day, somebody came along and standardized those 32 and 64 bit floats we still have today.

x87 always had a choice of 32/64/80-bit user-facing floats. It just operated internally on 80 bits.

You can set x87 to round each operation result to 32-bit or 64-bit.

With this setting in operates internally exactly on those sizes.

Operating internally on 80-bits is just the default setting, because it is the best for naive users, who are otherwise prone to computing erroneous results.

This is the same reason why the C language has made "double" the default precision in constants and intermediate values.

Unless you do graphics or ML/AI, single-precision computations are really only for experts who can analyze the algorithm and guarantee that it is correct.

Re: 4-bit floating point FP4

#70

> In ancient times, floating point numbers were stored in 32 bits. This was true only for cheap computers, typically after the mid sixties. Most of the earliest computers with vacuum tubes used longer floating-point number formats, e.g. 48-bit, 60-bit or even weird sizes like 57-bit. The 32-bit size has never been acceptable in scientific computing with complex computations where rounding errors accumulate. The early…

You are totally correct but I need you to recognize that "in ancient times" includes the 1990s. I am...very sorry to be the one delivering this news. It was not a pleasant realization for me, either.

A few years after 1980, especially after 1985, the computers with coprocessors like Intel 8087 or Motorola 68881 became the most numerous computers with floating-point hardware, and for them the default FP size was 80-bit.

So the 1990s were long after the time when 32-bit FP numbers were normal. FP32 was revived only by GPUs, for graphic applications where precision matters much less.

Already after 1974, the C programming language made double-precision the default FP size, not the 32-bit single-precision size, for the same reason why Intel 8087 introduced extended precision. Single-precision computations for traditional applications are suitable only for experts, not for ordinary computer users.

While before C the programming languages used single-precision 32-bit numbers as the default size, the recommendations were already to use only double-precision wherever complicated expressions were computed.

I have started using computers by punching cards for a mainframe, but that was already at a time when 32-bit FP numbers were not normally used, but only 64-bit FP numbers.

The best chances of seeing 32-bit single-precision numbers in use was in the decade from 1965 to 1975, at the users of cheap mainframes or of minicomputers without hardware floating-point units, where floating-point emulation was done in software and emulating double-precision was significantly slower.

Before the mid sixties, there were more chances to see 36-bit floating-point numbers as the smallest FP size.

Post reply on HN