51–53 of 53 posts
Posits are actually quite reasonable, but there's a lot of either ignorance or disingenuousness in this article, which is really too bad. I wish that John would ditch the hyperbole and solicit feedback from other experts, because posits are not a bad idea, but the presentation continues to give him the trappings of a crank. I'll unpack just the first example that jumped out at me: > Currently, half-precision (16-bit)…
Earlier quoted context omitted.
I'm just going to be blunt here. John and I have decided that we need to be more marketing savvy after he's had trouble with several rounds pitching other floating point formats. Posits are just an intermediate step to try to build acceptance for valids, so there's a lot of effort put into branding. A couple of points: 2-4x faster means 2x faster in dot product based simd and 4x faster in matrix simd, assuming that y…
If you're bound by memory throughput, you can't go beyond a 2x speedup (there's 1/2 as much data to move in an 8b format, whether it's in vectors or matrices doesn't matter). I still don't see any reasonable expectation for 4x.
[a b, c d] dot [e, f]
is four multiplies
[a b c, d e f, g h i] dot [j, k, l]
is nine multiplies.
Earlier quoted context omitted.
If you're bound by memory throughput, you can't go beyond a 2x speedup (there's 1/2 as much data to move in an 8b format, whether it's in vectors or matrices doesn't matter). I still don't see any reasonable expectation for 4x.
if your matrix contents are static during your rate-limiting step (as they are for most DL applications) your FLOPs scale with O(n^2) relative to your memory throughput on your vector component. [a b, c d] dot [e, f] is four multiplies [a b c, d e f, g h i] dot [j, k, l] is nine multiplies.