I somewhat feel that using floating point arithmetic for what should be a symbol manipulation exercise is cheating. The deserialisation technique is interesting enough that I'm not really upset, though. > The codex solution reversed the order which makes sense for making carry logic easy, but it is less clean. That's the approach I'd have gone with. I've long been an advocate of little-endian numerical representation…
symbol manipulation in transformers is fp arithmetic?
That's an implementation detail. The behaviour of trained transformer models remains similar even if you quantise them to 4-bit floats, or make every floating point operation noisy. This model only works if you use double-precision floating point.