Now I understand why ML uses floats for precision. It wasn't a choice, it was because graphics code uses them. Another piece in the "why is ML so inefficient" puzzle! I wonder what that memory copying overhead is IRL. If it's like normal stuff it'll be brutal. I mean, they offload tcp processing into hardware to avoid that. This is way more data, though it is done in bigger chunks.
But really, I wondered why the cpu-bound LLMs do quantization, which from what I understand is the process of reducing the precision of the weights to use less memory.
Does the lack of precision make a difference? It's unclear. If that's the case, then why use FP at all? If precision doesn't matter, then the extra precision is just making the process use more resources for no real reason. And likely orders of magnitude more resources than required.
I mean, this field wasn't started by people who understood performance. They used tools and built something...but there's no 'why.' This is what the tools did, so they did that.
Here's why that might be important: on a normal CPU, accessing data one way can be orders of magnitude faster than another way...but you have to be aware. Would you like to reduce your LLM costs by orders of magnitude?