101–110 of 202 posts
It seems like most breakthroughs I see are for efficiency? What are the most importsnt breakthroughs from the past two or three years for intelligence?
Efficiency gains can be used to make existing models more profitable, or to make new larger and more intelligent models.
Someone implementing it on llamacpp already https://github.com/mudler/llama.cpp/commit/dee102db1bfd723c9...
Hopefully Johnson–Lindenstrauss lemma applies in the same way for SRHTransformed vectors as they do for randomly rotated vectors and the independence of the distribution laws of the coordinates remains and therefore the quantization of each coordinates independently is still theoretically sound.
Earlier quoted context omitted.
Pardon my simplistic question, but when you mean rotation you’re essentially talking about diagonalization aren’t you? So storing the diagonal as a matrix and the new bases is more compact?
In this context, the rotation is for spreading energy and ensuring predictable coordinate distributions rather than diagonalization; it makes coordinate-wise quantization much more computationally efficient, though it throws away learnable structure.
Earlier quoted context omitted.
I think it is though- “ TurboQuant, QJL, and PolarQuant are more than just practical engineering solutions; they’re fundamental algorithmic contributions backed by strong theoretical proofs. These methods don't just work well in real-world applications; they are provably efficient and operate near theoretical lower bounds.”
Genius new idea: replace the em-dashes with semicolons so it looks less like AI.
There goes another bit of my writing style that will get mistaken for an LLM.
It seems like most breakthroughs I see are for efficiency? What are the most importsnt breakthroughs from the past two or three years for intelligence?
Most breakthroughs that are published are for efficiency because most breakthroughs that are published are for open source.' All the foundation model breakthroughs are hoarded by the labs doing the pretraining. That being said, RL reasoning training is the obvious and largest breakthrough for intelligence in recent years.
And a group has published an independent working implementation today, nice to see: https://github.com/tonbistudio/turboquant-pytorch
And a group has published an independent working implementation today, nice to see: https://github.com/tonbistudio/turboquant-pytorch
It has a lot clearer explanation of the method than Google's own post.