How flies get onboard with the latest trend these days.
First it was the LK-99 hype, now it is Q. Chasing hype like flies to a light bulb.
Q-Transformer
71–74 of 74 posts
Re: Q-Transformer
#72It's clear we're searching for the god algorithm of AI, just like physicists are searching for theory of everything. Are transformers the answer though?
There's this nice new paper ("Simplifying Transformer Blocks" https://arxiv.org/abs/2311.01906) about getting rid of all the unnecessary parts of a transformer. On their dataset, they manage to get rid of the layernorm, the skip connection, the value matrix, and the projection matrix.
Re: Q-Transformer
#73It's clear we're searching for the god algorithm of AI, just like physicists are searching for theory of everything. Are transformers the answer though?
No, it’s a much simpler architecture. Anything that evolved can’t require a really complicated niche structure.
The difficulty is the scale. Every synapse of a neuron is effectively a neuron itself, and every synapse acts on the synapses around it. So before you’ve even got to the neuron as a whole you’ve already got the equivalent of thousands of neurons and logic gates. Then the final result gets passed on to thousands more neurons.
I don’t know how you would recreate such complexity in programming. It’s not just the scale, it’s the flexibility of the structure.