Curious if anybody out there is trying to build a new model/architecture that would succeed the transformer?
I geek out on this subject in my spare time. Curious if anybody else is doing so and if you're willing to share ideas?
Ask HN: Is anybody building an alternative transformer?
1–10 of 96 posts
Re: Ask HN: Is anybody building an alternative transformer?
#2please define transformer
Re: Ask HN: Is anybody building an alternative transformer?
#3please define transformer
Robots in disguise.
Re: Ask HN: Is anybody building an alternative transformer?
#4The one from cybertron? The one that changes voltage levels? The AI algorithm one?
Edit: or perhaps you are working on a new insect sex regulation gene? If so that would be a great discussion here - https://en.wikipedia.org/wiki/Transformer_(gene)
Re: Ask HN: Is anybody building an alternative transformer?
#5I've a design in mind which is very simple and interesting but don't know if it would be scalable to the stage, rn it's just a superficial design inspired by IronMan's JARVIS, i'm working on preparing the architecture.
Re: Ask HN: Is anybody building an alternative transformer?
#6I hate that popular domains take ownership of highly generic words. Many years ago, I struggled for a while to understand that when people say "frontend" they often mean a website frontend, even without any further context.
Re: Ask HN: Is anybody building an alternative transformer?
#7The xLSTM could become a good alternative to transformers: https://arxiv.org/abs/2405.04517. On very long contexts, such as those arising in DNA models, these models perform really well.
There's a big state-space model comeback initiated by the S3-Mamba saga. RWKV, which is a hybrid between classical RNNs and transformers, is also worth mentioning.
Re: Ask HN: Is anybody building an alternative transformer?
#8There are alternatives that optimize around the edges. Like Deepseek’s Multi-head Latent Attention, or Grouped Query Attention. DeepSeek also showed an optimization on Mixture of Experts. These are all clear improvements to the Vaswani architecture.
There are optimizations like extreme 1.58 bit quant that can be applied to anything.
There are architectures that stray farther. Like SSMs and some attempts at bringing the RNN back from the dead. And even text diffusion models that try to generate paragraphs like we generate images i.e. not word by word.
Re: Ask HN: Is anybody building an alternative transformer?
#9please define transformer
Re: Ask HN: Is anybody building an alternative transformer?
#10Literally everybody doing cutting edge AI research is trying to replace the transformer, because transformers have a bunch of undesirable properties like being quadratic in context window size. But they're also surprisingly resilient: despite the billions of dollars and man-hours poured into the field and many attempted improvements, cutting-edge models aren't all that different architecturally from the original attention paper, aside from their size and a few incidental details like the ReLU activation function, because nobody has found anything better yet.
I do expect transformers to be replaced eventually, but they do seem to have their own "bitter lesson" where trying to outperform them usually ends in failure.