Yes. Happy to chat if u msg me. Using RL coupled with NNs to integrate search directly into inference instead of as an afterthought like Chain of though and test time training.
Are we able to "msg" people on here?
No, thank god... =3
Ask HN: Is anybody building an alternative transformer?
41–50 of 96 posts
Re: Ask HN: Is anybody building an alternative transformer?
#42Anyone know what the rwkv people are up to now? https://arxiv.org/abs/2305.13048
You can see all the development directly from them: https://github.com/BlinkDL/RWKV-LM
Last week version 7 was released and every time they make significant improvements.
Re: Ask HN: Is anybody building an alternative transformer?
#43Yes. Happy to chat if u msg me. Using RL coupled with NNs to integrate search directly into inference instead of as an afterthought like Chain of though and test time training.
Are we able to "msg" people on here?
Only if they explicitly make the email public in the profile. It's hidden by default.
Re: Ask HN: Is anybody building an alternative transformer?
#44The MAMBA [1] model gained some traction as a potential successor. It's basically an RNN without the non linearity applied across hidden states, which makes it logarithmic time (instead of linear time) inference with a parallelizable scan [2]. It promises much faster inference with much lower compute costs, and I think up to 7B params, performs on par with transformers. I've yet to see a 40B+ model trained. The resea…
Oh that would be awesome for that to work. Thanks for sharing
If I'm not misremembering, Mistral released a model based on MAMBA, but I haven't heard much about it since.
Re: Ask HN: Is anybody building an alternative transformer?
#45Related: There was buzz last year about Kolmogorov Arnold Networks, and https://arxiv.org/abs/2409.10594 was claiming KANs perform better than standard MLPs in the transformer architecture. Does anyone know of these being explored in the LLM space? KANs seem to have better properties regarding memory if I'm not mistaken.
I believe KAN hype died off due to practical reasons (e.g. FLOPs from implementation) and empirical results, i.e. people reproduced KANs and they found the claims/results made in the original paper were misleading.
Here's a paper showing KANs are no better than MLPs, if anything they are typically worse when comparing fairly. https://arxiv.org/pdf/2407.16674
Re: Ask HN: Is anybody building an alternative transformer?
#46I’m working with a group on an RL core with models as tool use, for explainable agentic tasks with actual discovery.
Re: Ask HN: Is anybody building an alternative transformer?
#47Earlier quoted context omitted.
Not everyone is aware of the details of AI/ML, "transformer" is actually a specific term in the space that also overlaps with "transformer" in other fields adjacent to Software Development. This is when we all need to wear our empathy hat and remind ourselves that we exist in a bubble, so when we see an overloaded term, we should add even the most minimal context to help. OP could have added "AI/ML" in the title for…
Fair enough! Bubble or not, I certainly have very regularly (weekly?) seen headlines on hn about transformers for at least a few years now. Like how bitcoin used to be on hn frontpage every week for a couple years circa 2010 (to the derision of half of the commenters). Not everyone is in the crypto space, but they know what bitcoin is. Anyhow I suppose the existence of such questions on hn is evidence that I'm in mor…
I, too, haven't seen the word "transformer" outside an ML context in months. Didn't stop me from wondering if the OP meant the thing that changes voltage.
Re: Ask HN: Is anybody building an alternative transformer?
#48Personally I think foundation models are for the birds, the cost of developing one is immense and the time involved is so great that you can't do many run-break-fix cycles so you will get nowhere on a shoestring. (Though maybe you can get somewhere on simple tasks and synthetic data) Personally I am working on a reliable model trainer for classification and sequence labeling tasks that uses something like ModernBERT…
> Personally I think foundation models are for the birds,
I might not quite that far, but I have publicly said (and will stand by the statement) that I think that training progressively larger and more complex foundation models is a waste of resources. But my view of AI is rooted in a neuro-symbolic approach, with emphasis on the "symbolic". I envision neural networks not as the core essence of an AI, but mainly as just adapters between different representations that are used by different sub-systems. And possibly as "scaffolding" where one can use the "intelligence" baked into an LLM as a bridge to get the overall system to where it can learn, and then eventually kick the scaffold down once it isn't needed anymore.
Re: Ask HN: Is anybody building an alternative transformer?
#49Personally I think foundation models are for the birds, the cost of developing one is immense and the time involved is so great that you can't do many run-break-fix cycles so you will get nowhere on a shoestring. (Though maybe you can get somewhere on simple tasks and synthetic data) Personally I am working on a reliable model trainer for classification and sequence labeling tasks that uses something like ModernBERT…
> Personally I think foundation models are for the birds, I might not quite that far, but I have publicly said (and will stand by the statement) that I think that training progressively larger and more complex foundation models is a waste of resources. But my view of AI is rooted in a neuro-symbolic approach, with emphasis on the "symbolic". I envision neural networks not as the core essence of an AI, but mainly as j…
I can sure talk your ear off about that one as I went way too far into the semantic web rabbit hole.
Training LLMs to use 'tools' of various types is a great idea, as it is to run them inside frameworks that check that their output satisfies various constraints. Still certain problems like the NP-complete nature of SAT solving (and many intelligent systems problems, such as word problems you'd expect an A.I. to solve, boil down to SAT solving) and problems such as the halting problem, Godel's theorem and such are still problems. I understand Doug Hofstader has softened his positions lately, but I think many of the problems set up in this book
https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach
(particularly the Achilles & Tortoise dialog) still stand today, as cringey as that book seems to me in 2025.
Re: Ask HN: Is anybody building an alternative transformer?
#50Right now as long as the rocket's heading straight up, everyone's on board with MLPs (Multilayer Perceptrons/Transformers)! Why not stay on the same rocket for now!? We're almost at AGI already!
Why are you conflating MLPs in general with specifically transformers?
I consider MLPs the building blocks of all this, and is what makes things a neural net, as opposed to some other data structure.