Ask HN: Is anybody building an alternative transformer? • Hacker News Board

Re: Ask HN: Is anybody building an alternative transformer?

#11

Post by bob1029 » Fri, Feb 14, 2025, 8:30 PM UTC

nextos wrote: post #7

The xLSTM could become a good alternative to transformers: https://arxiv.org/abs/2405.04517 . On very long contexts, such as those arising in DNA models, these models perform really well. There's a big state-space model comeback initiated by the S3-Mamba saga. RWKV, which is a hybrid between classical RNNs and transformers, is also worth mentioning.

I was just about to post this. There was a MLST podcast about it a few days ago:

https://www.youtube.com/watch?v=8u2pW2zZLCs

Lots of related papers referenced in the description.

Re: Ask HN: Is anybody building an alternative transformer?

#12

Post by perrygeo » Fri, Feb 14, 2025, 8:33 PM UTC

happytoexplain wrote: post #6

I hate that popular domains take ownership of highly generic words. Many years ago, I struggled for a while to understand that when people say "frontend" they often mean a website frontend, even without any further context.

The worst offender is "feature". In my domain (ML and geo) we have three definitions.

Feature could be referring to some addition to the user-facing product, a raster input to machine learning, or a vector entity in GeoJSON. Context is the only tool we have to make the distinction, it gets really confusing when you're working on features that involve querying the features with features.

Re: Ask HN: Is anybody building an alternative transformer?

#13

Post by ai-christianson » Fri, Feb 14, 2025, 8:37 PM UTC

Not an alternative transformer like you asked for, but OptiLLM looks interesting for squeezing more juice out of existing LLMs.

Re: Ask HN: Is anybody building an alternative transformer?

#14

Post by ipunchghosts » Fri, Feb 14, 2025, 8:40 PM UTC

Yes. Happy to chat if u msg me. Using RL coupled with NNs to integrate search directly into inference instead of as an afterthought like Chain of though and test time training.

Re: Ask HN: Is anybody building an alternative transformer?

#15

Post by htrp » Fri, Feb 14, 2025, 8:40 PM UTC

Anyone know what the rwkv people are up to now?

https://arxiv.org/abs/2305.13048

Re: Ask HN: Is anybody building an alternative transformer?

#16

Post by herpdyderp » Fri, Feb 14, 2025, 8:42 PM UTC

pestatije wrote: post #2

please define transformer

Until I read this comment I thought we were talking about https://en.wikipedia.org/wiki/Transformer and I was very confused...

Re: Ask HN: Is anybody building an alternative transformer?

#17

Post by cshimmin » Fri, Feb 14, 2025, 8:43 PM UTC

janalsncm wrote: post #9

pestatije wrote: post #2

please define transformer

https://en.m.wikipedia.org/wiki/Transformer_(deep_learning_a...

Yeah, it's literally the most important practical development in AI/ML of the decade. This is like reading an article (or headline, more like) on HN and saying "please define git".

Re: Ask HN: Is anybody building an alternative transformer?

#18

Post by quantadev » Fri, Feb 14, 2025, 8:46 PM UTC

Right now as long as the rocket's heading straight up, everyone's on board with MLPs (Multilayer Perceptrons/Transformers)! Why not stay on the same rocket for now!? We're almost at AGI already!

Re: Ask HN: Is anybody building an alternative transformer?

#19

Post by sgt101 » Fri, Feb 14, 2025, 8:50 PM UTC

I'll see your architectural innovation and raise you a loss function revolution.

https://arxiv.org/pdf/2412.21149

Re: Ask HN: Is anybody building an alternative transformer?

#20

Post by mvieira38 » Fri, Feb 14, 2025, 8:50 PM UTC

Related: There was buzz last year about Kolmogorov Arnold Networks, and https://arxiv.org/abs/2409.10594 was claiming KANs perform better than standard MLPs in the transformer architecture. Does anyone know of these being explored in the LLM space? KANs seem to have better properties regarding memory if I'm not mistaken.