Ask HN: Is anybody building an alternative transformer? • Hacker News Board

Re: Ask HN: Is anybody building an alternative transformer?

#91

Post by NewUser76312 » Sat, Feb 15, 2025, 4:48 PM UTC

PaulHoule wrote: post #23

Personally I think foundation models are for the birds, the cost of developing one is immense and the time involved is so great that you can't do many run-break-fix cycles so you will get nowhere on a shoestring. (Though maybe you can get somewhere on simple tasks and synthetic data) Personally I am working on a reliable model trainer for classification and sequence labeling tasks that uses something like ModernBERT…

Yeah I've been wondering how one can contribute and build in the LLM and AI world without the resources to work on foundation models.

Because personally I'm not a product/GPT wrapper person - it just doesn't suit my interests.

So then what can one do that's meaningful and valuable? Probably something around finetuning?

Re: Ask HN: Is anybody building an alternative transformer?

#92

Post by londons_explore » Sat, Feb 15, 2025, 6:42 PM UTC

AustinDev wrote: post #82

londons_explore wrote: post #81

There are a bunch of promising other ways to convert AC voltages. The main one is the observation that the transformer uses an amount of copper and steel proportional to the power transmitted but inversely proportional to the frequency of operation. The copper and steel cost of a transformer is the main cost (multiplied by the cost of capital for the 100+ years it will operate). So if you can use solid state electron…

Is this response AI generated or are you lost?

The poster didn't specify which type of transformer...

Re: Ask HN: Is anybody building an alternative transformer?

#93

Post by seydor » Sat, Feb 15, 2025, 6:59 PM UTC

Perhaps people feel that the problem of modeling long-range token relationships has been solved. The problem is now how to get this model to produce tokens that are valid and ingenious solutions to problems, with RL or otherwise.

Re: Ask HN: Is anybody building an alternative transformer?

#94

Post by imtringued » Sat, Feb 15, 2025, 7:08 PM UTC

czhu12 wrote: post #22

The MAMBA [1] model gained some traction as a potential successor. It's basically an RNN without the non linearity applied across hidden states, which makes it logarithmic time (instead of linear time) inference with a parallelizable scan [2]. It promises much faster inference with much lower compute costs, and I think up to 7B params, performs on par with transformers. I've yet to see a 40B+ model trained. The resea…

Mamba isn't really a competitor to transformers. Quadratic attention exists for a reason.

Mamba's strengths lie in being a better RNN as you said. Mamba is probably better than transformers for things like object permanence over a sequence of inputs, where each input is an image, for example.

However, it would still make sense for a transformer to actually process the image by cutting it up into patches and then performing quadratic attention on that and then feeding the transformer input into mamba to get the actual output e.g. a robot action while maintaining object permanence.

Re: Ask HN: Is anybody building an alternative transformer?

#95

Post by Jotalea » Mon, Feb 17, 2025, 12:05 AM UTC

Yes, I am building one.

Is it an alternative? Yes.

Is it better? Hell no.

Re: Ask HN: Is anybody building an alternative transformer?

#96

Post by marshughes » Mon, Feb 17, 2025, 2:41 AM UTC

Absolutely! The QNN architecture based on quantum computing concepts shows great potential. It breaks through traditional computing models and may outperform Transformers in complex tasks. Do you have any research on the combination of quantum computing and AI?