Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is anybody building an alternative transformer?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 96 posts

Re: Ask HN: Is anybody building an alternative transformer?

#51

Post by dr_dshiv »

mindcrime wrote: post #48
PaulHoule wrote: post #23

Personally I think foundation models are for the birds, the cost of developing one is immense and the time involved is so great that you can't do many run-break-fix cycles so you will get nowhere on a shoestring. (Though maybe you can get somewhere on simple tasks and synthetic data) Personally I am working on a reliable model trainer for classification and sequence labeling tasks that uses something like ModernBERT…

> Personally I think foundation models are for the birds, I might not quite that far, but I have publicly said (and will stand by the statement) that I think that training progressively larger and more complex foundation models is a waste of resources. But my view of AI is rooted in a neuro-symbolic approach, with emphasis on the "symbolic". I envision neural networks not as the core essence of an AI, but mainly as j…

Good old fashioned AI, amirite

Re: Ask HN: Is anybody building an alternative transformer?

#52

Post by quantadev »

cshimmin wrote: post #29
quantadev wrote: post #18

Right now as long as the rocket's heading straight up, everyone's on board with MLPs (Multilayer Perceptrons/Transformers)! Why not stay on the same rocket for now!? We're almost at AGI already!

I wouldn't conflate MLPs with transformers, MLP is a small building block of almost any standard neural architecture (excluding spiking/neuromorphic types). But to your point, the trend towards increasing inference-time compute costs, being ushered by CoT/reasoning models is one good reason to look for equally capable models that can be optimized for inference efficiency. Traditionally training was the main compute c…

What I meant by "NNs and Transformers" is that once we've found the magical ingredient (and we've found it) people tend to all be focused in the same area of research. Mankind just got kinda lucky that all this can run on essentially game graphics boards!

Re: Ask HN: Is anybody building an alternative transformer?

#53

Post by dr_dshiv »

janalsncm wrote: post #8

There are alternatives that optimize around the edges. Like Deepseek’s Multi-head Latent Attention, or Grouped Query Attention. DeepSeek also showed an optimization on Mixture of Experts. These are all clear improvements to the Vaswani architecture. There are optimizations like extreme 1.58 bit quant that can be applied to anything. There are architectures that stray farther. Like SSMs and some attempts at bringing t…

Mixture of depths, too.

Re: Ask HN: Is anybody building an alternative transformer?

#54

Post by kla-s »

czhu12 wrote: post #22

The MAMBA [1] model gained some traction as a potential successor. It's basically an RNN without the non linearity applied across hidden states, which makes it logarithmic time (instead of linear time) inference with a parallelizable scan [2]. It promises much faster inference with much lower compute costs, and I think up to 7B params, performs on par with transformers. I've yet to see a 40B+ model trained. The resea…

Jamba 1.5 Large is 398B params (94B active) and weights are available.

https://arxiv.org/abs/2408.12570

Credit https://news.ycombinator.com/user?id=sanxiyn for making me aware

Re: Ask HN: Is anybody building an alternative transformer?

#55

Post by tlb »

mindcrime wrote: post #48
PaulHoule wrote: post #23

Personally I think foundation models are for the birds, the cost of developing one is immense and the time involved is so great that you can't do many run-break-fix cycles so you will get nowhere on a shoestring. (Though maybe you can get somewhere on simple tasks and synthetic data) Personally I am working on a reliable model trainer for classification and sequence labeling tasks that uses something like ModernBERT…

> Personally I think foundation models are for the birds, I might not quite that far, but I have publicly said (and will stand by the statement) that I think that training progressively larger and more complex foundation models is a waste of resources. But my view of AI is rooted in a neuro-symbolic approach, with emphasis on the "symbolic". I envision neural networks not as the core essence of an AI, but mainly as j…

We learned something pretty big and surprising from each new generation of LLM, for a small fraction of the time and cost of a new particle accelerator or space telescope. Compared to other big science projects, they're giving pretty good bang for the buck.

Re: Ask HN: Is anybody building an alternative transformer?

#57

Post by mindcrime »

PaulHoule wrote: post #49
mindcrime wrote: post #48

Earlier quoted context omitted.

> Personally I think foundation models are for the birds, I might not quite that far, but I have publicly said (and will stand by the statement) that I think that training progressively larger and more complex foundation models is a waste of resources. But my view of AI is rooted in a neuro-symbolic approach, with emphasis on the "symbolic". I envision neural networks not as the core essence of an AI, but mainly as j…

I can sure talk your ear off about that one as I went way too far into the semantic web rabbit hole. Training LLMs to use 'tools' of various types is a great idea, as it is to run them inside frameworks that check that their output satisfies various constraints. Still certain problems like the NP-complete nature of SAT solving (and many intelligent systems problems, such as word problems you'd expect an A.I. to solve…

As somebody who considers himself something of a Semantic Web enthusiast / advocate, and has also read GEB, I can totally relate. To me, this is really one of those "THE ISSUE" things: how can we use some notion of formal logic to solve problems, without being forced to give up hope due to incompleteness and/or the Halting Problem. Clearly you have to give up something as a tradeoff for making this stuff tractable, but I suppose it's an open question what you can tradeoff and how exactly that factors into the algorithm, as well as what guarantees (if any) remain...

Re: Ask HN: Is anybody building an alternative transformer?

#58

Post by mindcrime »

dr_dshiv wrote: post #51
mindcrime wrote: post #48

Earlier quoted context omitted.

> Personally I think foundation models are for the birds, I might not quite that far, but I have publicly said (and will stand by the statement) that I think that training progressively larger and more complex foundation models is a waste of resources. But my view of AI is rooted in a neuro-symbolic approach, with emphasis on the "symbolic". I envision neural networks not as the core essence of an AI, but mainly as j…

Good old fashioned AI, amirite

Well, to the extent that people equate GOFAI with purely symbolic / logic-based processing, then no, not for my money anyway. I think it's possible to construct systems that use elements of symbolic processing along with sub-symbolic approaches and get useful results. I think of it as (although this is something of an over-simplification) taking symbolic reasoning, relaxing some of the constraints that go along with the guarantees that method makes out the outputs, and accepting a (hopefully only slightly) less desirable output. OR, think about flipping the whole thing around, get an output from, say, an LLM where there might be hallucination(s), and then use a symbolic reasoning system to post-process the output to ensure veracity before sending it to the user. Amazon has done some work along those lines, for example. https://aws.amazon.com/blogs/machine-learning/reducing-hallu...

Anyway this is all somewhat speculative, and I don't want to overstate the "weight" of anything I seem to be claiming here. This is just the direction my interests and inclinations have taken me in.

Post reply on HN