Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is anybody building an alternative transformer?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 96 posts

Re: Ask HN: Is anybody building an alternative transformer?

#61

Post by solresol »

I tried... it started with the idea was that log loss might not be the best option for training, and maybe it should be a loss related to how wrong the predicted word was. Predicting "dog" instead of "cat" should be less penalised than predicting "running".

That turns out to be an ultrametric loss, and the derivative of an ultrametric loss is zero in a large region around any local minimum, so it can't be trained by gradient descent -- it has to be trained by search.

Punchline: it's about one million times less effective than a more traditional architecture. https://github.com/solresol/ultratree-results

Re: Ask HN: Is anybody building an alternative transformer?

#62

Post by jmward01 »

I have an internal repo that does guided window attn. I figured out that One Weird Trick to get the model to learn how to focus so that you can move a fixed window around instead of full attn. I also built NNMemory (but that appears to be an idea others hae had now too [1]) and I have a completely bonkers mechanism for non-determanistic exit logic so that the model can spin until it thinks it has a good answer. I also built scale free connections between layers to completely remove residual connections. Plus some crazy things on sacrificial training (adding parameters that are removed after training in order to boost training performance with no prod penalty). There are more crazy things I have built but they aren't out there in the wild, yet. Some of the things I have built are in my repo. [2] I personally think we can get .5b models to outperform 8b+ SOTA models out there today (even the reasoning models coming out now)

The basic transformer block has been good at kicking things off, but it is now holding us back. We need to move to recurrent architectures again and switch to fixed guided attn windows + 'think' only layers like NNMemory. Attn is distracting and we know this as humans because we often close our eyes when we think hard about a problem on the page in front of us.

[1] https://arxiv.org/abs/2502.06049

[2] https://github.com/jmward01/lmplay

Re: Ask HN: Is anybody building an alternative transformer?

#63

Post by drdeca »

quantadev wrote: post #50
drdeca wrote: post #33

Earlier quoted context omitted.

Why are you conflating MLPs in general with specifically transformers?

I consider MLPs the building blocks of all this, and is what makes things a neural net, as opposed to some other data structure.

Sure. But that isn’t a reason to conflate the two?

OP wasn’t suggesting looking for an alternative/successor to MLPs, but for an alternative/successor to transformers (while presumably still using MLPs) in the same way that transformers are an alternative/successor to LSTMs.

Re: Ask HN: Is anybody building an alternative transformer?

#65

Post by throwawaymaths »

PaulHoule wrote: post #49
mindcrime wrote: post #48

Earlier quoted context omitted.

> Personally I think foundation models are for the birds, I might not quite that far, but I have publicly said (and will stand by the statement) that I think that training progressively larger and more complex foundation models is a waste of resources. But my view of AI is rooted in a neuro-symbolic approach, with emphasis on the "symbolic". I envision neural networks not as the core essence of an AI, but mainly as j…

I can sure talk your ear off about that one as I went way too far into the semantic web rabbit hole. Training LLMs to use 'tools' of various types is a great idea, as it is to run them inside frameworks that check that their output satisfies various constraints. Still certain problems like the NP-complete nature of SAT solving (and many intelligent systems problems, such as word problems you'd expect an A.I. to solve…

i am hoping for an slm "turing tape" small language model where the tokens are instructions for a copycat engine

Re: Ask HN: Is anybody building an alternative transformer?

#67

Post by PaulHoule »

mindcrime wrote: post #57
PaulHoule wrote: post #49

Earlier quoted context omitted.

I can sure talk your ear off about that one as I went way too far into the semantic web rabbit hole. Training LLMs to use 'tools' of various types is a great idea, as it is to run them inside frameworks that check that their output satisfies various constraints. Still certain problems like the NP-complete nature of SAT solving (and many intelligent systems problems, such as word problems you'd expect an A.I. to solve…

As somebody who considers himself something of a Semantic Web enthusiast / advocate, and has also read GEB, I can totally relate. To me, this is really one of those "THE ISSUE" things: how can we use some notion of formal logic to solve problems, without being forced to give up hope due to incompleteness and/or the Halting Problem. Clearly you have to give up something as a tradeoff for making this stuff tractable, b…

I would start with the fact that there is nothing consistent or complete about humans. Penrose's argument that he is a thetan because he can do math doesn't hold water.

Re: Ask HN: Is anybody building an alternative transformer?

#68

Post by [deleted] »

mindcrime wrote: post #48
PaulHoule wrote: post #23

Personally I think foundation models are for the birds, the cost of developing one is immense and the time involved is so great that you can't do many run-break-fix cycles so you will get nowhere on a shoestring. (Though maybe you can get somewhere on simple tasks and synthetic data) Personally I am working on a reliable model trainer for classification and sequence labeling tasks that uses something like ModernBERT…

> Personally I think foundation models are for the birds, I might not quite that far, but I have publicly said (and will stand by the statement) that I think that training progressively larger and more complex foundation models is a waste of resources. But my view of AI is rooted in a neuro-symbolic approach, with emphasis on the "symbolic". I envision neural networks not as the core essence of an AI, but mainly as j…

[deleted]

Re: Ask HN: Is anybody building an alternative transformer?

#69

Post by dr_dshiv »

mindcrime wrote: post #58
dr_dshiv wrote: post #51

Earlier quoted context omitted.

Good old fashioned AI, amirite

Well, to the extent that people equate GOFAI with purely symbolic / logic-based processing, then no, not for my money anyway. I think it's possible to construct systems that use elements of symbolic processing along with sub-symbolic approaches and get useful results. I think of it as (although this is something of an over-simplification) taking symbolic reasoning, relaxing some of the constraints that go along with…

Maybe gen AI coding is neurosymbolic AI, realized differently than expected

Re: Ask HN: Is anybody building an alternative transformer?

#70

Post by aqueueaqueue »

perrygeo wrote: post #12
happytoexplain wrote: post #6

I hate that popular domains take ownership of highly generic words. Many years ago, I struggled for a while to understand that when people say "frontend" they often mean a website frontend, even without any further context.

The worst offender is "feature". In my domain (ML and geo) we have three definitions. Feature could be referring to some addition to the user-facing product, a raster input to machine learning, or a vector entity in GeoJSON. Context is the only tool we have to make the distinction, it gets really confusing when you're working on features that involve querying the features with features.

Inference has 2 meanings too
Post reply on HN