Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Starting from scratch: Training a 30M Topological Transformer

tuned.org.uk

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Starting from scratch: Training a 30M Topological Transformer

#31

Post by thesz »

kannanvijayan wrote: post #6
keyle wrote: post #3

Does this make any sense, to anyone?

I think this is an attempt to try to enrich the locality model in transformers. One of the weird things you do in transformers is add a position vector which captures the distance between the token being attended to the some other token. This is obviously not powerful enough to express non-linear relationships - like graph relationships. This person seems to be experimenting with doing pre-processing of the input tok…

  > like graph relationships
Once upon a time during me being language modeling researcher I built and finetuned a big (at the time - about 5 billions parameters) Sparse Non-Negative Matrix Language Model [1].

[1] https://aclanthology.org/Q16-1024/

As this model allows for mix-and-match of various contexts, one thing that I did is to have a word-sorted context. This effectively transforms position-based context into a word-set based context, where "you and me", "me and you" and "and me you" are the same.

This allowed for longer contexts and better prediction.

Re: Starting from scratch: Training a 30M Topological Transformer

#32

Post by thesz »

adroniser wrote: post #12
kannanvijayan wrote: post #6

Earlier quoted context omitted.

I think this is an attempt to try to enrich the locality model in transformers. One of the weird things you do in transformers is add a position vector which captures the distance between the token being attended to the some other token. This is obviously not powerful enough to express non-linear relationships - like graph relationships. This person seems to be experimenting with doing pre-processing of the input tok…

Adding the position vector is basic sure, but it's naive to think the model doesn't develop its own positional system bootstrapping on top of the barebones one.

For some reason people are still adding position encodings into embeddings.

As if they are not relying on the model's ability to develop its own "positional system bootstrapping on top of the barebones one."

Re: Starting from scratch: Training a 30M Topological Transformer

#33

Post by inimino »

xtiansimon wrote: post #11
geoffbp wrote: post #9

I dug into this a bit (with AI ofc) and it spat this out. I found it an easy way to visualise and start to understand: > Standard AI models (like GPT-4) treat data using Global Geometry. They imagine every word as a point floating in a massive, flat, high-dimensional room. To see how two words relate, they draw a straight line between them. > Local Topology changes the "room" into a landscape (a manifold). Instead of…

What is a "high-dimensional room"? A "room" is by definition three-dimensional in so far as we're using metaphor for description. Then to add this "high-dimensional" modifier does little for me, since the only visualizable high-dimensional cube is a tesseract, which still leaves you at 4-d. The presented counterpoint to this metaphor has the "room" change into a "landscape". The room is a "flat void" compared to a la…

The metaphor works only if you already understand the maths.

Re: Starting from scratch: Training a 30M Topological Transformer

#34

Post by nickpsecurity »

Herring wrote: post #27
kouteiheika wrote: post #18

If you want to prove (i.e. show that it works and/or it's faster in a real-world scenario) a new alternative to attention without breaking the bank then one of the best ways to do that would probably be to retrain an already existing model, just with swapped attention modules. Then once you have such a model you can do apples-to-apples benchmarks. This has been done successfully in the past: https://huggingface.co/fe…

I'd say try the nanogpt speedrun. It's much easier to train, and gives you a better comparison vs optimized systems. https://github.com/KellerJordan/modded-nanogpt

Labs were also competing to train BERT's for $20 or less. People still use them a lot, too.

https://www.databricks.com/blog/mosaicbert

I'll add they should do a number of small, training runs with different architectures and data mixes. That proves generalization.

Re: Starting from scratch: Training a 30M Topological Transformer

#35

Post by nickpsecurity »

thesz wrote: post #31
kannanvijayan wrote: post #6

Earlier quoted context omitted.

I think this is an attempt to try to enrich the locality model in transformers. One of the weird things you do in transformers is add a position vector which captures the distance between the token being attended to the some other token. This is obviously not powerful enough to express non-linear relationships - like graph relationships. This person seems to be experimenting with doing pre-processing of the input tok…

> like graph relationships Once upon a time during me being language modeling researcher I built and finetuned a big (at the time - about 5 billions parameters) Sparse Non-Negative Matrix Language Model [1]. [1] https://aclanthology.org/Q16-1024/ As this model allows for mix-and-match of various contexts, one thing that I did is to have a word-sorted context. This effectively transforms position-based context into a…

I've saved it to look at it in the future. I also remembered Kristina Tautanova's name (your editor). Looking up recent publications, she's done interesting work on analyzing pretraining mixtures.

https://aclanthology.org/2025.acl-long.1564/

Thanks to you both for two, interesting papers tonight. :)

Re: Starting from scratch: Training a 30M Topological Transformer

#36

Post by naasking »

Herring wrote: post #27
kouteiheika wrote: post #18

If you want to prove (i.e. show that it works and/or it's faster in a real-world scenario) a new alternative to attention without breaking the bank then one of the best ways to do that would probably be to retrain an already existing model, just with swapped attention modules. Then once you have such a model you can do apples-to-apples benchmarks. This has been done successfully in the past: https://huggingface.co/fe…

I'd say try the nanogpt speedrun. It's much easier to train, and gives you a better comparison vs optimized systems. https://github.com/KellerJordan/modded-nanogpt

The linked paper tested nanoGPT with this new transformer:

https://www.techrxiv.org/users/685780/articles/1375955-topol...

Re: Starting from scratch: Training a 30M Topological Transformer

#37

Post by ashirviskas »

nl wrote: post #29
ashirviskas wrote: post #8

I wonder what if we just crammed more into the "tokens"? I am running an experiment of replacing discrete tokens with embeddings + small byte encoder/decoder. That way you can use embedding space much more efficiently and have it contain much more nuance. Experiments I want to build on top of it: 1. Adding lsp context to the embeddings - that way the model could _see_ the syntax better, closer to how we use IDEs and…

Isn't this just an awkward way of adding an extra layer to the NN, except without end-to-end training? Models like Stable Diffusion sort of do a similar thing using Clip embeddings. It works, and it's an easy way to benefit from the pre-training Clip has. But for a language model it would seemingly make more sense to just add the extra layer.

I mean this is exactly what it is. Just a wrapper to replace the tokenizer. That is exactly how LLMs can read images.

I'm just focusing on different parts

Re: Starting from scratch: Training a 30M Topological Transformer

#38

Post by tuned »

kouteiheika wrote: post #18

If you want to prove (i.e. show that it works and/or it's faster in a real-world scenario) a new alternative to attention without breaking the bank then one of the best ways to do that would probably be to retrain an already existing model, just with swapped attention modules. Then once you have such a model you can do apples-to-apples benchmarks. This has been done successfully in the past: https://huggingface.co/fe…

thanks for reading. I cannot retrain an existing model as the self-attention mechanism has been completely redesigned. The Keys and Values in self-attention are stored as scalars, so a latent space with traditional weights does not make sense if used in the context of a topological transformer. The two latent spaces would be somehow equivalent eventually but they would store totally different values.

Re: Starting from scratch: Training a 30M Topological Transformer

#39

Post by tuned »

naasking wrote: post #36
Herring wrote: post #27

Earlier quoted context omitted.

I'd say try the nanogpt speedrun. It's much easier to train, and gives you a better comparison vs optimized systems. https://github.com/KellerJordan/modded-nanogpt

The linked paper tested nanoGPT with this new transformer: https://www.techrxiv.org/users/685780/articles/1375955-topol...

thanks for linking.

Yes the paper compares the new architecture (that is also a fork of my implementation of nanoGPT) with Karpathy's nanoGPT. There are also links to the code and bench used.

Re: Starting from scratch: Training a 30M Topological Transformer

#40

Post by tuned »

andai wrote: post #24
kouteiheika wrote: post #18

If you want to prove (i.e. show that it works and/or it's faster in a real-world scenario) a new alternative to attention without breaking the bank then one of the best ways to do that would probably be to retrain an already existing model, just with swapped attention modules. Then once you have such a model you can do apples-to-apples benchmarks. This has been done successfully in the past: https://huggingface.co/fe…

This is interesting. Has there been more research into this architecture? I hear about it once every few years but it always seems like a niche / experimental thing. But based on the graph in their blog post you'd expect every company to be using this.

This is a novel re-interpretation of the Transformer, based on my previous research made with a library called `arrowspace`.

It is somehow what is called a "Grassmann-like flow" but without the Plucker embedding, or also similar to what is done in DavisTensor but relying on spectral Laplacian instead of purely geometric distances.

The problem with a lot of stuff done before is that it focuses on dense representations. This architecture is focuses on sparse representation and provides a new approximation computation based on energy-informed graphs.

Post reply on HN