Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Starting from scratch: Training a 30M Topological Transformer

tuned.org.uk

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Starting from scratch: Training a 30M Topological Transformer

#51

Post by tuned »

reactordev wrote: post #10
liteclient wrote: post #7

Earlier quoted context omitted.

it makes sense architecturally they replace dot-product attention with topology-based scalar distances derived from a laplacian embedding - that effectively reduces attention scoring to a 1D energy comparison which can save memory and compute that said, i’d treat the results with a grain of salt give there is no peer review, and benchmarks are only on 30M parameter model so far

Yup, keyword here is “under the right conditions”. This may work well for their use case but fail horribly in others without further peer review and testing.

no, from my point of view is being more domain-focused instead of going full-orthogonal.

Re: Starting from scratch: Training a 30M Topological Transformer

#52

Post by Yemoshino »

vixen99 wrote: post #25
postflopclarity wrote: post #15

Earlier quoted context omitted.

that's a strange way to spell "no, I didn't understand the paper"

Perhaps someone who does understand the paper will kindly make it a bit clearer for those of who get a bit lost.

Honestly why I would really apprechiate something like this, hn is not an explain platform.

For sure, some words or feedback on what you understood (did you get it right) etc. yeah.

But otherwise, if you do not understand a research paper, you have to do the same hard work as everyone else. Sitting down, going through it paragraph by paragraph and learning it. This takes massive time.

and for a high level overview, chatgpt and co are really really good getting papers.

Re: Starting from scratch: Training a 30M Topological Transformer

#53

Post by thesz »

nickpsecurity wrote: post #35
thesz wrote: post #31

Earlier quoted context omitted.

> like graph relationships Once upon a time during me being language modeling researcher I built and finetuned a big (at the time - about 5 billions parameters) Sparse Non-Negative Matrix Language Model [1]. [1] https://aclanthology.org/Q16-1024/ As this model allows for mix-and-match of various contexts, one thing that I did is to have a word-sorted context. This effectively transforms position-based context into a…

I've saved it to look at it in the future. I also remembered Kristina Tautanova's name (your editor). Looking up recent publications, she's done interesting work on analyzing pretraining mixtures. https://aclanthology.org/2025.acl-long.1564/ Thanks to you both for two, interesting papers tonight. :)

I am not an author of SNMLM paper. ;)

I was using their model in my work.

Re: Starting from scratch: Training a 30M Topological Transformer

#54

Post by Herring »

tuned wrote: post #39
naasking wrote: post #36

Earlier quoted context omitted.

The linked paper tested nanoGPT with this new transformer: https://www.techrxiv.org/users/685780/articles/1375955-topol...

thanks for linking. Yes the paper compares the new architecture (that is also a fork of my implementation of nanoGPT) with Karpathy's nanoGPT. There are also links to the code and bench used.

Note I didn't say Karpathy's nanoGPT, I said use the speedrun.

Transformers are universal function approximators. When well-tuned, they often start to approximate other innovations. Not always, thank god, but often enough that you have to be careful.

Re: Starting from scratch: Training a 30M Topological Transformer

#55

Post by nickpsecurity »

thesz wrote: post #53
nickpsecurity wrote: post #35

Earlier quoted context omitted.

I've saved it to look at it in the future. I also remembered Kristina Tautanova's name (your editor). Looking up recent publications, she's done interesting work on analyzing pretraining mixtures. https://aclanthology.org/2025.acl-long.1564/ Thanks to you both for two, interesting papers tonight. :)

I am not an author of SNMLM paper. ;) I was using their model in my work.

I misunderstood what you said.

Well, in your work, whay benefit did you get from it? And do you think it would be beneficial today combined with modern techniques? Or obsoleted by other technqiue?

(I ask because I'm finding many old techniques are still good or could be mixed with deep learning.)

Re: Starting from scratch: Training a 30M Topological Transformer

#56

Post by throwaway314155 »

tuned wrote: post #45
throwaway314155 wrote: post #28

Earlier quoted context omitted.

That doesn’t tell you if the new method continues to perform better at higher parameter counts.

it most-likely will in terms of performance as it uses 50% less memory (for sure it will at inference time that is the most used operation on web services), because it can leverage longer T and D if the design is confirmed and the quality of generation is comparable to other models. If this very basic assumption is correct, it means a lot of savings in electricity as the same GPUs can resolve more requests.

By performance, I meant the accuracy of the model, not the runtime/memory characteristics.

Re: Starting from scratch: Training a 30M Topological Transformer

#57

Post by thesz »

nickpsecurity wrote: post #55
thesz wrote: post #53

Earlier quoted context omitted.

I am not an author of SNMLM paper. ;) I was using their model in my work.

I misunderstood what you said. Well, in your work, whay benefit did you get from it? And do you think it would be beneficial today combined with modern techniques? Or obsoleted by other technqiue? (I ask because I'm finding many old techniques are still good or could be mixed with deep learning.)

At the time (2018), it had perplexity close to LSTM, while having more coefficients and much shorter (hours vs days) training time.

I tried to apply SNMLM's ideas to the byte-level prediction modeling here: https://github.com/thesz/snmlm-per-byte

It was not bad, but I had trouble scaling it to the 1B set. Mostly because I have not enough time.

I do hold same mindset as yours, that many old techniques are misunderstood or underapplied. For example, decision trees, in my experiments, allow for bit-length-per-byte comparable to LSTM (lstm-compress or LSTM in nncp experiments): https://github.com/thesz/codeta

Re: Starting from scratch: Training a 30M Topological Transformer

#58

Post by xtiansimon »

inimino wrote: post #33
xtiansimon wrote: post #11

Earlier quoted context omitted.

What is a "high-dimensional room"? A "room" is by definition three-dimensional in so far as we're using metaphor for description. Then to add this "high-dimensional" modifier does little for me, since the only visualizable high-dimensional cube is a tesseract, which still leaves you at 4-d. The presented counterpoint to this metaphor has the "room" change into a "landscape". The room is a "flat void" compared to a la…

The metaphor works only if you already understand the maths.

Maths I’ve never heard of. Possible. Probable. And what you’re saying is the words “room” and “landscape” are _over coded_ to such an extent the natural logic of 3-d rooms and 2-d landscapes are easily overcome by scaffolds of mathematical instruction—such that the latter could be imagined as having *higher* dimensionality than the former, for example? Or whatever other idea orbiting those words counter to their nature. That’s very interesting.

Re: Starting from scratch: Training a 30M Topological Transformer

#59

Post by tuned »

Herring wrote: post #54
tuned wrote: post #39

Earlier quoted context omitted.

thanks for linking. Yes the paper compares the new architecture (that is also a fork of my implementation of nanoGPT) with Karpathy's nanoGPT. There are also links to the code and bench used.

Note I didn't say Karpathy's nanoGPT, I said use the speedrun. Transformers are universal function approximators. When well-tuned, they often start to approximate other innovations. Not always, thank god, but often enough that you have to be careful.

ok, thanks. I am taking it slow then
Post reply on HN