Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Starting from scratch: Training a 30M Topological Transformer

tuned.org.uk

Post reply on HN
Print view View original

41–50 of 59 posts

Re: Starting from scratch: Training a 30M Topological Transformer

#41

Post by tuned »

lostmsu wrote: post #5
Lerc wrote: post #4

Earlier quoted context omitted.

I'm not sure if that is the right calculation. Provided the flops are not prohibitive. Output quality per model bytes might be better. In general people run the largest model they can. I certainly think trading speed for quality at the same size is worth looking at. Especially if it uses methods that can benefit from the efforts of others to improve speed in general. That said performance difference at 30M may not be…

So no comparison?

comparisons will be run when the quality of generation will be on pair with other available models. It is useless to have preformance if the quality is not at lease on par.

The paper runs a bench (code and bench in the paper) to compare the performance with a causal attention GPT-2 model (nanoGPT) at inference (20% faster) and at training (equivalent for T and D larger than a threshold).

Re: Starting from scratch: Training a 30M Topological Transformer

#42

Post by tuned »

amelius wrote: post #30
throwaway314155 wrote: post #28

Earlier quoted context omitted.

That doesn’t tell you if the new method continues to perform better at higher parameter counts.

Nor that the training from scratch will even work.

exactly, that is the current objective. To proove that generation for a specific domain is on-par with causal attention models

Re: Starting from scratch: Training a 30M Topological Transformer

#43

Post by tuned »

oofbey wrote: post #26
lostmsu wrote: post #2

Comparison with vanilla of the same size/flops budget?

The big bet with this technique is in having a fixed (non learned) matrix which converts the tokens latent space to the linear attention space. So you can kinda cheat and say your model is small because a bunch of the smarts are in this fixed big graph laplacian matrix L. So how do you scale this up from a toy problem? Well that L would Have to get bigger. And it’s hard to imagine it being useful if L is not trained.…

the idea is to have a lot of "narrow" models to work with RAG instead of one model for all the knowledge domains or also distil the metadata that is currently in enterprise Knowledge Graphs

Re: Starting from scratch: Training a 30M Topological Transformer

#45

Post by tuned »

throwaway314155 wrote: post #28
kouteiheika wrote: post #18

If you want to prove (i.e. show that it works and/or it's faster in a real-world scenario) a new alternative to attention without breaking the bank then one of the best ways to do that would probably be to retrain an already existing model, just with swapped attention modules. Then once you have such a model you can do apples-to-apples benchmarks. This has been done successfully in the past: https://huggingface.co/fe…

That doesn’t tell you if the new method continues to perform better at higher parameter counts.

it most-likely will in terms of performance as it uses 50% less memory (for sure it will at inference time that is the most used operation on web services), because it can leverage longer T and D if the design is confirmed and the quality of generation is comparable to other models. If this very basic assumption is correct, it means a lot of savings in electricity as the same GPUs can resolve more requests.

Re: Starting from scratch: Training a 30M Topological Transformer

#46

Post by tuned »

keyle wrote: post #3

Does this make any sense, to anyone?

it made sense to me as it is a very simple idea I guess: causal self-attention compute QKV distances computing on the full vectors for Q,K and V; the topological transformer can provide the same computation using Q, scalar K and V. Instead of [N², N², N²] -> [N², N, N²] is used. If generation is confirmed to be on par in terms of quality, the gains are evident.

Re: Starting from scratch: Training a 30M Topological Transformer

#47

Post by tuned »

Yemoshino wrote: post #16
ashirviskas wrote: post #8

I wonder what if we just crammed more into the "tokens"? I am running an experiment of replacing discrete tokens with embeddings + small byte encoder/decoder. That way you can use embedding space much more efficiently and have it contain much more nuance. Experiments I want to build on top of it: 1. Adding lsp context to the embeddings - that way the model could _see_ the syntax better, closer to how we use IDEs and…

I found a few papers in this direction with perplexity like this one https://ceur-ws.org/Vol-4005/paper1.pdf and it doesn't seem to be that relevant for now. The progress of a handful models seem to be so much better (because limited compute, we have only a handful of big ones, i presume) that these finetunings are just not yet relevant. I'm also curious if a english java + html + css + javascript only model would lo…

if you have a corpus of code snippets to train the manifold (Laplacian) on (and a good embedding model), it is definitely possible to try something like this.

Re: Starting from scratch: Training a 30M Topological Transformer

#48

Post by tuned »

oofbey wrote: post #26
lostmsu wrote: post #2

Comparison with vanilla of the same size/flops budget?

The big bet with this technique is in having a fixed (non learned) matrix which converts the tokens latent space to the linear attention space. So you can kinda cheat and say your model is small because a bunch of the smarts are in this fixed big graph laplacian matrix L. So how do you scale this up from a toy problem? Well that L would Have to get bigger. And it’s hard to imagine it being useful if L is not trained.…

also: precomputing a sparse Laplacian for N vectors at dimension D (NxD) is infinitely cheaper (if using `arrowspace`, my previous paper) than computing distances on the same full dense vectors billions of times. There are published tests that compute a Laplacian on 300Kx384 space in 500 secs on a laptop on CPU. So it is a trade-off: potentially few minutes of pretaining or hours of dot-product on dense matrices

Re: Starting from scratch: Training a 30M Topological Transformer

#49

Post by tuned »

kannanvijayan wrote: post #6
keyle wrote: post #3

Does this make any sense, to anyone?

I think this is an attempt to try to enrich the locality model in transformers. One of the weird things you do in transformers is add a position vector which captures the distance between the token being attended to the some other token. This is obviously not powerful enough to express non-linear relationships - like graph relationships. This person seems to be experimenting with doing pre-processing of the input tok…

> This is obviously not powerful enough to express non-linear relationships - like graph relationships.

the distance metrics used is based on energy-informed graphs that encode energy relations in a distribution called taumode, see my previous paper on spectral indexing for vector databases for a complete roll-out

Re: Starting from scratch: Training a 30M Topological Transformer

#50

Post by tuned »

liteclient wrote: post #7
keyle wrote: post #3

Does this make any sense, to anyone?

it makes sense architecturally they replace dot-product attention with topology-based scalar distances derived from a laplacian embedding - that effectively reduces attention scoring to a 1D energy comparison which can save memory and compute that said, i’d treat the results with a grain of salt give there is no peer review, and benchmarks are only on 30M parameter model so far

right. this is a proposal that needs to be tested. I started testing it on 30M parameters then I will move to a 100M and evaluate the generation on domain-specific assisting tasks
Post reply on HN