Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Starting from scratch: Training a 30M Topological Transformer

tuned.org.uk

Post reply on HN
Print view View original

21–30 of 59 posts

Re: Starting from scratch: Training a 30M Topological Transformer

#21

Post by stephantul »

ashirviskas wrote: post #8

I wonder what if we just crammed more into the "tokens"? I am running an experiment of replacing discrete tokens with embeddings + small byte encoder/decoder. That way you can use embedding space much more efficiently and have it contain much more nuance. Experiments I want to build on top of it: 1. Adding lsp context to the embeddings - that way the model could _see_ the syntax better, closer to how we use IDEs and…

There’s many examples of noisily encoding a large embedding vocabulary. This sounds a bit like T-free or H-net? Or BLT?

One of the main issues with lines of work around this are that you end up trading embedding parameters for active parameters. This is rarely a good trade-off for the sake of compute.

Re: Starting from scratch: Training a 30M Topological Transformer

#22

Post by ashirviskas »

appplication wrote: post #14
ashirviskas wrote: post #8

I wonder what if we just crammed more into the "tokens"? I am running an experiment of replacing discrete tokens with embeddings + small byte encoder/decoder. That way you can use embedding space much more efficiently and have it contain much more nuance. Experiments I want to build on top of it: 1. Adding lsp context to the embeddings - that way the model could _see_ the syntax better, closer to how we use IDEs and…

Not an expert in the space, but I’m not sure you need to modify tokens to get the model to see syntax, you basically get that exact association from attention.

You get that association that is relevant to your project only if you can cram the whole codebase. Otherwise it is making rough estimates and some of the time that seems to be where the models fail.

It can only be fully resolved with either infinite context length, or doing it similar to how humans do it - add some LSP "color" to the code tokens.

You can get a feel of what LLMs deal with when you try opening 3000 lines of code in a simple text editor and try to do something. May work for simple fixes, but not whole codebase refactors. Only ultra skilled humans can be productive in it (using my subjective definition of "productive")

Re: Starting from scratch: Training a 30M Topological Transformer

#24

Post by andai »

kouteiheika wrote: post #18

If you want to prove (i.e. show that it works and/or it's faster in a real-world scenario) a new alternative to attention without breaking the bank then one of the best ways to do that would probably be to retrain an already existing model, just with swapped attention modules. Then once you have such a model you can do apples-to-apples benchmarks. This has been done successfully in the past: https://huggingface.co/fe…

This is interesting. Has there been more research into this architecture? I hear about it once every few years but it always seems like a niche / experimental thing. But based on the graph in their blog post you'd expect every company to be using this.

Re: Starting from scratch: Training a 30M Topological Transformer

#25

Post by vixen99 »

postflopclarity wrote: post #15
pwndByDeath wrote: post #13

Earlier quoted context omitted.

No, its a new form of alchemy that turns electricity into hype. The technical jargon is more.of.a thieves cant to help identity other conmen to one another

that's a strange way to spell "no, I didn't understand the paper"

Perhaps someone who does understand the paper will kindly make it a bit clearer for those of who get a bit lost.

Re: Starting from scratch: Training a 30M Topological Transformer

#26

Post by oofbey »

lostmsu wrote: post #2

Comparison with vanilla of the same size/flops budget?

The big bet with this technique is in having a fixed (non learned) matrix which converts the tokens latent space to the linear attention space. So you can kinda cheat and say your model is small because a bunch of the smarts are in this fixed big graph laplacian matrix L.

So how do you scale this up from a toy problem? Well that L would Have to get bigger. And it’s hard to imagine it being useful if L is not trained. Then it starts to look a lot more like a conventional transformer, but probably harder to train, with the benefit of smaller KV caches. (Half the size - not a massive win.)

So overall doesn’t seem to me like it’s gonna amount to anything.

Re: Starting from scratch: Training a 30M Topological Transformer

#27

Post by Herring »

kouteiheika wrote: post #18

If you want to prove (i.e. show that it works and/or it's faster in a real-world scenario) a new alternative to attention without breaking the bank then one of the best ways to do that would probably be to retrain an already existing model, just with swapped attention modules. Then once you have such a model you can do apples-to-apples benchmarks. This has been done successfully in the past: https://huggingface.co/fe…

I'd say try the nanogpt speedrun. It's much easier to train, and gives you a better comparison vs optimized systems.

https://github.com/KellerJordan/modded-nanogpt

Re: Starting from scratch: Training a 30M Topological Transformer

#28

Post by throwaway314155 »

kouteiheika wrote: post #18

If you want to prove (i.e. show that it works and/or it's faster in a real-world scenario) a new alternative to attention without breaking the bank then one of the best ways to do that would probably be to retrain an already existing model, just with swapped attention modules. Then once you have such a model you can do apples-to-apples benchmarks. This has been done successfully in the past: https://huggingface.co/fe…

That doesn’t tell you if the new method continues to perform better at higher parameter counts.

Re: Starting from scratch: Training a 30M Topological Transformer

#29

Post by nl »

ashirviskas wrote: post #8

I wonder what if we just crammed more into the "tokens"? I am running an experiment of replacing discrete tokens with embeddings + small byte encoder/decoder. That way you can use embedding space much more efficiently and have it contain much more nuance. Experiments I want to build on top of it: 1. Adding lsp context to the embeddings - that way the model could _see_ the syntax better, closer to how we use IDEs and…

Isn't this just an awkward way of adding an extra layer to the NN, except without end-to-end training?

Models like Stable Diffusion sort of do a similar thing using Clip embeddings. It works, and it's an easy way to benefit from the pre-training Clip has. But for a language model it would seemingly make more sense to just add the extra layer.

Re: Starting from scratch: Training a 30M Topological Transformer

#30

Post by amelius »

throwaway314155 wrote: post #28
kouteiheika wrote: post #18

If you want to prove (i.e. show that it works and/or it's faster in a real-world scenario) a new alternative to attention without breaking the bank then one of the best ways to do that would probably be to retrain an already existing model, just with swapped attention modules. Then once you have such a model you can do apples-to-apples benchmarks. This has been done successfully in the past: https://huggingface.co/fe…

That doesn’t tell you if the new method continues to perform better at higher parameter counts.

Nor that the training from scratch will even work.
Post reply on HN