Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Starting from scratch: Training a 30M Topological Transformer

tuned.org.uk

Post reply on HN
Print view View original

11–20 of 59 posts

Re: Starting from scratch: Training a 30M Topological Transformer

#11

Post by xtiansimon »

geoffbp wrote: post #9

I dug into this a bit (with AI ofc) and it spat this out. I found it an easy way to visualise and start to understand: > Standard AI models (like GPT-4) treat data using Global Geometry. They imagine every word as a point floating in a massive, flat, high-dimensional room. To see how two words relate, they draw a straight line between them. > Local Topology changes the "room" into a landscape (a manifold). Instead of…

What is a "high-dimensional room"? A "room" is by definition three-dimensional in so far as we're using metaphor for description. Then to add this "high-dimensional" modifier does little for me, since the only visualizable high-dimensional cube is a tesseract, which still leaves you at 4-d.

The presented counterpoint to this metaphor has the "room" change into a "landscape". The room is a "flat void" compared to a landscape with "hills, valleys, and paths". None of these landscape features evoke higher dimensionality in my imagination. Certainly not in the way, say, the metaphor of the "coastline" of Great Britain does when discussing the unusual properties of a fractal.

These moves don't shift my railroad mind from one track onto another. So I wonder, if a metaphoric usage is not in some way universal, how can it be instructive?

Re: Starting from scratch: Training a 30M Topological Transformer

#12

Post by adroniser »

kannanvijayan wrote: post #6
keyle wrote: post #3

Does this make any sense, to anyone?

I think this is an attempt to try to enrich the locality model in transformers. One of the weird things you do in transformers is add a position vector which captures the distance between the token being attended to the some other token. This is obviously not powerful enough to express non-linear relationships - like graph relationships. This person seems to be experimenting with doing pre-processing of the input tok…

Adding the position vector is basic sure, but it's naive to think the model doesn't develop its own positional system bootstrapping on top of the barebones one.

Re: Starting from scratch: Training a 30M Topological Transformer

#14

Post by appplication »

ashirviskas wrote: post #8

I wonder what if we just crammed more into the "tokens"? I am running an experiment of replacing discrete tokens with embeddings + small byte encoder/decoder. That way you can use embedding space much more efficiently and have it contain much more nuance. Experiments I want to build on top of it: 1. Adding lsp context to the embeddings - that way the model could _see_ the syntax better, closer to how we use IDEs and…

Not an expert in the space, but I’m not sure you need to modify tokens to get the model to see syntax, you basically get that exact association from attention.

Re: Starting from scratch: Training a 30M Topological Transformer

#15

Post by postflopclarity »

pwndByDeath wrote: post #13
keyle wrote: post #3

Does this make any sense, to anyone?

No, its a new form of alchemy that turns electricity into hype. The technical jargon is more.of.a thieves cant to help identity other conmen to one another

that's a strange way to spell "no, I didn't understand the paper"

Re: Starting from scratch: Training a 30M Topological Transformer

#16

Post by Yemoshino »

ashirviskas wrote: post #8

I wonder what if we just crammed more into the "tokens"? I am running an experiment of replacing discrete tokens with embeddings + small byte encoder/decoder. That way you can use embedding space much more efficiently and have it contain much more nuance. Experiments I want to build on top of it: 1. Adding lsp context to the embeddings - that way the model could _see_ the syntax better, closer to how we use IDEs and…

I found a few papers in this direction with perplexity like this one https://ceur-ws.org/Vol-4005/paper1.pdf and it doesn't seem to be that relevant for now.

The progress of a handful models seem to be so much better (because limited compute, we have only a handful of big ones, i presume) that these finetunings are just not yet relevant.

I'm also curious if a english java + html + css + javascript only model would look like in size and speed for example.

Unfortunate whenever i ask myself the question of finetunging tokens (just a few days ago this question came up again), deep diving takes too much time.

Claude only got lsp support in november i think. And its not even clear to me to what extend. So despite the feeling we are moving fast, tons of basic ideas haven't even made it in yet

Re: Starting from scratch: Training a 30M Topological Transformer

#17

Post by Yemoshino »

pwndByDeath wrote: post #13
keyle wrote: post #3

Does this make any sense, to anyone?

No, its a new form of alchemy that turns electricity into hype. The technical jargon is more.of.a thieves cant to help identity other conmen to one another

Try get over your ai hate.

If you need help getting more out of ai, you can use chatgpt and co to go through papers and let yourself eli5 paragarphs. 1blue3brown also has a few great videos about transformer and how they work

Re: Starting from scratch: Training a 30M Topological Transformer

#18

Post by kouteiheika »

If you want to prove (i.e. show that it works and/or it's faster in a real-world scenario) a new alternative to attention without breaking the bank then one of the best ways to do that would probably be to retrain an already existing model, just with swapped attention modules. Then once you have such a model you can do apples-to-apples benchmarks.

This has been done successfully in the past:

https://huggingface.co/featherless-ai/QRWKV-72B

Note that this is a 72B model which would be very expensive to train from scratch, but here they did the conversion for less than $2000.

Re: Starting from scratch: Training a 30M Topological Transformer

#19

Post by Workaccount2 »

Yemoshino wrote: post #17
pwndByDeath wrote: post #13

Earlier quoted context omitted.

No, its a new form of alchemy that turns electricity into hype. The technical jargon is more.of.a thieves cant to help identity other conmen to one another

Try get over your ai hate. If you need help getting more out of ai, you can use chatgpt and co to go through papers and let yourself eli5 paragarphs. 1blue3brown also has a few great videos about transformer and how they work

Ideologues usually aren't great at primary source understanding/reasoning, hence why they end up with such strong opinions.

Re: Starting from scratch: Training a 30M Topological Transformer

#20

Post by oofbey »

kouteiheika wrote: post #18

If you want to prove (i.e. show that it works and/or it's faster in a real-world scenario) a new alternative to attention without breaking the bank then one of the best ways to do that would probably be to retrain an already existing model, just with swapped attention modules. Then once you have such a model you can do apples-to-apples benchmarks. This has been done successfully in the past: https://huggingface.co/fe…

Depending on how different the attention mechanism is, that might not work. If it’s just a faster / different way of finding the tokens to attend to, sure. But I get the sense the author is implying this method uses different semantics somehow. Although tbh I didn’t follow it entry.
Post reply on HN