Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made a transformer to predict a simple sequence manually

vgel.me

Post reply on HN
Print view View original

61–70 of 104 posts

Re: I made a transformer to predict a simple sequence manually

#61

Post by shawntan »

enriquto wrote: post #14
PaulHoule wrote: post #12

It's some kind of abstract machine, like a turing machine or the machine that parses regexes, isn't it?

Neural networks are Turing machines. You can make them perform any computation by carefully setting up their weights. It would be nice to have compilers for them that were not based on approximation, though.

Would I be able to solve the Travelling Salesman Problem with a Transformer with the appropriately assigned weights? That would be an achievement. You'd beat some known bounds of the complexity of TSP.

Re: I made a transformer to predict a simple sequence manually

#62

Post by tostadora1 »

I always wanted to at least have a shallow understanding of Transformers but the paper was way too technical for me.

This really helped me understand how they work! Or at least I understood your example, it was very clear. And I also got to brush up my matrix stuff from uni lol.

Thanks!

Re: I made a transformer to predict a simple sequence manually

#63

Post by ntonozzi »

PaulHoule wrote: post #12

It's some kind of abstract machine, like a turing machine or the machine that parses regexes, isn't it?

Yes! Check out this paper describing how Linear Transformers are secretly Fast Weight Programmers: https://arxiv.org/abs/2102.11174.

Re: I made a transformer to predict a simple sequence manually

#65

Post by shawntan »

nyrikki wrote: post #27
enriquto wrote: post #18

Earlier quoted context omitted.

People typically set the weights of a neural network using heuristic approximation algorithms, by looking at a large set of example inputs/outputs and trying to find weights that perform the needed computation as accurately as possible. This approximation process is called training . But this approximation happens because nobody really knows how to set the weights otherwise. It would be nice if we had "compilers" for…

A turing complete system doesn't necessarily mean it's useful, it just means that it's equivalent with a turing machine. The ability to describe any possible algorithm is not that powerful in itself. As an example, algebraic type systems are often TC simply because general recursion is allowed. Feed forward networks are effectively DAGs and while you may be able to express any algorithms using them they are also pair…

Short version of this without the caveats: It's not even Turing complete.

I review a few papers on the topic here: https://blog.wtf.sg/posts/2023-02-03-the-new-xor-problem/

Re: I made a transformer to predict a simple sequence manually

#67

Post by shawntan »

enriquto wrote: post #18
PartiallyTyped wrote: post #15

Earlier quoted context omitted.

> It would be nice to have compilers for them that were not based on approximation, though. Could you elaborate?

People typically set the weights of a neural network using heuristic approximation algorithms, by looking at a large set of example inputs/outputs and trying to find weights that perform the needed computation as accurately as possible. This approximation process is called training . But this approximation happens because nobody really knows how to set the weights otherwise. It would be nice if we had "compilers" for…

Something close exists:

RASP https://arxiv.org/abs/2106.06981

python implementation: https://srush.github.io/raspy/

Re: I made a transformer to predict a simple sequence manually

#68

Post by mistrial9 »

Supply5411 wrote: post #44
elesiuta wrote: post #42

Earlier quoted context omitted.

> weights of a model, that a domain expert can tweak by hand This sounds similar to how image recognition was done before deep learning [1] [1] https://www.youtube.com/watch?v=8SF_h3xF3cE&t=1358s

Great example. Right, the deep learning approach uncovers all kinds of hidden features and relationships automatically that a team of humans might miss. I guess I'm thinking about this problem from the perspective of these GPT models requiring more training data than a normal person can acquire. Currently, it seems you need the entire internet worth of training data (and a lot of money) to get something that can comm…

> the deep learning approach uncovers all kinds of hidden features and relationships automatically that a team of humans might miss

sitting in a lecture from a decent DeepLearning practitioner, there were two questions from the audience (among others). The first question asked "How can we check the results using other models, so that computers will catch the errors that humans miss?"

The second question was more like "when a model is built across a non-trivial input space, the features and classes that come out are one set of possibilities, but there are many more possibilities. How can we discover more about the model that is built, knowing that there are inherent epistemological conflicts in any model?"

I also thought it was interesting that the two questioners were from large but very different demographic groups, and at different stages of learning and practice (the second question was from a senior coder).

Re: I made a transformer to predict a simple sequence manually

#69

Post by astrange »

Supply5411 wrote: post #41
amilios wrote: post #38

Earlier quoted context omitted.

The technical reason it can't be done (or would be very difficult to do) is that weights are typically very uninterpretable. There aren't specific clusters of neurons that map to one concept or another, everything kind of does everything.

I wonder if an expert can "impose" weights onto a model and the model will opt to continue with them when it resumes training. For example, in the vision example, the expert may not know where "orangeness" currently exists, but if they impose their own collection of weight adjustments that represent orangeness, will the model continue to use these weights as the path of least resistance when continuing to optimize? J…

That would make it less efficient - since learning is compression, a less compressed model will also learn less at the same size.

Re: I made a transformer to predict a simple sequence manually

#70

Post by sterlind »

tayo42 wrote: post #16

> maybe even feel inspired to make your own model by hand as well! Other then a learning exercise to satisfy your curiosity what are you doing with this? I'm starting to get the feeling that anything complex with ml models is unreasonable for a at home blog reader?

In nanoGPT, you pre-train a model on Shakespeare, and in 3 minutes it gets to a Lewis Carroll's Jabberwocky level of fidelity on the source material. It makes up lots of plausible-seeming old English words, learns the basics of English grammar, the layout of the plays, etc. I was pretty amazed that it got that good in such a short period of time.

I think locally training a bunch of models to the fidelity of Shakespeare-from-Wish.com might tell you when you've hit on a winning architecture, and when to try scaling up.

Post reply on HN