Earlier quoted context omitted.
Somehow I expected he made it out of mechanical parts. Like literally “by hand”.
Nitpick: the author is female
[flagged]
I made a transformer to predict a simple sequence manually
51–60 of 104 posts
Re: I made a transformer to predict a simple sequence manually
#52Earlier quoted context omitted.
> It would be nice to have compilers for them that were not based on approximation, though. Could you elaborate?
People typically set the weights of a neural network using heuristic approximation algorithms, by looking at a large set of example inputs/outputs and trying to find weights that perform the needed computation as accurately as possible. This approximation process is called training . But this approximation happens because nobody really knows how to set the weights otherwise. It would be nice if we had "compilers" for…
The point of training is to create computer programs through optimization, because there are many problems (like understanding language) that we just don't know how to write programs to do.
It's not that we don't know how to set the weights - neural networks are only designed with weights because it makes them easy to optimize.
There is no reason to use them if you plan to write your own code for them. You won't be able to do anything that you couldn't do in a normal programming language, because what makes NNs special is the training process.
Re: I made a transformer to predict a simple sequence manually
#53[stub for offtopicness]
[deleted]
Re: I made a transformer to predict a simple sequence manually
#54[stub for offtopicness]
i thought, here is a guy building an electrical transformer by hand, but no, more AI shite
Re: I made a transformer to predict a simple sequence manually
#55Re: I made a transformer to predict a simple sequence manually
#56[stub for offtopicness]
Minor request: can we have 'neural network' or something in the title? This is related to the machine learning 'transformer' architecture, rather than the bundle of coils that couples two circuits electromagnetically.
It's always mildly annoying when different technologies have the same name or acronym
Re: I made a transformer to predict a simple sequence manually
#57Re: I made a transformer to predict a simple sequence manually
#58A related line of work is "Thinking Like Transformers" [1]. They introduce a primitive programming language, RASP, which is composed of operations capable of being modeled with transformer components, and demonstrate how different programs can be written with it, e.g. histograms, sorting. Sasha Rush and Gail Weiss have an excellent blog post on it as well [2]. Follow on work actually demonstrated how RASP-like progra…
Huge fan of RASP et al. If you enjoy this space, might be fun to take a glance at some of my work on HandCrafted Transformers [1] wherein I hand-pick the weights in a transformer model to do long-handed addition similar to how humans learn to do it in gradeshcool.
[1] https://colab.research.google.com/github/newhouseb/handcraft...
Re: I made a transformer to predict a simple sequence manually
#59[stub for offtopicness]
Re: I made a transformer to predict a simple sequence manually
#60> maybe even feel inspired to make your own model by hand as well! Other then a learning exercise to satisfy your curiosity what are you doing with this? I'm starting to get the feeling that anything complex with ml models is unreasonable for a at home blog reader?
Author states in the first paragraph of their blog post: "I've been wanting to understand transformers and attention better for awhile now—I'd read The Illustrated Transformer, but still didn't feel like I had an intuitive understanding of what the various pieces of attention were doing. What's the difference between q and k? And don't even get me started on v!"
lol three people said the same thing, i get that. thats why i said other than learning and satisfying curiosity...