Earlier quoted context omitted.
Nitpick: the author is female
[flagged]
This isn't a color comment. It's a small correction to a factual error. No culture war to be had here.
I made a transformer to predict a simple sequence manually
21–30 of 104 posts
Re: I made a transformer to predict a simple sequence manually
#22Earlier quoted context omitted.
> It would be nice to have compilers for them that were not based on approximation, though. Could you elaborate?
People typically set the weights of a neural network using heuristic approximation algorithms, by looking at a large set of example inputs/outputs and trying to find weights that perform the needed computation as accurately as possible. This approximation process is called training . But this approximation happens because nobody really knows how to set the weights otherwise. It would be nice if we had "compilers" for…
Why would you do that when it's better to do the opposite? Given a model quantize it and compile it to direct code objects that do the same thing much much much faster?
The generality of the approach [NNs] implies that they are effectively a union of all programs that may be represented, and as such there needs to be the capacity for that, this capacity is in size, which makes them wasteful for exact solutions.
it is fairly trivial to create FFNNs that behave as decision trees using just relus if you can encode your problem as a continuous problem with a finite set of inputs. Then you can very well say that this decision tree is, well, a program, and there you have it.
The actual problem is the encoding, which is why NNs are so powerful, that is, they learn the encodings themselves through grad descent and variants.
Re: I made a transformer to predict a simple sequence manually
#23Re: I made a transformer to predict a simple sequence manually
#24Re: I made a transformer to predict a simple sequence manually
#25Re: I made a transformer to predict a simple sequence manually
#26Earlier quoted context omitted.
"no training" gave it away for me.
I misinterpreted that as "I have no training; I am a beginner" rather than "I am not going to train this thing" :)
That's how I interpreted as well.
"Electrical transformer without any formal training in electric engineering" is basically how the title read to me. Followed by a lot of confusion when the article was not that.
Good on him, it's just... an odd way to phrase it :)
Re: I made a transformer to predict a simple sequence manually
#27Earlier quoted context omitted.
> It would be nice to have compilers for them that were not based on approximation, though. Could you elaborate?
People typically set the weights of a neural network using heuristic approximation algorithms, by looking at a large set of example inputs/outputs and trying to find weights that perform the needed computation as accurately as possible. This approximation process is called training . But this approximation happens because nobody really knows how to set the weights otherwise. It would be nice if we had "compilers" for…
A turing complete system doesn't necessarily mean it's useful, it just means that it's equivalent with a turing machine. The ability to describe any possible algorithm is not that powerful in itself.
As an example, algebraic type systems are often TC simply because general recursion is allowed.
Feed forward networks are effectively DAGs and while you may be able to express any algorithms using them they are also pairwise linear in respect to inputs.
Statistical learning is powerful in finding and matching patterns, but graph rewriting, which is what your doing with initial random weights and training is not trivial.
More importantly it doesn't make issues like the halting problem decidable.
I don't see why the same limits in graph rewriting languages which were explored in the 90s won't hit using feed forward networks as computation systems outside of the application of nation-state scale computing power.
But I am open to understanding where I am wrong.
Re: I made a transformer to predict a simple sequence manually
#28A related line of work is "Thinking Like Transformers" [1]. They introduce a primitive programming language, RASP, which is composed of operations capable of being modeled with transformer components, and demonstrate how different programs can be written with it, e.g. histograms, sorting. Sasha Rush and Gail Weiss have an excellent blog post on it as well [2]. Follow on work actually demonstrated how RASP-like programs could actually be compiled into model weights without training [3].
[1] https://arxiv.org/abs/2106.06981
Re: I made a transformer to predict a simple sequence manually
#29Earlier quoted context omitted.
> It would be nice to have compilers for them that were not based on approximation, though. Could you elaborate?
People typically set the weights of a neural network using heuristic approximation algorithms, by looking at a large set of example inputs/outputs and trying to find weights that perform the needed computation as accurately as possible. This approximation process is called training . But this approximation happens because nobody really knows how to set the weights otherwise. It would be nice if we had "compilers" for…
That makes no entropy or cyber-netic sense at all. You would just get a neural network that outputs the exact formula, or algo. Like, if you would just do a sine it would be a taylor series encoded into neurons
Its like going from computing PI as a constant to computing it as a giantic float.
You lose info
Re: I made a transformer to predict a simple sequence manually
#30Earlier quoted context omitted.
And not about "alien robots who can disguise themselves by transforming into everyday machinery, primarily vehicles" [1] either. :) [1] https://en.wikipedia.org/wiki/Transformers_(film)
Ha, I didn't think of that. But if it was, that might be more impressive than either coiling wire or writing Python code :)
You might appreciate this then https://m.youtube.com/watch?v=uFmV0Xxae18