Smallest transformer that can add two 10-digit numbers
1–10 of 105 posts
Re: Smallest transformer that can add two 10-digit numbers
#2> In short: if you can swap in a different set of weights and use the exact same inference code for a different task, your setup is legitimate. If the inference code is inseparable from the algorithm, it's not.
I wonder why they don't just write the code themselves, so by design the focus can be on the model.
Re: Smallest transformer that can add two 10-digit numbers
#3You can do that in a single matmul of course.
Re: Smallest transformer that can add two 10-digit numbers
#4You can do that in a single matmul of course.
So can you take an arbitrary transformer and somehow turn it into a compact set of low-power fast gates by some algorithm?
Re: Smallest transformer that can add two 10-digit numbers
#5Re: Smallest transformer that can add two 10-digit numbers
#6Not sure how much this fits into the rules but I saw on twitter someone claimed 28 params : https://gist.github.com/SeuperHakkerJa/da3050739bea97aabd86e...
Re: Smallest transformer that can add two 10-digit numbers
#7So, hand-coded weights can do it with 36 params and 311 for trained weights - did anyone try the former architecture, but starting with random weights and learning?
Re: Smallest transformer that can add two 10-digit numbers
#8You can do that in a single matmul of course.
[deleted]