A dissenting voice from the positive reception here on HN, I thought that this paper was a joke. Single author, no affiliation, snarky language. Why not be civil instead?
Because it is funny
Single Headed Attention RNN
11–20 of 40 posts
Re: Single Headed Attention RNN
#12Honestly, I wish all research papers were written this way. Easy to understand, kept me entertained, and presented meaningful results with a way to reproduce (on a single GPU). I grant all research papers on deep learning can't be reproducible with a single GPU in a reasonable time, but it should happen more often IMO. It seems lazy to just toss out a paper saying "we hit new benchmarks, by increasing the parameters…
Now imagine reading papers is your job and you try to skim through dozens of wannabe stand up comedians each day.
Re: Single Headed Attention RNN
#13A dissenting voice from the positive reception here on HN, I thought that this paper was a joke. Single author, no affiliation, snarky language. Why not be civil instead?
If you don't know he's a relatively successful (if you count citations) author (seems previously) from Salesforce research.
He has worked on YOLO (computer vision) and NLP related problems.
Re: Single Headed Attention RNN
#14The writing style is amusing. :)
Some notes from a first glance:
* In the experiments, I see that he uses the Single Headed Attention model actually also with 4 heads, which is kind of a contradiction to the name, isn't it?
* The main motivation is performance (training speed mostly). So some absolute number of e.g. training time would be nice to have in the comparisons. He e.g. mentions that the Adaptive Transformer can also be trained on a single GPU within hours, and in the comparison, the Adaptive Transformer gets much better BPC (enwik8), and uses even slightly less parameters. So, isn't the Adaptive Transformer thus better in every aspect (speed and BPC)? Or how does it compare in speed? As far as I remember, also the Sparse Transformer is more efficient (as it has sparsity), so again the speed comparison would be interesting here. Or is the argumentation for inference speed? But then the inference speed should be compared, or not?
Re: Single Headed Attention RNN
#15You just have to love Stephen Merity. His work on QRNN's saved me quite a bit of time and money when I was doing my undergrad dissertation on language models. This SHA-RNN seems to have surfaced from a similar line of thinking that spawned the QRNN.
Are qRNNs still used much?
Re: Single Headed Attention RNN
#16Now I really want pop music made by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin
Re: Single Headed Attention RNN
#17ok
Re: Single Headed Attention RNN
#18The writing style is amusing. :) Some notes from a first glance: * In the experiments, I see that he uses the Single Headed Attention model actually also with 4 heads, which is kind of a contradiction to the name, isn't it? * The main motivation is performance (training speed mostly). So some absolute number of e.g. training time would be nice to have in the comparisons. He e.g. mentions that the Adaptive Transformer…
I don't think that was his motivation, I think his motivation was stated quite clearly in the abstract:
> The author's lone goal is to show that the entire field might have evolved a different direction if we had instead been obsessed with a slightly different acronym and slightly different result.
Re: Single Headed Attention RNN
#19A dissenting voice from the positive reception here on HN, I thought that this paper was a joke. Single author, no affiliation, snarky language. Why not be civil instead?
What makes this uncivil to you?
Re: Single Headed Attention RNN
#20A dissenting voice from the positive reception here on HN, I thought that this paper was a joke. Single author, no affiliation, snarky language. Why not be civil instead?
While informal, I do not think his tone lacked civility.
I strongly prefer papers written in this style. Not only are they more enjoyable to read, but they are often easier to understand and more geniune as well. Papers written in a formal style often obscure the real motivation and instead provide a fancy-sounding retroactive justification. It makes the authors feel smarter, and I guess some readers feel smarter as well, but it belies the reality of research.