Single Headed Attention RNN
arxiv.org
Single Headed Attention RNN
1–10 of 40 posts
Re: Single Headed Attention RNN
#2You just have to love Stephen Merity.
His work on QRNN's saved me quite a bit of time and money when I was doing my undergrad dissertation on language models.
This SHA-RNN seems to have surfaced from a similar line of thinking that spawned the QRNN.
Re: Single Headed Attention RNN
#3[deleted]
Re: Single Headed Attention RNN
#4The paper rises a great point on tokenization affecting perplexity, that we can't compare perplexities of different tokenizers even re-normalizing taking token counts into account, say BPE vs word tokenization. This example nails it: https://twitter.com/Smerity/status/1192252147598909441
Re: Single Headed Attention RNN
#5Honestly, I wish all research papers were written this way. Easy to understand, kept me entertained, and presented meaningful results with a way to reproduce (on a single GPU).
I grant all research papers on deep learning can't be reproducible with a single GPU in a reasonable time, but it should happen more often IMO. It seems lazy to just toss out a paper saying "we hit new benchmarks, by increasing the parameters and throwing more compute". I'd like to see "we hit new benchmarks with a new design, the old ones had this issue, etc.
Anyway, great read, recommend. Also, happy for the author haha
"The author has also moved to a one bedroom apartment in San Francisco, removing themselves from proximity to the alley of questionable odors and unsavory noises."
Re: Single Headed Attention RNN
#6A dissenting voice from the positive reception here on HN, I thought that this paper was a joke. Single author, no affiliation, snarky language. Why not be civil instead?
Re: Single Headed Attention RNN
#7for those who aren't familiar with the author, he previously worked at metamind / salesforce research doing nlp and has published many successful nlp papers [0]. he opted to write an informal paper for this project (similar to yolov3 [1]), but the work itself should still be taken seriously.
Re: Single Headed Attention RNN
#8A dissenting voice from the positive reception here on HN, I thought that this paper was a joke. Single author, no affiliation, snarky language. Why not be civil instead?
Language can be debated, I agree with you, but
What’s wrong with single authorship and no affiliation?
At the end of the day, if the paper proposes some idea or method, and achieves the stated claims (with reproducible code), then I don’t care who wrote it, how many authors there were, and who the authors work for.
Re: Single Headed Attention RNN
#9A dissenting voice from the positive reception here on HN, I thought that this paper was a joke. Single author, no affiliation, snarky language. Why not be civil instead?
> Single author, no affiliation, snarky language.
I'd say that all of these are factors that don't add or detract from the value of the paper itself - it's a "hey I tried this and it works ok despite not going in the obvious direction". So, limited experiments but IMO competently done and with usable information.
It's a pity that all papers nowadays have a gazillion authors, from well-funded research labs, with as-dry-as-possible language that hides the real research behind a "we knew this all along rather than figuring it out along the way" facade. OTOH that's what you get in a large fairly mature research field, where most competent people get hired by research labs and then do lots of collaborative research that scales well and subsequently need to show publication counts to secure further funding.
Re: Single Headed Attention RNN
#10Hilarious papers, i'm about to drop a SHA-RNN on my GPu to make it sweat