Earlier quoted context omitted.
Now imagine reading papers is your job and you try to skim through dozens of wannabe stand up comedians each day.
Is dozens of papers per day how academics work? Holy crap.
Single Headed Attention RNN
31–40 of 40 posts
Re: Single Headed Attention RNN
#32Earlier quoted context omitted.
Now imagine reading papers is your job and you try to skim through dozens of wannabe stand up comedians each day.
Is dozens of papers per day how academics work? Holy crap.
Re: Single Headed Attention RNN
#33Re: Single Headed Attention RNN
#34Honestly, I wish all research papers were written this way. Easy to understand, kept me entertained, and presented meaningful results with a way to reproduce (on a single GPU). I grant all research papers on deep learning can't be reproducible with a single GPU in a reasonable time, but it should happen more often IMO. It seems lazy to just toss out a paper saying "we hit new benchmarks, by increasing the parameters…
1) Reducing the density of information per paragraph (vs. packing information in)
2) Clearly outlined motivation and context (vs. just referencing some other papers and assuming they've been read)
3) Deploying comedy (vs. professionalism)
The first two improve the paper, the third is a step backwards because the reader has to spend effort separating fact & fiction. The combination in this case works and is a lot of fun but it would have been a catastrophic and cringeworthy exercise if the execution of (1) and (2) hadn't worked out so well.
The real trick here is excellent writing and the comedy is simply draws attention to it. Much like how an army marching in a silly way draws attention to its discipline. The silly march itself is not a good idea.
Re: Single Headed Attention RNN
#35A dissenting voice from the positive reception here on HN, I thought that this paper was a joke. Single author, no affiliation, snarky language. Why not be civil instead?
Re: Single Headed Attention RNN
#36Re: Single Headed Attention RNN
#37You just have to love Stephen Merity. His work on QRNN's saved me quite a bit of time and money when I was doing my undergrad dissertation on language models. This SHA-RNN seems to have surfaced from a similar line of thinking that spawned the QRNN.
Are qRNNs still used much?
[0] http://nlp.fast.ai/classification/2019/09/10/multifit.html
Re: Single Headed Attention RNN
#38Honestly, I wish all research papers were written this way. Easy to understand, kept me entertained, and presented meaningful results with a way to reproduce (on a single GPU). I grant all research papers on deep learning can't be reproducible with a single GPU in a reasonable time, but it should happen more often IMO. It seems lazy to just toss out a paper saying "we hit new benchmarks, by increasing the parameters…
The author is very intelligent and is doing three things differently from a 'standard' paper: 1) Reducing the density of information per paragraph (vs. packing information in) 2) Clearly outlined motivation and context (vs. just referencing some other papers and assuming they've been read) 3) Deploying comedy (vs. professionalism) The first two improve the paper, the third is a step backwards because the reader has t…
Re: Single Headed Attention RNN
#39Earlier quoted context omitted.
> Single author, no affiliation, snarky language. I'd say that all of these are factors that don't add or detract from the value of the paper itself - it's a "hey I tried this and it works ok despite not going in the obvious direction". So, limited experiments but IMO competently done and with usable information. It's a pity that all papers nowadays have a gazillion authors, from well-funded research labs, with as-dr…
Its a shame that professionalism and showing personality are so at odds all over the place, from papers to the workplace. For the most part, professional has aligned with formal. It's clear why, but still sad :(
Re: Single Headed Attention RNN
#40Honestly, I wish all research papers were written this way. Easy to understand, kept me entertained, and presented meaningful results with a way to reproduce (on a single GPU). I grant all research papers on deep learning can't be reproducible with a single GPU in a reasonable time, but it should happen more often IMO. It seems lazy to just toss out a paper saying "we hit new benchmarks, by increasing the parameters…
Now imagine reading papers is your job and you try to skim through dozens of wannabe stand up comedians each day.