Live data from Hacker News

Single Headed Attention RNN

arxiv.org

31–40 of 40 posts

Re: Single Headed Attention RNN

#31

Earlier quoted context omitted.

Now imagine reading papers is your job and you try to skim through dozens of wannabe stand up comedians each day.

Is dozens of papers per day how academics work? Holy crap.

Depends. If the paper is dense you're taking hours per paper, rereading many times and rederiving details. You can skim through many you don't want to dive deeply though (if it's a topic you just need a feel for).

Re: Single Headed Attention RNN

#32

Earlier quoted context omitted.

Now imagine reading papers is your job and you try to skim through dozens of wannabe stand up comedians each day.

Is dozens of papers per day how academics work? Holy crap.

Notice the word skim. Yes when I'm trying to figure something out I often have 10 tabs with papers open and I'm flipping between them.

Re: Single Headed Attention RNN

#34
post #5

Honestly, I wish all research papers were written this way. Easy to understand, kept me entertained, and presented meaningful results with a way to reproduce (on a single GPU). I grant all research papers on deep learning can't be reproducible with a single GPU in a reasonable time, but it should happen more often IMO. It seems lazy to just toss out a paper saying "we hit new benchmarks, by increasing the parameters…

The author is very intelligent and is doing three things differently from a 'standard' paper:

1) Reducing the density of information per paragraph (vs. packing information in)

2) Clearly outlined motivation and context (vs. just referencing some other papers and assuming they've been read)

3) Deploying comedy (vs. professionalism)

The first two improve the paper, the third is a step backwards because the reader has to spend effort separating fact & fiction. The combination in this case works and is a lot of fun but it would have been a catastrophic and cringeworthy exercise if the execution of (1) and (2) hadn't worked out so well.

The real trick here is excellent writing and the comedy is simply draws attention to it. Much like how an army marching in a silly way draws attention to its discipline. The silly march itself is not a good idea.

Re: Single Headed Attention RNN

#35
post #6

A dissenting voice from the positive reception here on HN, I thought that this paper was a joke. Single author, no affiliation, snarky language. Why not be civil instead?

I'll second this. The style is clunky and reads as though the author were trying too hard to make every sentence entertaining which mostly detracts from the work. I know Stephen Merity is a serious researcher and the content here is legit given his body of work. But the style /prose in this preprint reminded me a lot of the some of garbage Siraj Raval peddled. Again, to reiterate, I am not commenting on the substance, only the style.

Re: Single Headed Attention RNN

#37

You just have to love Stephen Merity. His work on QRNN's saved me quite a bit of time and money when I was doing my undergrad dissertation on language models. This SHA-RNN seems to have surfaced from a similar line of thinking that spawned the QRNN.

Are qRNNs still used much?

check out MultiFiT [0] from fastai, it uses QRNN for speed.

[0] http://nlp.fast.ai/classification/2019/09/10/multifit.html

Re: Single Headed Attention RNN

#38
post #34
post #5

Honestly, I wish all research papers were written this way. Easy to understand, kept me entertained, and presented meaningful results with a way to reproduce (on a single GPU). I grant all research papers on deep learning can't be reproducible with a single GPU in a reasonable time, but it should happen more often IMO. It seems lazy to just toss out a paper saying "we hit new benchmarks, by increasing the parameters…

The author is very intelligent and is doing three things differently from a 'standard' paper: 1) Reducing the density of information per paragraph (vs. packing information in) 2) Clearly outlined motivation and context (vs. just referencing some other papers and assuming they've been read) 3) Deploying comedy (vs. professionalism) The first two improve the paper, the third is a step backwards because the reader has t…

I'm not sure reducing the density of information is a good thing. It probably makes it easier to read for someone who is not entirely familiar with a field but makes it slower for the people for whom the paper is written. (Too dense papers are really hard to read, but that's relatively rare)

Re: Single Headed Attention RNN

#39
post #9

Earlier quoted context omitted.

> Single author, no affiliation, snarky language. I'd say that all of these are factors that don't add or detract from the value of the paper itself - it's a "hey I tried this and it works ok despite not going in the obvious direction". So, limited experiments but IMO competently done and with usable information. It's a pity that all papers nowadays have a gazillion authors, from well-funded research labs, with as-dr…

Its a shame that professionalism and showing personality are so at odds all over the place, from papers to the workplace. For the most part, professional has aligned with formal. It's clear why, but still sad :(

Why is it sad? The whole point in professionalism is disaffective communication.

Re: Single Headed Attention RNN

#40
post #5

Honestly, I wish all research papers were written this way. Easy to understand, kept me entertained, and presented meaningful results with a way to reproduce (on a single GPU). I grant all research papers on deep learning can't be reproducible with a single GPU in a reasonable time, but it should happen more often IMO. It seems lazy to just toss out a paper saying "we hit new benchmarks, by increasing the parameters…

Now imagine reading papers is your job and you try to skim through dozens of wannabe stand up comedians each day.

As opposed to wannabe tax lawyers? I'll take the comedians thank you very much.
Post reply on HN