51–60 of 197 posts
https://www.cambridge.org/core/services/aop-cambridge-core/c...
I've always hated build systems. Stuff cobbled together that barely works, yet a necessary step towards working software. This paper showed me there's hope. If we take build systems seriously, we can come up with something much better than most systems out there.
As a meta observation, its fascinating how few of these are computer related. Which is actually great because it gives me something to read on subjects im not familar with.
Abstraction has made Programming a hybrid of Tradition/Authority/Science/Art. It's nearly impossible to have a scientific paper on anything with abstraction. At best you can create some "after the fact" optimizations using time studies and statistics.
Meaningful Availability , Hauer et al.: https://www.usenix.org/system/files/nsdi20spring_hauer_prepu... A good incremental improvement in service level indicator measurements for large-scale cloud services. Obligatory The Morning Paper post: https://blog.acolyer.org/2020/02/26/meaningful-availability/
At one startup we worked at we had availability problems for some time, with the service going down in a semi-predictable manner ~2 times a day (and the proper bugfix a few weeks away). Because once a day the service went down was in the middle of the night with no one on call, pure availability was 80-90%. Given that it was a single country app with no one trying to do any business during the night, meaningful availability was ~99%. Knowing that gave us peace of mind and made tackling the problem a much more relaxed ordeal than the crunch time for a few weeks I've seen at other companies in similar situations.
https://arxiv.org/abs/1706.03762
It's from 2017 but I first read it this year. This is the paper that defined the "transformer" architecture for deep neural nets. Over the past few years, transformers have become a more and more common architecture, most notably with GPT-3 but also in other domains besides text generation. The fundamental principle behind the transformer is that it can detect patterns among an O(n) input size without requiring an O(n^2) size neural net.
If you are interested in GPT-3 and want to read something beyond the GPT-3 paper itself, I think this is the best paper to read to get an understanding of this transformer architecture.
Some CogSci & Neuro papers I found interesting in 2020: Constantinescu, Alexandra O., Jill X. O’Reilly, and Timothy EJ Behrens. "Organizing conceptual knowledge in humans with a gridlike code." Science 352.6292 (2016): 1464-1468. Kriegeskorte, Nikolaus, and Katherine R. Storrs. "Grid cells for conceptual spaces?." Neuron 92.2 (2016): 280-284. Klukas, Mirko, Marcus Lewis, and Ila Fiete. "Efficient and flexible represe…
Thanks for putting this list together! I took some cog sci courses in college and have been meaning to dive more into the research around it and these papers seem like a good place to start. I expect to run into lots of jargon and concepts I don't understand. Would it be possible for me to reach out to you for questions when I'm unable to make sense of the content after having researched the unknown concepts online?
Attention Is All You Need https://arxiv.org/abs/1706.03762 It's from 2017 but I first read it this year. This is the paper that defined the "transformer" architecture for deep neural nets. Over the past few years, transformers have become a more and more common architecture, most notably with GPT-3 but also in other domains besides text generation. The fundamental principle behind the transformer is that it can detec…
Recurrent neural network size is also independent of input sequence length.
The successful removal of inductive bias is really what differentiates this from previous sequence-to-sequence neural networks.
Some CogSci & Neuro papers I found interesting in 2020: Constantinescu, Alexandra O., Jill X. O’Reilly, and Timothy EJ Behrens. "Organizing conceptual knowledge in humans with a gridlike code." Science 352.6292 (2016): 1464-1468. Kriegeskorte, Nikolaus, and Katherine R. Storrs. "Grid cells for conceptual spaces?." Neuron 92.2 (2016): 280-284. Klukas, Mirko, Marcus Lewis, and Ila Fiete. "Efficient and flexible represe…
Here's a cool paper on bird song during lockdown.
Derryberry, Elizabeth P., et al. Singing in a silent spring: Birds respond to a half-century soundscape reversion during the COVID-19 shutdown (2020) (DOI: 10.1126/science.abd5777)
Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures [2] applies DFA, an approach to training neural nets without backprop, to modern architectures like the Transformer. It does surprisingly well and is a step in the right direction for biologically plausible neural nets as well as potentially significant efficiency gains.
Hopfield Networks is All You Need [3] analyzes the Transformer architecture as the classical Hopfield Network. This one got a lot of buzz on HN so I won't talk about it too much, but it's part of a slew of other analyses of the Transformer that basically show how generalizable the attention mechanism is. It also sorta confirms many researchers' inkling that Transformers are likely just memorizing patterns in their training corpus.
Edit: Adding a few interesting older NLP papers that I came across this year.
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding [4]
Do Syntax Trees Help Pre-trained Transformers Extract Information? [5]
Learning to Compose Neural Networks for Question Answering [6]
Parsing with Compositional Vector Grammars [7]
[1] https://arxiv.org/abs/2006.11287
[2] https://arxiv.org/abs/2006.12878
[3] https://arxiv.org/abs/2008.02217
[4] https://arxiv.org/abs/1908.04577
[5] https://arxiv.org/abs/2008.09084
Attention Is All You Need https://arxiv.org/abs/1706.03762 It's from 2017 but I first read it this year. This is the paper that defined the "transformer" architecture for deep neural nets. Over the past few years, transformers have become a more and more common architecture, most notably with GPT-3 but also in other domains besides text generation. The fundamental principle behind the transformer is that it can detec…
This might be misleading, the amount of computation for processing a sequence size N with a vanilla transformer is still N^2. There has been recent work however which has tried to make them scale better.