Jeremy from fast.ai here. Happy to answer any questions about the course, fastai, or anything else relevant! BTW the 2018 version of the course is being discussed in this forum, for those interested: http://forums.fast.ai/c/part1-v2
Two questions: 1. You've made an off-hand comment on one of your videos that a sequential dense network is just a generalization of any other type of neural network architecture. In theory you could re-create an RNN or CNN through just Dense layers. But obviously it's not practical. Why isn't it practical? Is it because the network would have to be too deep, or too wide? Would the optimizer just get stuck in a local…
Schmidhuber did a paper a few years ago showing near SoTA performance on computer vision using just a fully connected net. One of our students showed how a convolution is just a weight-tied matrix multiply here: https://medium.com/impactai/cnns-from-different-viewpoints-f...
So the issue is that without the weight-tying, you've got more parameters to regularize (which can decrease performance) and train (which takes longer). So you should use weight tying where you can - e.g. by using convolutions.
In general, domain-specific architectures try to find structure in the underlying data and problem, and use that to decrease the number of parameters we need. The use of implicit factorizations in the inception and xception architectures is a good example.