71–75 of 75 posts
This is new to me, and therefore bad and scary. It's great that you know NN well enough to fold it into regular work. But think of all us poor regular developers! Who now have to grapple with: - an unfamiliar architecture - uncertainty / effectively non-deterministic results in program flow
NN are in principle deterministic (unless you add randomness to it such as is the case with LLM top p/k temperature). Uncertainty is probably the better word of the two, but I feel like there should be a different term.
Earlier quoted context omitted.
The Taylor Series dates to 1715. Fourier Series dates to the 1820s. Both are universal function approximators and both can be learned via gradient descent. For the case where the function you want to learn actually is polynomial or periodic (respectively), these are better than neural networks.
For your interest, Taylor Series are not universal function approximators - the Taylor Series around 0 for f(x) = e^(-1/x^2) if x != 0 else 0 is identically zero (all partial derivatives are 0 at 0) but the function is clearly not identically zero. So the radius of convergence for this Taylor series is infinite but it only equals the approximated function at one point. I'm sure there are some conditions you can put o…
Generally, they assume the function to be approximated is continuous.
Earlier quoted context omitted.
Is that true also of LSTMs?
Yes. We cover Jordan and Elman RNN, LSTMs, and GRUs. Assignments only really test for LSTM knowledge, though.
I'm surprised about only testing for LSTMs- of all the sequence/memory models, they seem like the most arbitrary and hacky, but I've never been able to determine if that's simply because I don't understand those types of models (my training is in HMMs- do you teach/test those?)
Earlier quoted context omitted.
Yes. We cover Jordan and Elman RNN, LSTMs, and GRUs. Assignments only really test for LSTM knowledge, though.
Thanks. The reason I asked the question is that I've struggled to understand RNNs and other networks (compared to MLPs, CNNs, and transformers) due to the subtlety of their design and my hope was that I could simply forget about them. I'm surprised about only testing for LSTMs- of all the sequence/memory models, they seem like the most arbitrary and hacky, but I've never been able to determine if that's simply becaus…
A lot of my research has focused on LSTMs, and so I am partial to them. I think they are super useful and have a lot of properties, but frankly speaking if you had to choose one architectures of the ones you mentioned, LSTMs/RNNs are probably the most OK to skip.
That said, if you just look at a simple RNN like the Jordan RNNs and focus on understanding that, then LSTMs just become fancy RNNs with some forgetting and remembering logic.