MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video] • Hacker News Board

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#61

Post by tysam_and » Mon, Apr 03, 2023, 1:50 AM UTC

ramraj07 wrote: post #56

tysam_and wrote: post #42

Earlier quoted context omitted.

Karpathy's zero to hero series is excellent, and I really recommend it. I also made a few repos that are geared around readability and being a good 'working code demonstration' of certain best-practices in neural networks. If you're like me and you grok code better than symbols, this could be a helpful adjunct as well if you're wanting to dig deep a bit. https://github.com/tysam-code/hlb-CIFAR10 (convolutional networ…

Thanks, will check these out for sure. Already digging Karpathys code first approach in the first few hours!

Yeah, he is great. Hard to really estimate the impact he's had on the deep learning community on the whole/at large.

We could always use more people sharing and spreading knowledge around like him! I hope to find a similar brand to that someday for myself, though I'm constantly growing into those boots, I think. :') If you have any questions on that too feel free to let me know here and I can try to answer them. It's certainly a very interesting field! <3 :DDDD :)))) <3

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#62

Post by matt_langston » Mon, Apr 03, 2023, 3:05 AM UTC

lupire wrote: post #60

matt_langston wrote: post #50

Earlier quoted context omitted.

As defined by the lecturer herself, "h(t) = h(t-1)", the very definition of hysteresis. My point is that the lecturer missed a golden opportunity to give her students a natural intuition of "h" that they can see, feel and touch and that will serve them well for their entire careers. The only thing "hidden" about "h" is that hysteresis is hidden in plain site in her lecture - maybe the lecturer did not know herself. N…

> As defined by the lecturer herself, "h(t) = h(t-1)", the very definition of hysteresis. How is that a definition of hysteresis? Hysteresis is when state is a function of previous state, not identical to previous state.

It's just simplified pseudoscope using the lecturer's own notation from her slides to make my point.

The following is the lecturer's full TeX form if that helps:

h(t) = \tanh \left(h(t-1) W_{\text{hh}}^T+x(t) W_{\text{hx}}^T\right)

However, I don't want our readers to get distracted by line noise; h(t) = h(t - 1) makes my point.

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#63

Post by rattray » Mon, Apr 03, 2023, 3:18 AM UTC

Are these the sorts of Transformers and "Attention" heads that play a prominent role in LLMs like GPT3 et al? Would this lecture help build a foundational understanding of those technologies?