Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

youtube.com

Post reply on HN
Print view View original

41–50 of 63 posts

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#41

Post by tysam_and »

matt_langston wrote: post #4

I wonder why that parameter is called "h"? Hmmm ... Why not just say the word "hysteresis" and bring some magnets to class for show-and-tell to help your students develop an intuition for the "h" parameter in RNNs.

Like jimsimmons said below, I believe it traditionally refers to 'hidden'. which was in vogue at the time for both feedforward nets and RNNs as well as any other other neural networks in the 90's or so and on. This trend actually continued for a while and I learned it in one of Hintons' main online classes which was made somewhere between 2012-2015 or so IIRC (though I opted to switch to reading and trying to implement raw papers instead as my brain works intuitively strangely, on the whole).

You can think of it as everything the RNN knows about what you're doing and a thing that evolves from place to place as you go. Because it is iterated on itself as a map, it abides by some very interesting properties that let it represent some very difficult functions, though actually attaining a representation of those functions is rather difficult indeed in my experience from what I've seen.

There are one or two rather successful projects trying to keep RNNs both alive and competitive with transformers. I think they do very well on the whole, though the transformers seem to have slightly improved parameter efficiency, generally speaking.

I hope this helps you with your question, please do let me know if you have any other follow up questions on this topic/matter. (: (: :) :)

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#42

Post by tysam_and »

ramraj07 wrote: post #13

Which is the best course or set of videos to learn the basics of neural networks and deep learning? Something that really gets the best explanation of things like backprop?

Karpathy's zero to hero series is excellent, and I really recommend it.

I also made a few repos that are geared around readability and being a good 'working code demonstration' of certain best-practices in neural networks. If you're like me and you grok code better than symbols, this could be a helpful adjunct as well if you're wanting to dig deep a bit.

https://github.com/tysam-code/hlb-CIFAR10 (convolutional networks and imaging stuff. it's fast. very fast.)

https://github.com/tysam-code/hlb-gpt (pruned-down base of nanoGPT with training-speed-focused changes built on top of it. Check out the 0.0.0 tag from the repo if you want the barest of bare-bones implementations. Thanks!)

Both of these implementations are pretty straightforward for what they do but CIFAR-10 has less dynamic scheduling and stuff so it might be easier to fit in your head. However, both are meant to be simple (and extremely hackable if you want to poke around and take apart some pieces/add different watchpoints to see how different pieces evolve, etc. I am partially inspired by, among many things, one of those see-through engine kits that I saw in a magazine growing up as a child that I thought was a very cool, dynamic, and hands-on way to just watch how the pieces moved in a difficult topic. Sometimes that is the best way that our brains can learn, though we are all different and learn best differently through different mediums in my experience).

Feel free to let me know if you have any specific questions and I'll endeavor to do my best to help you here. Welcome to an interest in the field!

I guess to briefly touch on one topic -- some people focus on the technical only first, like backprop, and though math is required heavily for more advanced research, I don't learn concepts very well through details only. Knowing that backprop is "Calculate the slope for the error in this high-dimensional space for how a neural network was wrong at a certain point, then take a tiny step towards minimizing the error. After N steps, we converge to a representation that is like a zip file of our input data within a mathematical function" is probably enough for 90-95% of the usecases you will do as a ML practitioner, if you do so. The math is cool but there are more important things to sweat over IMO, and I think messaging to the contrary raises the barrier to entry to the field and distracts from the important things, which we do not need as much. It's good to learn after you have space in your brain for it after you understand how the whole thing works together, though that is just my personal opinion after all.

Much love and care and all that and again feel free to let me know if you have any questions please. :) <3

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#43

Post by tysam_and »

rlt wrote: post #31
fnbr wrote: post #28

Earlier quoted context omitted.

It is not common for new projects. The vast majority of new projects use PyTorch, with some using tensorflow and some using JAX.

Can PyTorch be used to implement LLMs / GPT?

quite easily, yes

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#44

Post by ta8903 »

joshvm wrote: post #33
ramraj07 wrote: post #13

Which is the best course or set of videos to learn the basics of neural networks and deep learning? Something that really gets the best explanation of things like backprop?

http://cs231n.stanford.edu/ cs231n is still a really solid course, despite the more recent lecture videos not being publicly available.

[flagged]

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#45

Post by wslh »

ramraj07 wrote: post #13

Which is the best course or set of videos to learn the basics of neural networks and deep learning? Something that really gets the best explanation of things like backprop?

Since the topic of this thread is this MIT course, Is this video great for you? I liked it.

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#46

Post by armcat »

ramraj07 wrote: post #13

Which is the best course or set of videos to learn the basics of neural networks and deep learning? Something that really gets the best explanation of things like backprop?

Not a course, but I would highly recommend Deep Learning with Python, by Francois Chollet, creator of Keras. Incredibly approachable book that covers everything from tensors and backprop, to mixed precision and multi gpu scaling, and includes time series, language, vision and audio in between

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#47

Post by credit_guy »

sudosysgen wrote: post #23
credit_guy wrote: post #16

Earlier quoted context omitted.

> They have blood on their hands and history of silencing and oppressing dissidents such as Aaron Schwartz. I guess what you have in mind is that Aaron Swartz was arrested by the MIT Campus Police (together with a Secret Service agent). This is not oppression by MIT. The MIT Campus Police received the arrest warrant from a federal prosecutor (Carmen Ortiz) and they needed to carry out the arrest. They did not have a…

Just because other militaries are bad doesn't make the US military good. It's perfectly analogous in its wars of agression, and supporting it is morally bad, no matter how hard you whatabout it.

Ukraine has great fighters and great spirit. But fighters and spirit can take you only that far. Without the Javelins and Himars, Ukraine would have collapsed in a matter of weeks. Without a doubt some of the tech that goes in those Javelins and Himars was invented at MIT.

This is not whataboutism. This is how you stop tyranny.

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#48

Post by sudosysgen »

credit_guy wrote: post #47
sudosysgen wrote: post #23

Earlier quoted context omitted.

Just because other militaries are bad doesn't make the US military good. It's perfectly analogous in its wars of agression, and supporting it is morally bad, no matter how hard you whatabout it.

Ukraine has great fighters and great spirit. But fighters and spirit can take you only that far. Without the Javelins and Himars, Ukraine would have collapsed in a matter of weeks. Without a doubt some of the tech that goes in those Javelins and Himars was invented at MIT. This is not whataboutism. This is how you stop tyranny.

The weapons you're talking about killed over a million people in tyrannical wars and terrorism. They are weapons of tyranny.

Re: MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]

#50

Post by matt_langston »

matt_langston wrote: post #4

I wonder why that parameter is called "h"? Hmmm ... Why not just say the word "hysteresis" and bring some magnets to class for show-and-tell to help your students develop an intuition for the "h" parameter in RNNs.

As defined by the lecturer herself, "h(t) = h(t-1)", the very definition of hysteresis.

My point is that the lecturer missed a golden opportunity to give her students a natural intuition of "h" that they can see, feel and touch and that will serve them well for their entire careers.

The only thing "hidden" about "h" is that hysteresis is hidden in plain site in her lecture - maybe the lecturer did not know herself.

Neural networks have an undeserved reputation for being mysterious, and maybe that is partly due to a lack of basic physics knowledge.

Post reply on HN