Wish it were mobile friendly.
LLM Visualization
71–80 of 138 posts
Re: LLM Visualization
#72Earlier quoted context omitted.
It’s definitely not obvious no matter how smart you are! The common metaphor used is it’s like a conversation. Imagine you read one comment in some forum, posted in a long conversation thread. It wouldn’t be obvious what’s going on unless you read more of the thread right? A single paper is like a single comment, in a thread that goes on for years and years. For example, why don’t papers explain what tokens/vectors/e…
Is there a way of finding interesting "chains" of such papers, short of scanning the references / "cited by" page? (For example, Google Scholar lists 98797 citations for Attention is all you need!)
As a prerequisite to the attention paper? One to check out is:
A Survey on Contextual Embeddings https://arxiv.org/abs/2003.07278
Embeddings are sort of what all this stuff is built on so it should help demystify the newer papers (it’s actually newer than the attention paper but a better overview than starting with the older word2vec paper).
Then after the attention paper an important one is:
Language Models are Few-Shot Learners https://arxiv.org/abs/2005.14165
I’m intentionally trying to not give a big list because they’re so time-consuming. I’m sure you’ll quickly branch out based on your interests.
Re: LLM Visualization
#73Honestly reading the pytorch implementation of minGTP is a lot more informative than an inscrutable 3d rendering. It's a well commended and pedagogical implementation. I applaud the intention, and it looks slick, but I'm not sure it really conveys information in an efficient way.
Re: LLM Visualization
#74Earlier quoted context omitted.
Isn't it amazing that a random person on the internet can produce free educational content that trumps university courses? With all the resources and expertise that universities have, why do they get shown up all the time? Do they just not know how to educate?
You're betting on the hundreds of top university cs professors to produce better content than the hundreds of thousands of industry veterans or hobbyists... Why does YouTube sometimes have better content than professionally produced media? It's a really long tail of creators and educators
This isn't new. Textbooks exist for the same reason, so we don't need to duplicate effort creating teaching materials and can have a kind of accepted core cirriculum.
Re: LLM Visualization
#75A lot of transformer explanations fail to mention what makes self attention so powerful. Unlike traditional neural networks with fixed weights, self-attention layers adaptively weight connections between inputs based on context. This allows transformers to accomplish in a single layer what would take traditional networks multiple layers.
Just to add on, a good way to learn these terms is to look at the history of neural networks rather than looking at transformer architecture in a vacuum
This [1] post from 2021 goes over attention mechanisms as applied to RNN / LSTM networks. It's visual and goes into a bit more detail, and I've personally found RNN / LSTM networks easier to understand intuitively.
[1] https://medium.com/swlh/a-simple-overview-of-rnn-lstm-and-at...
Re: LLM Visualization
#76Earlier quoted context omitted.
The audience of this paper are other researchers who already know the concept of attention, which was very well known already in the field. In such research papers, such things are never explained again, as all the researchers already know this or can read other sources, which are cited, but focus on the actual research questions. In this case, the research question was simply: Can we get away by just using attention…
What books cover exclusively about this topic ? Thanks
This is frequently a topic here on HN. E.g.:
https://udlbook.github.io/udlbook/ (https://news.ycombinator.com/item?id=38424939)
https://fleuret.org/francois/lbdl.html (https://news.ycombinator.com/item?id=35767789)
https://www.fast.ai/ (https://news.ycombinator.com/item?id=24237207)
https://d2l.ai/ (https://news.ycombinator.com/item?id=38428225)
Some more:
https://news.ycombinator.com/item?id=35543774
There is a lot more. Just google for "deep learning", and you'll find a lot of content. And most of that will cover attention, as it is a really basic concept now.
Re: LLM Visualization
#77Thank you. I'd just completed doing this manually (much uglier and less accurate) and so can really appreciate the effort behind this.
Re: LLM Visualization
#78The score on this post just went down by a factor of 10 and the time went to "1 hour" ago?!
Re: LLM Visualization
#79This is by far the best resource I've seen to understand LLMs. Incredibly well done! Thanks for this awesome tool
Re: LLM Visualization
#80What happened to this thread? When I saw it before it had 700+ upvotes.