The Illustrated Transformer (2018)
jalammar.github.io
The Illustrated Transformer (2018)
1–10 of 14 posts
Re: The Illustrated Transformer (2018)
#2I gave a talk on using Google BERT for financial services problems at a machine learning conference in early 2019. During my preparation, this was the only resource on transformers I could find that was even remotely understandable to me.
I had a lot of trouble understand what was going on from just the original publication[0].
Re: The Illustrated Transformer (2018)
#3[deleted]
Re: The Illustrated Transformer (2018)
#4I gave a talk on using Google BERT for financial services problems at a machine learning conference in early 2019. During my preparation, this was the only resource on transformers I could find that was even remotely understandable to me. I had a lot of trouble understand what was going on from just the original publication[0]. [0] https://arxiv.org/abs/1706.03762
Maybe Im dumb but I still can't make much sense of this.
Re: The Illustrated Transformer (2018)
#5I gave a talk on using Google BERT for financial services problems at a machine learning conference in early 2019. During my preparation, this was the only resource on transformers I could find that was even remotely understandable to me. I had a lot of trouble understand what was going on from just the original publication[0]. [0] https://arxiv.org/abs/1706.03762
[deleted]
Re: The Illustrated Transformer (2018)
#6I go back regligiously to this post whenever I need a quick visual refresh on how transformers work, I can’t overstate how fantastic it is
Re: The Illustrated Transformer (2018)
#7Illustrated Transformer is amazing as a way of understanding the original transformer architecture step-by-step, but if you want to truly visualize how information flows through a decoder-only architecture - from nanoGPT all the way up to a fully represented GPT-3 - nothing beats this:
Re: The Illustrated Transformer (2018)
#8Illustrated Transformer is amazing as a way of understanding the original transformer architecture step-by-step, but if you want to truly visualize how information flows through a decoder-only architecture - from nanoGPT all the way up to a fully represented GPT-3 - nothing beats this: https://bbycroft.net/llm
whoa, that's awesome.
Re: The Illustrated Transformer (2018)
#9I gave a talk on using Google BERT for financial services problems at a machine learning conference in early 2019. During my preparation, this was the only resource on transformers I could find that was even remotely understandable to me. I had a lot of trouble understand what was going on from just the original publication[0]. [0] https://arxiv.org/abs/1706.03762
Maybe it's easier to understand in the format of annotated code
Re: The Illustrated Transformer (2018)
#10While I absolutely love this illustration (and frankly everything Jay Alammar does), it is worth recognizing there is a distinction between visualizing how a transformer (or any model really works) and what the transformer is doing.
My favorite article on the latter is Cosma Shalizi's excellent post showing that all "attention" is really doing is kernel smoothing [0]. Personally having this 'click' was a bigger insight for me than walking through this post and implementing "attention is all you need".
In a very real sense transformers are just performing compression and providing a soft lookup functionality on top of an unimaginably large dataset (basically the majority of human writing). This understanding of LLMs helps to better understand their limitations as well as their, imho untapped, usefulness.
0. http://bactra.org/notebooks/nn-attention-and-transformers.ht...