Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Engineer’s Guide to Deep Learning: Understanding the Transformer Model

interdb.jp

Post reply on HN
Print view View original

31–35 of 35 posts

Re: The Engineer’s Guide to Deep Learning: Understanding the Transformer Model

#31

Post by mr_puzzled »

trybackprop wrote: post #30
mr_puzzled wrote: post #24

Earlier quoted context omitted.

Slightly off topic: I'm interested in taking part in the Vesuvius challenge[0], but I don't have a background in ML, just a regular web developer. Does anyone have suggestions on how to get started? I planned to get some background on practical ML by working through Karpathy's Zero to Hero series along with the Understanding Deep Learning book. Would that be enough or anything else I should learn? I plan to understan…

I made a list of all the free resources I used to study ML and deep learning to become an ML engineer at FAANG, so I think it'll be helpful to follow these resources: https://www.trybackprop.com/blog/top_ml_learning_resources (links in the blog post) Fundamentals Linear Algebra – 3Blue1Brown's Essence of Linear Algebra series, binged all these videos on a one hour train ride visiting my parents Multivariable Calculus…

Wow, thanks for the links to all the resources. Lot of interesting stuff for me to learn!

Re: The Engineer’s Guide to Deep Learning: Understanding the Transformer Model

#33

Post by uoaei »

One of the most frustrating things about all the documentation on Transformers is the sole emphasis on NLP.

In particular, one of the most interesting parts of the Transformer architecture to me is the attention mechanism which is permutation invariant (if not for the positional embeddings people use to counteract this inherent quality of attention layers). Also the ability to arbitrarily mask this or that node in the graph -- or even individual edges -- gives the whole thing so much flexibility for encoding domain knowledge into your architecture.

Positional embeddings may still be required in many cases but you can be clever about them beyond the overly restrictive perspective of attention layer inputs purely as one-dimensional sequences.

Re: The Engineer’s Guide to Deep Learning: Understanding the Transformer Model

#35

Post by bitmaks »

rvnx wrote: post #18
MAXPOOL wrote: post #10

There are many others that are better. 1/ The Annotated Transformer Attention is All You Need http://nlp.seas.harvard.edu/annotated-transformer/ 2/ Transformers from Scratch https://e2eml.school/transformers.html 3/ Andrej Karpathy has really good series of intros: https://karpathy.ai/zero-to-hero.html Let's build GPT: from scratch, in code, spelled out. https://www.youtube.com/watch?v=kCc8FmEb1nY GPT with Andrej Kar…

In addition, these websites are totally free. The website listed here: > I consider requests for full commercial use of all content on this site (and the github repository). For a complete buyout of all content rights, the cost is €10,000,000. > I’d like to ask you what problems you have by that I keep on having the copyright of my document. + no commercial-use without paying 20% royalty. So fairly expensive for a Ke…

I think that's pretty obviously a joke, no?
Post reply on HN