Live data from Hacker News

Theoretical limitations of multi-layer Transformer

arxiv.org

21–24 of 24 posts

Re: Theoretical limitations of multi-layer Transformer

#21
post #7

Loosely related thought: A year ago, there was a lot of talk about the Mamba SSM architecture replacing transformers. Apparently that didn't happen so far.

Just like with neural networks and Adam [1], LLMs evolve to make transformers their best building block.

[1] https://parameterfree.com/2020/12/06/neural-network-maybe-ev...

Re: Theoretical limitations of multi-layer Transformer

#24
post #8
post #4

Earlier quoted context omitted.

Most of it is linear algebra and convex optimization. You can learn a lot of it with free resources from MIT, Stanford, Georgia Tech, or YouTube. If you want more of a school style learning environment you can enroll in the Georgia Tech OMSCS program and just take the classes related to the math etc that you are interested in. No reason you have to graduate and it is maybe $800 a course.

Thanks! Now might actually be a great time for me to pick up the material. If anyone has suggestions about particularly good free/online mathematics courses for ML, I'd really love to hear it. Or books!

ML texts often have refresher appendixes/intro chapters:

- https://www.deeplearningbook.org/

- https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearnin...

___________________________

and math for physics texts by these authors shd be in any technical lib

- Mary Boas

- Arfken Webber Harris

Post reply on HN