I built a large language model "from scratch"
brettgfitzgerald.com
I built a large language model "from scratch"
1–10 of 17 posts
Re: I built a large language model "from scratch"
#2Worthwhile read. It helps to learn (or confirm) how things are working under the hood. What's interesting is that understanding how it all works makes it clear that all the hype around models "thinking" or being "sentient" is just marketing fluff for "the math works and it's really impressive how that translates to human-like cognition."
Re: I built a large language model "from scratch"
#3I don't see any description of the resulting model in the post. Or any results for that matter. Reads more like a book plug.
Am I missing something?
Re: I built a large language model "from scratch"
#4My wife and I are working through this book, and while I see its value and plan to finish it, I've found it (thus far) a bit underwhelming. It feels like a collection of Jupyter notebooks stitched together with a loosely edited narrative. Concepts are sometimes introduced without explanation, instructions lack context, and the growing errata list on Manning's website makes me question if I'm absorbing the right information.
Re: I built a large language model "from scratch"
#5My wife and I are working through this book, and while I see its value and plan to finish it, I've found it (thus far) a bit underwhelming. It feels like a collection of Jupyter notebooks stitched together with a loosely edited narrative. Concepts are sometimes introduced without explanation, instructions lack context, and the growing errata list on Manning's website makes me question if I'm absorbing the right infor…
Also went through this book recently and I agree with you. I found it difficult to build a working LLM using the code snippets included in the book.
Re: I built a large language model "from scratch"
#6I've found none of the explanations of how LLMs are built have been satisfying, especially considering how impressive the applications of them are.
Re: I built a large language model "from scratch"
#7I've found none of the explanations of how LLMs are built have been satisfying, especially considering how impressive the applications of them are.
Karpathy's recent video[1] is quite good.
1. Deep Dive into LLMs like ChatGPT (https://youtube.com/watch?v=7xTGNNLPyMI)
Re: I built a large language model "from scratch"
#8I've found none of the explanations of how LLMs are built have been satisfying, especially considering how impressive the applications of them are.
Karpathy's recent video[1] is quite good. 1. Deep Dive into LLMs like ChatGPT ( https://youtube.com/watch?v=7xTGNNLPyMI )
It's a good introduction at the pop-sci level, but most technically-inclined HN'ers are probably going to get more benefit from his earlier Zero to Hero series.
Re: I built a large language model "from scratch"
#9I've found none of the explanations of how LLMs are built have been satisfying, especially considering how impressive the applications of them are.
To get the rough idea how things work, you can watch the Karpathy's series on YouTube. To get the actual understanding how things work, you will have to read through the papers. You probably want both. Finally, to really understand how it works then there's no better way than implementing an inference engine yourself. All other material I also found to be superficial and not satisfactory ... too much information at the hand-waving level.
Re: I built a large language model "from scratch"
#10I don't see any description of the resulting model in the post. Or any results for that matter. Reads more like a book plug. Am I missing something?
I get the same vibe, especially after reading the update where the book author contacts him to clarify stuff.
The whole post reads like hasty clumsy grey marketing.