Recently, I started a personal project to build an LLM from zero. I've spent a ton of time reading up on math, ML, and DL through books, open courses, and papers, while also studying all the major open-source LLM architectures. Since I only have one DGX Spark machine to run experiments, I can't train a massive LLM from the get-go. Instead, I'm experimenting with an auto-scaling parameter mechanism, which has led me t…
Do share! I read all the blog posts where people share their experiences of building small scale LLMs "from scratch".
[dead]