Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making a vintage LLM from scratch

crlf.link

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Making a vintage LLM from scratch

#21

Post by giancarlostoro »

croqaz wrote: post #16
giancarlostoro wrote: post #14

Earlier quoted context omitted.

I feel like this is the true frontier, making smaller models that can do more than their predecessors. If we can crack this space to where you can get reasonable outputs from "mediocre hardware" it would be worthwhile, even if its somewhat inferior to frontier models, we can't forget that not long ago, frontier models are nowhere near as good as they are today, and tomorrow's models will likely be even better.

That's exactly what I had in mind. When I started this, I was jumping back and forth between this thought: "Can this model size actually generate logical English text?" and I played with a few different models of the same size and I was really really depressed when seeing how bad they are.... but then I discovered more and more tiny models and LaMini-125M, LaMini-256M, and nanowhale-100m, and SmolLM2-135M-Instruct ar…

In my case, I have a local branch where I'm experimenting with BitNet since it can run on a CPU too.

Re: Making a vintage LLM from scratch

#22

Post by skerit »

mg794613 wrote: post #6

"The code is semi-vibe-coded with whatever LLM I had with VS-Code and PI (OpenRouter models)." I appreciate the honesty, but now there's no journey, and that's what I'm interested in. I can ask a LLM myself.

I've been creating my own little from-scratch LLM for months now with Claude's help. I can safely say I learned a thing or two along the way.

Re: Making a vintage LLM from scratch

#23

Post by skerit »

croqaz wrote: post #16
giancarlostoro wrote: post #14

Earlier quoted context omitted.

I feel like this is the true frontier, making smaller models that can do more than their predecessors. If we can crack this space to where you can get reasonable outputs from "mediocre hardware" it would be worthwhile, even if its somewhat inferior to frontier models, we can't forget that not long ago, frontier models are nowhere near as good as they are today, and tomorrow's models will likely be even better.

That's exactly what I had in mind. When I started this, I was jumping back and forth between this thought: "Can this model size actually generate logical English text?" and I played with a few different models of the same size and I was really really depressed when seeing how bad they are.... but then I discovered more and more tiny models and LaMini-125M, LaMini-256M, and nanowhale-100m, and SmolLM2-135M-Instruct ar…

I've been working on something like this too, for quite a while! Though I'm trying to get a non-quadratic-attention LLM (or SLM) up and running.

And anyway, I think the most important thing is dataset quality. Dumping in whatever dataset you find on Huggingface is a recipe for mediocrity, so I'm also spending a lot of time on that.

Re: Making a vintage LLM from scratch

#24

Post by dennysora-main »

Recently, I started a personal project to build an LLM from zero.

I've spent a ton of time reading up on math, ML, and DL through books, open courses, and papers, while also studying all the major open-source LLM architectures.

Since I only have one DGX Spark machine to run experiments, I can't train a massive LLM from the get-go. Instead, I'm experimenting with an auto-scaling parameter mechanism, which has led me to create a pretty unconventional and fun architecture!

Why go through all this effort when modern LLMs can basically write simple LLMs themselves, and I clearly can't out-compute the big tech giants?

Honestly, it's because I'm obsessed with the core mechanics of LLMs. I want to build something exclusively for myself and hopefully discover some completely undiscovered mechanisms along the way.

Just keeping a record and sharing my progress—having fun with it is truly the biggest reward!

I'll share it when I get a chance!

Re: Making a vintage LLM from scratch

#25

Post by charcircuit »

dennysora-main wrote: post #24

Recently, I started a personal project to build an LLM from zero. I've spent a ton of time reading up on math, ML, and DL through books, open courses, and papers, while also studying all the major open-source LLM architectures. Since I only have one DGX Spark machine to run experiments, I can't train a massive LLM from the get-go. Instead, I'm experimenting with an auto-scaling parameter mechanism, which has led me t…

Most hobbyists rent the compute for training models instead of needing to purchase it all out right.

Re: Making a vintage LLM from scratch

#26

Post by croqaz »

mg794613 wrote: post #6

"The code is semi-vibe-coded with whatever LLM I had with VS-Code and PI (OpenRouter models)." I appreciate the honesty, but now there's no journey, and that's what I'm interested in. I can ask a LLM myself.

That's a fair point TBH. I said in my post that this LLM is first of all a learning project and I skipped an important step: the training loop. But on the other hand, how many data scientists are writing their own training loops? Is it even worth it? And how much learning do you want for one project, I mean, where do you stop? Why use "Huggingface Transformers" when you can write it from scratch, for learning? Why use Torch when you can write it from scratch, for learning? Why use Python when you can write in C, etc. It's cheating, right? In my case, I decided to skip the training loop and focus on the data processing and the hyper params and the rest of the higher level steps that took a ton of time anyway, and I reduced the friction. I do get your point tho. Now that I know how to train an LLM, maybe I'll write a training loop from scratch as a project, to learn how to do it.

Re: Making a vintage LLM from scratch

#27

Post by croqaz »

dennysora-main wrote: post #24

Recently, I started a personal project to build an LLM from zero. I've spent a ton of time reading up on math, ML, and DL through books, open courses, and papers, while also studying all the major open-source LLM architectures. Since I only have one DGX Spark machine to run experiments, I can't train a massive LLM from the get-go. Instead, I'm experimenting with an auto-scaling parameter mechanism, which has led me t…

Do share! I read all the blog posts where people share their experiences of building small scale LLMs "from scratch".

Re: Making a vintage LLM from scratch

#29

Post by charcircuit »

cyberge99 wrote: post #4

There are certain things you can only truly learn by doing. I remember doing Linux From Scratch over a weekend and the depth of linux that I still understand to this day. Thanks for the writeup. A more granular followup would be cool too.

The depth of running configure, make, make install? If you want depth in Linux I recommend looking at its source repository and reading the documentation or code. Or in the current times asking AI to help explain it to you.

Re: Making a vintage LLM from scratch

#30

Post by dennysora-main »

charcircuit wrote: post #25
dennysora-main wrote: post #24

Recently, I started a personal project to build an LLM from zero. I've spent a ton of time reading up on math, ML, and DL through books, open courses, and papers, while also studying all the major open-source LLM architectures. Since I only have one DGX Spark machine to run experiments, I can't train a massive LLM from the get-go. Instead, I'm experimenting with an auto-scaling parameter mechanism, which has led me t…

Most hobbyists rent the compute for training models instead of needing to purchase it all out right.

It's mainly just my personal preference to run a local machine. It gives me better privacy and security, and I can keep all my heavy data and projects right there.

Cloud rentals are usually billed hourly. Since I constantly tweak the architecture and run it again, having a local rig completely kills any cost anxiety—it's just a one-off payment.

Plus, regular users can't even get access to H100s anyway. I applied on AWS and GCP before and couldn't get them.

Post reply on HN