Making a vintage LLM from scratch
crlf.link
Making a vintage LLM from scratch
1–10 of 32 posts
Re: Making a vintage LLM from scratch
#2I am creating my tiny Llama 340M base model from scratch. If you're curious about the steps, challenges and cost, read on. I am still working on the instruct model.
Re: Making a vintage LLM from scratch
#3Nice project. I’m curious to see how it writes after instruct.
Re: Making a vintage LLM from scratch
#4There are certain things you can only truly learn by doing. I remember doing Linux From Scratch over a weekend and the depth of linux that I still understand to this day.
Thanks for the writeup. A more granular followup would be cool too.
Re: Making a vintage LLM from scratch
#5There are certain things you can only truly learn by doing. I remember doing Linux From Scratch over a weekend and the depth of linux that I still understand to this day. Thanks for the writeup. A more granular followup would be cool too.
Except in this case he vibe-coded it
Re: Making a vintage LLM from scratch
#6"The code is semi-vibe-coded with whatever LLM I had with VS-Code and PI (OpenRouter models)."
I appreciate the honesty, but now there's no journey, and that's what I'm interested in. I can ask a LLM myself.
Re: Making a vintage LLM from scratch
#7> These samples have very good scores overall, but they are useless. I am guessing it's not English text... I counted a few hundred examples mostly from LOC-PD and other few hundred in the OTA datasets. Imagine if I feed that crap to my LLM, what will it learn?
im pretty sure its a real text in Welsh. there might be typos from ocr but yeah thats what the language really looks like, i dont speak it but its easy to recognize.
Re: Making a vintage LLM from scratch
#8> These samples have very good scores overall, but they are useless. I am guessing it's not English text... I counted a few hundred examples mostly from LOC-PD and other few hundred in the OTA datasets. Imagine if I feed that crap to my LLM, what will it learn? im pretty sure its a real text in Welsh. there might be typos from ocr but yeah thats what the language really looks like, i dont speak it but its easy to rec…
It looks like ROT13 text to me, I hope it's not Welsh. Don't want to offend anyone if that's their actual language :)
Re: Making a vintage LLM from scratch
#9> These samples have very good scores overall, but they are useless. I am guessing it's not English text... I counted a few hundred examples mostly from LOC-PD and other few hundred in the OTA datasets. Imagine if I feed that crap to my LLM, what will it learn? im pretty sure its a real text in Welsh. there might be typos from ocr but yeah thats what the language really looks like, i dont speak it but its easy to rec…
It looks like ROT13 text to me, I hope it's not Welsh. Don't want to offend anyone if that's their actual language :)
It's actually Welsh, and the funny thing is that one of the sentences in the example "gibberish" text (although with some further OCR errors) means:
"It will be easy for the knowledgeable to fix the few errors that remain [in the text]". (Bydd yn rwydd iawn i'r cyfarwydd ddiwygio'r ychydig.")
Which is exactly what the OP is doing.
Re: Making a vintage LLM from scratch
#10Instead of always trying to make models more current and general, there may be value in making them deliberately narrow, historically constrained and weird in a well-defined way