11–20 of 32 posts
> These samples have very good scores overall, but they are useless. I am guessing it's not English text... I counted a few hundred examples mostly from LOC-PD and other few hundred in the OTA datasets. Imagine if I feed that crap to my LLM, what will it learn? im pretty sure its a real text in Welsh. there might be typos from ocr but yeah thats what the language really looks like, i dont speak it but its easy to rec…
There are certain things you can only truly learn by doing. I remember doing Linux From Scratch over a weekend and the depth of linux that I still understand to this day. Thanks for the writeup. A more granular followup would be cool too.
I am creating my tiny Llama 340M base model from scratch. If you're curious about the steps, challenges and cost, read on. I am still working on the instruct model.
"The code is semi-vibe-coded with whatever LLM I had with VS-Code and PI (OpenRouter models)." I appreciate the honesty, but now there's no journey, and that's what I'm interested in. I can ask a LLM myself.
I am creating my tiny Llama 340M base model from scratch. If you're curious about the steps, challenges and cost, read on. I am still working on the instruct model.
I feel like this is the true frontier, making smaller models that can do more than their predecessors. If we can crack this space to where you can get reasonable outputs from "mediocre hardware" it would be worthwhile, even if its somewhat inferior to frontier models, we can't forget that not long ago, frontier models are nowhere near as good as they are today, and tomorrow's models will likely be even better.
There are certain things you can only truly learn by doing. I remember doing Linux From Scratch over a weekend and the depth of linux that I still understand to this day. Thanks for the writeup. A more granular followup would be cool too.
Do you mind expanding this question? More granular in what way? what would you like to know that is missing from the post?
I am creating my tiny Llama 340M base model from scratch. If you're curious about the steps, challenges and cost, read on. I am still working on the instruct model.
I feel like this is the true frontier, making smaller models that can do more than their predecessors. If we can crack this space to where you can get reasonable outputs from "mediocre hardware" it would be worthwhile, even if its somewhat inferior to frontier models, we can't forget that not long ago, frontier models are nowhere near as good as they are today, and tomorrow's models will likely be even better.
"The code is semi-vibe-coded with whatever LLM I had with VS-Code and PI (OpenRouter models)." I appreciate the honesty, but now there's no journey, and that's what I'm interested in. I can ask a LLM myself.
There's a lot of pre-processing, experimentation and validation that went into this project. The training data collection and sanitization alone is a big undertaking.
As for the blog post itself, from the article:
> Note: This blog post is 100% written by me. No AI has been used whatsoever.
Put another way: You can ask the LLM yourself to do this project? Please do, share your prompt, I'd like to see it.