[flagged]
Training a 3.8B LLM to 0.384 CORE for $998
11–20 of 22 posts
Re: Training a 3.8B LLM to 0.384 CORE for $998
#12This is really neat! I'd be tempted to try this again targetting ~1B params and the entire cookbook of "current" small model ideas: gated delta nets (or is ~2K context too short to benefit?), per layer or n-gram embeddings, gated residuals etc.
Re: Training a 3.8B LLM to 0.384 CORE for $998
#13I don’t blame you for using an LLM to write an article about an LLM that you built. I’ve been reading so much LLM output that now I see it everywhere. I wonder if humans will start writing more like LLMs?
When we hyper focus on finding something, we find it all the time. The world of tiny LLMs is so interesting. It is unlocking novel ways to encode information. Why focus on the style of writing instead of the subject matter?
> Why focus on the style of writing instead of the subject matter?
Because just as with human writing, a poor writing style obscures the topic where it should illuminate
Re: Training a 3.8B LLM to 0.384 CORE for $998
#14[flagged]
Re: Training a 3.8B LLM to 0.384 CORE for $998
#15I don’t blame you for using an LLM to write an article about an LLM that you built. I’ve been reading so much LLM output that now I see it everywhere. I wonder if humans will start writing more like LLMs?
I realized the same while reading it, but my impression is that the output has been well edited. The typical Claude density has been pulled out into something more human readable, but with many of the Claud-isms still intact.
Personally I find this a productive way to produce documents. Claude makes the first pass and I edit line by line. It’s a good understanding check. If I can’t reword something into my own voice it means I don’t understand it. Then have Claude loop again to fact check the rewrites and repeat until clear (or I disagree with Claude on all it’s remaining points).
That’s the same edit loop I used without LLMs, but I get speed in drafting up front (and fact/number pulling) and the check passes make the edits better.
Re: Training a 3.8B LLM to 0.384 CORE for $998
#16[flagged]
Re: Training a 3.8B LLM to 0.384 CORE for $998
#17More and more such experiments. I felt sad for a couple months when I realized that writing code will not be the same since. Now I am on the other side. LLMs are interesting in their own ways but as an engineer, this is a way to unlock a new way of building software. I recently build a Claude-assisted Excel/CSV parser for a US based property management system (tax compliance). Uses Haiku and has a lot of deterministi…
I think I am on your arc as well. My learning on different topics is growing every day, but there’s a limit to how much I can absorb. With the LLMs the experiments stay just beyond that horizon and I keep chasing.
Stated too strongly, but I think this could be the model for education (some subjects anyway). Everything personalized to your learning goals, grounded in experiments that give a tight feedback loop and with a model that never gets tired of re-explaining something for the 10th time.
Re: Training a 3.8B LLM to 0.384 CORE for $998
#18More and more such experiments. I felt sad for a couple months when I realized that writing code will not be the same since. Now I am on the other side. LLMs are interesting in their own ways but as an engineer, this is a way to unlock a new way of building software. I recently build a Claude-assisted Excel/CSV parser for a US based property management system (tax compliance). Uses Haiku and has a lot of deterministi…
I think I am on your arc as well. My learning on different topics is growing every day, but there’s a limit to how much I can absorb. With the LLMs the experiments stay just beyond that horizon and I keep chasing. Stated too strongly, but I think this could be the model for education (some subjects anyway). Everything personalized to your learning goals, grounded in experiments that give a tight feedback loop and wit…
Yes they are great for learning at own pace, trying out new things.
I have accepted two things that make me a happy engineer now: AGI is not here no matter what they say and LLMs are still very useful if one knows how to use them.
They are another layer of abstraction and like you said they do not tire. There is a lot of optimization needed so we can reduce wastage (running 1T+ LLMs for most work is wastage).
Re: Training a 3.8B LLM to 0.384 CORE for $998
#19I hope we will soon have an open-source project for training such small LLMs where one can just pick the architecture (like Qwen / DeepSeek / etc.), parameter count, dataset, ... and then let it run on a local/rented GPUs...
Re: Training a 3.8B LLM to 0.384 CORE for $998
#20I don’t blame you for using an LLM to write an article about an LLM that you built. I’ve been reading so much LLM output that now I see it everywhere. I wonder if humans will start writing more like LLMs?
Pangram says 50% human. That seems to check out, heavily edited yet the LLM aha sentences and cadence are still definitely there, but not overbearing.
I read some Dostoevsky recently and I found my speech to be substantially altered for a while, so it makes sense that LLM writing would have an influence.