[dead]
Show HN: Fine-tune an 8B model on a 4 GB laptop GPU
31–40 of 47 posts
Re: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU
#32Small open weight local models are the future. While hosted mega models make headlines for doing cool stuff, the vast majority of applications for AI simply don't need all that power, and thus cost. That’s a big part of why businesses are screaming that there’s no ROI from AI. Brining this tech down into small local models is likely where this all converges for the vast majority of use cases and what solves the prese…
If you are into small local models I highly recommend vibe thinker. It's a model trained specifically for reasoning. Basically a problem solver. When compared with other models, on math problems benchmarks, it's closer to models hundred times its size than ten times its size which it beats comfortably. It supports long contexts on limited VRAM and is blazing fast. https://github.com/WeiboAI/VibeThinker
this is great! thanks for sharing
Re: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU
#33Re: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU
#34Earlier quoted context omitted.
Can you just get over it and read what's already there? Anyway, your loss.
If their native language isn’t English then I’ll come off my pedestal
It isn't. Kazakh and Russian.
I said this further down but that comment is dead so you would not have
seen it. The later replies are mine, written by me.
Re: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU
#35There are some samples of training data in this folder - https://github.com/MakazhanAlpamys/Soup/tree/main/examples/d...
They're all very short though. Anyone got a good rule for how much data of this nature is needed to successfully fine-tune a model of this size?
Re: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU
#36[dead]
Re: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU
#37There are some samples of training data in this folder - https://github.com/MakazhanAlpamys/Soup/tree/main/examples/d... They're all very short though. Anyone got a good rule for how much data of this nature is needed to successfully fine-tune a model of this size?
[dead]
Re: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU
#38There are some samples of training data in this folder - https://github.com/MakazhanAlpamys/Soup/tree/main/examples/d... They're all very short though. Anyone got a good rule for how much data of this nature is needed to successfully fine-tune a model of this size?
Those are format examples and test fixtures. Five to ten rows each. Not training data.
You did spot a real problem though. Eight configs in `examples/configs` pointed at those fixtures as training data. Seven were still on the old schema and would not even parse. I've fixed that, added a README to the folder, and a test that parses every config so it doesn't quietly break again.
Dataset size mostly depends on the task. Format or style, a few hundred examples is often enough. A task the model already half knows, usually a few thousand. New facts are often a bad fit for fine-tuning. I'd reach for RAG instead.
I haven't measured how this changes with model size, so I'm not going to make up a number.
Re: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU
#39This looks cool - what 4 GB GPU laptop do you recommend?
Do not buy a 4 GB card for this. Mine is an RTX 3050 Laptop, I picked it because it is boring hardware that many people already have. If you are buying, buy VRAM. At 0.5B where I could measure both, resident training was 1.43x faster than streaming. And if you do stream, system RAM matters more, the base sits there and has to page-lock. 16 GB is about the floor for 8B.
what price is the cheapest
Re: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU
#40Earlier quoted context omitted.
Can you just get over it and read what's already there? Anyway, your loss.
If their native language isn’t English then I’ll come off my pedestal
Still, there is a big difference between "Translate my answer into English: " and "Answer this question: ".