21–30 of 47 posts
Author here. The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one…
I run a fine-tuned 4B for AML compliance at community banks — the ROI math is exactly this
Author here. The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one…
Can you write one answer that isn't AI-sourced?
Tangential/meta: Holy shit, I've never seen a thread where almost half the comments are dead (and LLM written), especially for a post that's (currently) at 86 points and 20 comments (4x ratio is "pretty good quality" post signal generally for me).
Writing them myself now.
This seems really interesting - I was curious about this line from the website. “The whole post-training stack in one CLI. Soup doctors your data pre-flight, picks the method, writes the config, derives evals from your own data, gates every save, and self-corrects reward hacking mid-run instead of just halting.” How does soup auto tune the hyper parameters and make some of these more complex training decisions?
How much data do you need to fine tune a model?
New facts is where people waste a week. The model comes back wrong in a new way. Use RAG for facts.
Why is there still a hard VRAM requirement that's dependent on the model size? Isn't that exactly what this project is supposed to solve?
So it goes from "whole model must fit" to "embeddings plus one layer plus logits must fit". That is why 8B works and why I did not try 14B.
The table on the site is the normal resident path, streaming is opt-in and BETA. Should be clearer, my fault.
This looks cool - what 4 GB GPU laptop do you recommend?
Author here. The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one…
Can you write one answer that isn't AI-sourced?