131–140 of 563 posts
Earlier quoted context omitted.
How much VRAM does it need? I haven't run a local model yet, but I did recently pick up a 16GB GPU, before they were discontinued.
If you have to ask then your GPU is too small. With 16 GB you'll be only able to run a very compressed variant with noticable quality loss.
I recall a Qwen exec posted a public poll on Twitter, asking which model from Qwen3.6 you want to see open-sourced; and the 27b variant was by far the most popular choice. Not sure why they ignored it lol.
I'm guessing 3.5-27b would beat 3.6-35b. MoE is a bad idea. Because for the same VRAM 27b would leave a lot more room, and the quality of work directly depends on context size, not just the "B" number.
I'm a newbie here and lost how I'm supposed to use these models for coding. When I use them with Continue in VSCode and start typing basic C: #include int m I get nonsensical autocompletions like: #include int m What is going on?
I'm broadly curious how people are using these local models. Literally, how are they attaching harnesses to this and finding more value than just renting tokens from Anthropic of OpenAI?
I'm broadly curious how people are using these local models. Literally, how are they attaching harnesses to this and finding more value than just renting tokens from Anthropic of OpenAI?
The local models don’t really compete with the flagship labs for most tasks
But there are things you may not want to send to them for privacy reasons or tasks where you don’t want to use tokens from your plan with whichever lab. Things like openclaw use a ton of tokens and most of the time the local models are totally fine for it (assuming you find it useful which is a whole different discussion)
Earlier quoted context omitted.
How much VRAM does it need? I haven't run a local model yet, but I did recently pick up a 16GB GPU, before they were discontinued.
It's on the page: Precision Quantization Tag File Size 1-bit UD-IQ1_M 10 GB 2-bit UD-IQ2_XXS 10.8 GB UD-Q2_K_XL 12.3 GB 3-bit UD-IQ3_XXS 13.2 GB UD-Q3_K_XL 16.8 GB 4-bit UD-IQ4_XS 17.7 GB UD-Q4_K_XL 22.4 GB 5-bit UD-Q5_K_XL 26.6 GB 16-bit BF16 69.4 GB
I'm a newbie here and lost how I'm supposed to use these models for coding. When I use them with Continue in VSCode and start typing basic C: #include int m I get nonsensical autocompletions like: #include int m What is going on?
These are not autocomplete models. It’s built to be used with an agentic coding harness like Pi or OpenCode.
Qwen specifically calls out FIM (“fill in the middle”) support on the model card and you can see it getting confused and posting the control tokens in the example here.
Earlier quoted context omitted.
If you have to ask then your GPU is too small. With 16 GB you'll be only able to run a very compressed variant with noticable quality loss.
> If you have to ask then your GPU is too small. What's the minimum memory you need to run a decent model? Is it pretty much only doable by people running Macs with unified memory?
When I get home today I totally look forward to trying the unsloth variants of this out (assuming I can get it working in anything.) I expect due to the limited active parameter count it should perform very well. It's obviously going to be a long time before you can run current frontier quality models at home for less than the price of a car, but it does seem like it is bound to happen. (As long as we don't allow general purpose computers to die or become inaccessible. Surely...)
How does this compare to the commercial models like Sonnet 4.5 or GPT? Close enough that the price is right (free)?
I think its worth noting that if you are paying for electricity Local LLM is NOT free. In most cases you will find that Haiku is cheaper, faster, and better than anything that will run on your local machine.