481–490 of 563 posts
I have been using Qwen3.5-35B-A3B a lot in local testing, and it is by far the most capable model that could fit on my machine. I think quantization technology has really upped its game around these models, and there were two quants that blew me away Mudler APEX-I-Quality. then later I tried Byteshape Q3_K_S-3.40bpw Both made claims that seemed too good to be true, but I couldn't find any traces of lobotomization doi…
Earlier quoted context omitted.
I can (barely, but sustainably) run Q3.5 397B on my Mac Studio with 256GB unified. It cost $10,000 but that's well within reach for most people who are here, I expect.
$10k is well outside my budget for frivolous computer purchases.
Earlier quoted context omitted.
Hey thanks - yes agreed - for now we do: 1. Split metadata into shard 0 for huge models so 10B is for chat template fixes - however sometimes fixes cause a recalculation of the imatrix, which means all quants have to be re-made 2. Add HF discussion posts on each model talking about what changed, and on our Reddit and Twitter 3. Hugging Face XET now has de-duplication downloading of shards, so generally redownloading…
If you would know - is this also why LM Studio and Ollama model downloads often fail with a signature mismatch error?
Earlier quoted context omitted.
Hey thanks - yes agreed - for now we do: 1. Split metadata into shard 0 for huge models so 10B is for chat template fixes - however sometimes fixes cause a recalculation of the imatrix, which means all quants have to be re-made 2. Add HF discussion posts on each model talking about what changed, and on our Reddit and Twitter 3. Hugging Face XET now has de-duplication downloading of shards, so generally redownloading…
Ah thanks, I wasn't aware of #3, that should be a huge boon.
Earlier quoted context omitted.
We re-uploaded Gemma4 4 times - 3 times were due to 20 llama.cpp bug fixes, which we helped solve some as well. The 4th is an official Gemma chat template improvement from Google themselves, so these are out of our hands. All providers had to re-fix their uploads, so not just us. For MiniMax 2.7 - there were NaNs, but it wasn't just ours - all quant providers had it - we identified 38% of bartowski's had NaNs. Ours w…
Appreciate the work of your team very much. Though chat templates seem like they need a better solution. So many issues, seems quite fragile.
Earlier quoted context omitted.
We re-uploaded Gemma4 4 times - 3 times were due to 20 llama.cpp bug fixes, which we helped solve some as well. The 4th is an official Gemma chat template improvement from Google themselves, so these are out of our hands. All providers had to re-fix their uploads, so not just us. For MiniMax 2.7 - there were NaNs, but it wasn't just ours - all quant providers had it - we identified 38% of bartowski's had NaNs. Ours w…
Just curious, the fixes are not about weights but about templates, am I right?
Earlier quoted context omitted.
I sense that I don't really understand enough of your comment to know why this is important. I hope you can explain some things to me: - Why is Qwen's default "quantization" setup "bad" - Who is Unsloth? - Why is his format better? What gains does a better format give? What are the downsides of a bad format? - What is quantization? Granted, I can look up this myself, but I thought I'd ask for the full picture for oth…
The default Qwen "quantization" is not "bad", it's "large". Unsloth releases lower-quality versions of the model (Qwen in this case). Think about taking a 95% quality JPEG and converting it to a 40% quality JPEG. Models are quantized to lower quality/size so they can run on cheaper/consumer GPUs.
Earlier quoted context omitted.
Some tasks don’t require SOTA models. For translating small texts I use Gemma 4 on my iPhone because it’s faster and better than Apple Translate or Google Translate and works offline. Also if you can break down certain tasks like JSON healing into small focused coding tasks then local models are useful
How does that work? Wouldn't it be slow loading the weights into memory every time you launch it?
Earlier quoted context omitted.
Yes we collab with them!
Sorry this is a bit of a tangent, but I noticed you also released UD quants of ERNIE-Image the same day it released, which as I understand requires generating a bunch of images. I've been working to do something similar with my CLI program ggufy, and was curious of you had any info you could share on the kind of compute you put into that, and if you generate full images or look at latents?
Earlier quoted context omitted.
> that these really are ultimately just stochastic sentence machines I thought that's exactly what they are?
I am so perplexed what exactly where people thinking they were. Its nothing else than highly sofisticated statistics.