641–650 of 848 posts
As usual, the Jinja templates are messed up so use this [0] to reduce or turn off thinking, fix tool calling, keep a 100% KV cache hit rate, etc. [0] https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Earlier quoted context omitted.
what token/s?
27 t/s. I suspect there will be significant speed ups in the coming weeks.
Earlier quoted context omitted.
"inclined to overthinking," holy cats you're not kidding! On a Mac mini M4 Pro 64GB I prompted it with "svg owl" and it thought for 17m12s, outputting 36.3KiB of thinking chatter. It did end up producing a 20.2KiB HTML+JS+SVG file with a very nice owl, including cursor-tracking animation, but it ran for more than a half hour! The MoE models are stupendously faster.
what harness did you use?
For comparison's sake, Qwen 3.6 35B A3B MLX, same prompt, same machine, thought for 1.59 seconds, used 2,398 tokens, and hit 80.83 tok/s. It actually output just a straight SVG file of an owl (not a bad owl either!), whereas Qwen 3.8 27B spent a zillion times longer gold-plating everything and output HTML+JS+SVG with interactive animation, sound, etc., etc.
Credit where it's due. Qwen 3.8 27B is only the second local model after Gemma 4 that managed to correctly reason through one of my private benchmarks. It took 5x as many tokens to do it and 12m30s with MTP enabled, but it did do it. Gemma 4 reasoned through it more implicitly, while Qwen 3.8 reasoned more explicitly. Laguna and Muse Glimmer failed hard on it, though they're useful for other tasks. The VRAM usage see…
Earlier quoted context omitted.
> The VRAM usage seems way less efficient than Gemma 4 or Glimmer though Maybe it's implicit that you're using llama.cpp (although you don't mention GGUF), but it's hard to reach concrete conclusions about the model architecture based on one implementation in one runtime.
Aren't things like KV size inherent to the model?
Llama.cpp checkpoints and moves snapshots of the cache to main RAM for faster resumption after truncation.
Earlier quoted context omitted.
IDK, 30GB is a lot of data when we're talking about text! Moby Dick, uncompressed, is ~1MB. Compressed, it's around 500KB. I feel fairly certain that one could fit all of the textual knowledge required to cultivate a world-class in <60,000 Moby Dicks. (Arguably in <5,000 Moby Dicks with intense effort/pruning).
I think a specialized model could squeeze all you need to know about a certain profession in 30GB. But not all professions at once, which is what these models try to do. Or maybe not, maybe there's a world model needed for human level at any profession that is very hard to quantify and requires more than 30GB by itself.
I just think about all the content I’ve consumed in my life to become a professional software developer and I would be very surprised if it couldn’t be adequately represented by The “foundational overlap” of K-12 education is identical for most professions with 2-8 years of “finishing” on top.
My mental model is that the budget is spread across maybe 20% genetics (most of our instinctive/genetic information is surely pretty useless), 50% k-12 education, 30% for professionally-specific knowledge.
Earlier quoted context omitted.
What kind of things you only get with a larger model?
Asking it factual information[1]. You just can't compress the entire human knowledge into a 30GB file. [1] Without searching the internet. And even if you allow it, you'll get much worse results because search means browsing and parsing the top results, and search results are horrible, whereas internal knowledge from training encompasses the entire internet plus all books including very niche stuff.
Fortunately, that isn’t necessary! What LLMs need is a level of fluency with key concepts so that they can (1) make effective use of retrieval tools and (2) understand the material in the context window. 30GB-sized models can absolutely store enough knowledge to do this.
Here is an example from the field of law. Most lawyers who have litigated contract disputes in California know about Civil Code § 1717, which makes any contract providing for attorney fees to a prevailing party mutual, so that even if the contract was written to be one-sided, it won’t be enforced that way. It’s a simple enough concept, but there are many more particulars to it, such as what happens when the fee provision is only written to apply to part of the contract. (Answer: it depends on other facts.)
When a lawyer recognizes that they’re in a situation where § 1717 is relevant, the first thing they will do is pull the statute and read it, because nobody has it memorized. And they don’t need to.