281–290 of 507 posts
I ran these in LM Studio and got unrecognizable pelicans out of the 2B and 4B models and an outstanding pelican out of the 26b-a4b model - I think the best I've seen from a model that runs on my laptop. https://simonwillison.net/2026/Apr/2/gemma-4/ The gemma-4-31b model is completely broken for me - it just spits out "---\n" no matter what prompt I feed it. I got a pelican out of it via the AI Studio API hosted model…
Earlier quoted context omitted.
Disclaimer: I'm an AI novice relative to many here. FWIW last wknd I spent a couple hours setting up self-hosted n8n with ollama and gemma3:4b [EDIT: not Qwen-3.5], using PDF content extraction for my PoC. 100% local workflow, no runtime dependency on cloud providers. I doubt it'd scale very well (macbook air m4, measly 16GB RAM), but it works as intended.
How do you extract the content? OCR? Pdf to text then feed into qwen? I tried something similar where I needed a bunch of tables extracted from the pdf over like 40 pages. It was crazy slow on my MacBook and innacurate
2. The n8n workflow passes a given binary pdf to gemma, which (based on a detailed prompt) analyzes it and produces JSON output.
See https://github.com/LinkedInLearning/build-with-ai-running-lo... if you want more details. :)
Earlier quoted context omitted.
We are always figuring out what parameter size makes sense. The decision is always a mix between how good we can make the models from a technical aspect, with how good they need to be to make all of you super excited to use them. And its a bit of a challenge what is an ever changing ecosystem. I'm personally curious is there a certain parameter size you're looking for?
For the many DGX Spark and Strix Halo users with 128GB of memory, I believe the ideal model size would probably be a MoE with close to 200B total parameters and a low active count of 3B to 10B. I would personally love to see a super sparse 200B A3B model, just to see what is possible. These machines don't have a lot of bandwidth, so a low active count is essential to getting good speed, and a high total parameter cou…
That seems pointless. You can achieve that with a single 24G graphics card already.
I wonder if it would even hold up at that level, as 3B active is really not a lot to work with. Qwen 3.5 uses 122B-A10B and still is neck and neck with the 27B dense model.
I don't see any value proposition for these little boxes like DGX Spark and Strix Halo. Lots of too-slow RAM to do anything useful except run mergekit. imo you'd have been better building a desktop computer with two 3090s.
Earlier quoted context omitted.
For the many DGX Spark and Strix Halo users with 128GB of memory, I believe the ideal model size would probably be a MoE with close to 200B total parameters and a low active count of 3B to 10B. I would personally love to see a super sparse 200B A3B model, just to see what is possible. These machines don't have a lot of bandwidth, so a low active count is essential to getting good speed, and a high total parameter cou…
Following the current rule of thumb MoE = `sqrt(param*active)` a 200B-A3B would have the intelligence of a ~24B dense model. That seems pointless. You can achieve that with a single 24G graphics card already. I wonder if it would even hold up at that level, as 3B active is really not a lot to work with. Qwen 3.5 uses 122B-A10B and still is neck and neck with the 27B dense model. I don't see any value proposition for…
For the sake of argument, even if we take that old rule of thumb at face value, you can see how the MoE still wins:
- (DGX Spark) 273GB/s of memory bandwidth with 3B active parameters at Q4 = 273 / 1.5 = 182 tokens per second as the theoretical maximum.
- (RTX 3090) 936GB/s with 24B parameters at Q4 = 936 / 12 = 78 tokens per second. Or 39 tokens per second if you wanted to run at Q8 to maximize the memory usage on the 24GB card.
The "slow" DGX Spark is now more than twice as fast as the RTX 3090, thanks to an appropriate MoE architecture. Even with two RTX 3090s, you would still be slower. All else being equal, I would take 182 tokens per second over 78 any day of the week. Yes, an RTX 5090 would close that gap significantly, but you mentioned RTX 3090s, and I also have an RTX 3090-based AI desktop.
(The above calculation is dramatically oversimplified, but the end result holds, even if the absolute numbers would probably be less for both scenarios. Token generation is fundamentally bandwidth limited with current autoregressive models. Diffusion LLMs could change that.)
The mid-size frontier models are rumored to be extremely sparse like that, but 10x larger on both total and active. No one has ever released an open model that sparse for us to try out.
As I said, I wanted to see what it is possible for Google to achieve.
> Qwen 3.5 uses 122B-A10B and still is neck and neck with the 27B dense model.
From what I've seen, having used both, I would anecdotally report that the 122B model is better in ways that aren't reflected in benchmarks, with more inherent knowledge and more adaptability. But, I agree those two models are quite close, and that's why I want to see greater sparsity and greater total parameters: to push the limits and see what happens, for science.
Here the 26B-A4B variant is head and shoulders above recent open-weight models, at least on my trusty M1 Max 64GB MacBook.
I set up Claude Code to use this variant via llama-server, with 37K tokens initial context, and it performs very well: ~40 tokens/sec, far better than Qwen3.5-35B-A3B, though I don't know yet about the intelligence or tool-calling consistency. Prompt processing speed is comparable to the Qwen variant at ~400 tok/s.
My informal tests, all with roughly 30K-37K tokens initial context:
┌────────────────────┬───────────────┬────────────┐
│ Model │ Active Params │ tg (tok/s) │
├────────────────────┼───────────────┼────────────┤
│ Gemma-4-26B-A4B │ 4B │ ~40 │
├────────────────────┼───────────────┼────────────┤
│ GPT-OSS-20B │ 3.6B │ ~17-38 │
├────────────────────┼───────────────┼────────────┤
│ Qwen3-30B-A3B │ 3B │ ~15-27 │
├────────────────────┼───────────────┼────────────┤
│ GLM-4.7-Flash │ 3B │ ~12-13 │
├────────────────────┼───────────────┼────────────┤
│ Qwen3.5-35B-A3B │ 3B │ ~12 │
├────────────────────┼───────────────┼────────────┤
│ Qwen3-Next-80B-A3B │ 3B │ ~3-5 │
└────────────────────┴───────────────┴────────────┘
https://pchalasani.github.io/claude-code-tools/integrations/...
Earlier quoted context omitted.
How do you extract the content? OCR? Pdf to text then feed into qwen? I tried something similar where I needed a bunch of tables extracted from the pdf over like 40 pages. It was crazy slow on my MacBook and innacurate
If you have a basic ARM MacBook, GLM-OCR is the best single model I have found for OCR with good table extraction/formatting. It's a compact 0.9b parameter model, so it'll run on systems with only 8 GB of RAM. https://github.com/zai-org/GLM-OCR Use mlx-vlm for inference: https://github.com/zai-org/GLM-OCR/blob/main/examples/mlx-de... Then you can run a single command to process your PDF: glmocr parse example.pdf Load…
Hi all! I work on the Gemma team, one of many as this one was a bigger effort given it was a mainline release. Happy to answer whatever questions I can
Do any of you use this as a replacement for Claude Code? For example, you might use it with openclaw. I have a 24 GB integrated RAM Mac Mini M4 I currently run Claude Code on, do you think I can replace it with OpenClaw and one of these models?
Earlier quoted context omitted.
For the many DGX Spark and Strix Halo users with 128GB of memory, I believe the ideal model size would probably be a MoE with close to 200B total parameters and a low active count of 3B to 10B. I would personally love to see a super sparse 200B A3B model, just to see what is possible. These machines don't have a lot of bandwidth, so a low active count is essential to getting good speed, and a high total parameter cou…
Following the current rule of thumb MoE = `sqrt(param*active)` a 200B-A3B would have the intelligence of a ~24B dense model. That seems pointless. You can achieve that with a single 24G graphics card already. I wonder if it would even hold up at that level, as 3B active is really not a lot to work with. Qwen 3.5 uses 122B-A10B and still is neck and neck with the 27B dense model. I don't see any value proposition for…
Earlier quoted context omitted.
If you have a basic ARM MacBook, GLM-OCR is the best single model I have found for OCR with good table extraction/formatting. It's a compact 0.9b parameter model, so it'll run on systems with only 8 GB of RAM. https://github.com/zai-org/GLM-OCR Use mlx-vlm for inference: https://github.com/zai-org/GLM-OCR/blob/main/examples/mlx-de... Then you can run a single command to process your PDF: glmocr parse example.pdf Load…
Thanks! Just tried it on a 40 page pdf. Seems to work for single images but the large pdf gives me connection timeouts
Earlier quoted context omitted.
Nemotron 3 Super was released recently. That's a direct competitor to gpt-oss-120b. https://developer.nvidia.com/blog/introducing-nemotron-3-sup...
In terms of ability, maybe, in terms of speed, it's not even close. Check out the Prompt Processing speeds between them: https://kyuz0.github.io/amd-strix-halo-toolboxes/ gpt-oss-120b is over 600 tokens/s PP for all but one backend. nemotron-3-super is at best 260 tokens/s PP. Comparing token generation, it's again like 50 tokens/sec vs 15 tokens/sec That really bogs down agentic tooling. Something needs to be catego…
gpt-oss-120b: (unknown prefill), ~175 tok/s generation. I don't remember the prefill speed but it certainly was below 10k
Nemotron-3-Super: 14070 tok/s prefill, ~194.5 tok/s generation. (Tested fresh after reload, no caching, I have a screenshot.)
Nemotron-3-Super using NVFP4 and speculative decoding via MTP 5 tokens at a time as mentioned in Nvidia cookbook: https://docs.nvidia.com/nemotron/nightly/usage-cookbook/Nemo...