Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Granite 4.1: IBM's 8B Model Matching 32B MoE

firethering.com

Post reply on HN
Print view View original

161–170 of 223 posts

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#161

Post by 59nadir »

celrod wrote: post #143
59nadir wrote: post #87

Earlier quoted context omitted.

Counter-point: I built an agent that can only interface with Kakoune, a much less common and more challenging situation for an LLM to find itself in, and Gemma4-A4B 8bit quantized does remarkably better in actually figuring out how to get text in buffers than Qwen3.6-35B-A3B in a similar class as Gemma4 A4B. Now, is this the usual use case? No, it's a benchmark I created specifically in order to put LLMs in situation…

Fellow kakoune user here. I'm curious about your use case/ what you're doing with it!

I'm just messing around with building agents, that's all. I'm not super interested in making ones that just sit in a terminal executing shell scripts because truth be told they're absolutely trivial to make and don't show any interesting parts of LLMs, whereas telling an agent that they are sitting in Kakoune is a whole lot more interesting and really shows a lot of what LLMs aren't great at, and how they'll have to fight their urge to spit out overwrought bash invocations or at the very least find a way to fit those into something new.

So far the only tools the agent has access to are `evaluate_commands(commands=["...", "..."])` and `get_buffer_contents()`, which really makes them have to work for doing things. I could make it super easy for them but then it wouldn't be an interesting experiment.

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#162

Post by seemaze »

zkmon wrote: post #95
vessenes wrote: post #32

Earlier quoted context omitted.

Have you tried the Gemma 4 series, out of curiosity? I haven’t run a local model in a while, but the benchmarks look good. I’d take a free local tool-use model if it was relatively consistent.

I have tested Gemma4-26B against Qwen3.6-35B. Gemma beats Qwen on structured data extraction and instruction following. Gemma is far more precise than Qwen in these tasks, while Qwen gets a bit more creative, verbose, and imprecise. However Qwen has far more general smartness, high token throughput. Qwen could precisely pinpoint the issues in data quality and code, while Gemma had no clue. On the coding skills, Qwen…

That's interesting. I've been using Qwen3.5-35B for (poorly) structured table extraction based largely on the reports that Qwen had a much better vision implementation.

I have not benchmarked Qwen3.5 vs. Qwen3.6 for the same task, nor trialed Gemma4-26B. Guess it's time for some testing!

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#163

Post by tredre3 »

simonw wrote: post #134

The Granite 4.1 3B model is only 2GB from Unsloth: https://huggingface.co/unsloth/granite-4.1-3b-GGUF I ran it in LM Studio and got a pleasingly abstract pelican on a bicycle (genuinely not bad for a tiny 3B model - it can at least output valid SVG): https://gist.github.com/simonw/5f2df6093885a04c9573cf5756d34...

Do you have any reasons to believe that granite is more immune to the effects of quantization than other tiny models? Otherwise it seems odd to judge a tiny model true capabilities by using its 4bit quant.

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#164

Post by simonw »

tredre3 wrote: post #163
simonw wrote: post #134

The Granite 4.1 3B model is only 2GB from Unsloth: https://huggingface.co/unsloth/granite-4.1-3b-GGUF I ran it in LM Studio and got a pleasingly abstract pelican on a bicycle (genuinely not bad for a tiny 3B model - it can at least output valid SVG): https://gist.github.com/simonw/5f2df6093885a04c9573cf5756d34...

Do you have any reasons to believe that granite is more immune to the effects of quantization than other tiny models? Otherwise it seems odd to judge a tiny model true capabilities by using its 4bit quant.

This model is small enough that it might be sensible to try the same prompts against all of the quant sizes to try and spot any differences.

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#165

Post by lambda »

xrd wrote: post #138
lambda wrote: post #76

Earlier quoted context omitted.

Gemma 4 31b was working ok for me; but it was consuming tons of memory on SWA checkpoints, I had to turn them way down, and as a 31b dense model is fairly slow on a Strix Halo. I did have a lot of tool calling issues on 26b-a4b, though. The Qwen models are quite solid though.

What are you using to run it vllm, llama.cpp or other? Can you share your switches and approach for using tools?

llama.cpp

My setup is a bit of a mess as I experiment with different ways of configuring and hosting local models. So at some point I was experimenting with the router server but stopped doing that, but some of my settings are still in models.ini while some are on the command line.

podman run --env "HF_TOKEN=$HF_TOKEN" --env "LLAMA_SERVER_SLOTS_DEBUG=1" -p 8080:8080 --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --security-opt label=disable --rm -it -v ~/.cache/huggingface/:/root/.cache/huggingface/ -v ./unsloth:/app/unsloth -v ./models.ini:/app/models.ini llama.cpp-rocm7.2 -hf unsloth/gemma-4-31B-it-GGUF:UD-Q8_K_XL --chat-template-file /root/.cache/huggingface/gemma-4-31B-it-chat_template.jinja -ctxcp 8 --port 8080 --host 0.0.0.0 -dio --models-preset models.ini

With the following as the relevant settings in models.ini (I actually have no idea if these settings are applied when not using the router server, it's been hard for me to figure out what settings are actually applied when using bot the command line and models.ini

  [*]
  jinja = true
  seed = 3407
  flash-attn = on

  [unsloth/gemma-4-31B-it-GGUF:UD-Q8_K_XL]
  temperature = 1.0
  top_p = 0.95
  top_k = 64
And it looks like the chat_template.jinja I have is actually out of date by now, there was a new one pushed just a couple of days ago that seems to have some further tool calling fixes: https://huggingface.co/google/gemma-4-31B-it/blob/main/chat_...

As my harness, I'm using pi, with a pretty vanilla config.

Anyhow, Gemms 4 31b worked in this config, but it was slow and RAM hungry. Since then, I've mostly moved to Qwen 3.6 35b-a3b because it's a lot faster.

I'm not actually doing anything useful with these yet, but I've used them for some experiments and Qwen 3.6 35b-a3b was capable of doing some pretty long mostly unsupervised agentic loops in my experimentation.

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#166

Post by UncleOxidant »

2ndorderthought wrote: post #4

I test drove it yesterday. It's pretty impressive at 8b. Runs on commodity hardware quickly. Qwen3.6 35b a3b is still my local champion but I may use this for auto complete and small tasks. Granite has recent training data which is nice. If the other small models got fine tuned on recent data I don't know if I would use this at all, but that alone makes it pretty decent. The 4b they released was not good for my needs…

> I may use this for auto complete

Using an 8B LLM for auto complete seems kind of like overkill. Couldn't a much smaller model handle that? IIRC there's a Qwen 1B model.

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#167

Post by albedoa »

rnadomvirlabe wrote: post #20
steveharing1 wrote: post #16

Earlier quoted context omitted.

Because Qwen 3.6 pushes way above its weight. Granite 8B is impressive, but Qwen still wins on raw capability, especially for coding.

You just asserted the same thing again. Why do you say this is the case?

Maybe you could tell him what you want instead of making him guess.

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#168

Post by UncleOxidant »

cyanydeez wrote: post #60
2ndorderthought wrote: post #4

I test drove it yesterday. It's pretty impressive at 8b. Runs on commodity hardware quickly. Qwen3.6 35b a3b is still my local champion but I may use this for auto complete and small tasks. Granite has recent training data which is nice. If the other small models got fine tuned on recent data I don't know if I would use this at all, but that alone makes it pretty decent. The 4b they released was not good for my needs…

Qwen3-Coder-Next seems to be perfect sized for coding. I tried the new and just found the verbosity not really useful for coding. But probably for more analytical tasks or writing docs.

Qwen3-coder-next is still my favorite local model. Qwen3.6-27b is probably a bit better, but it also runs much slower on my Strix Halo box. Hoping we see a Qwen3.6-coder soon!

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#169

Post by joquarky »

kevin42 wrote: post #114
Forgeties79 wrote: post #113

Earlier quoted context omitted.

If you can’t distinguish between fake images and real ones why should you care?

That depends on the purpose of the image. If it's used to create a false narrative (like a deep fake), sure, you should care. But if it's used as an alternative to a stock photo, or as an easy way to make an infographic then no, I don't think you should care.

> you should care

Why should I care? The world is full of false narratives.

How can I have the bandwidth to care about everything all of the time?

I swear that more than half of the complaining that I find here comes from priveledged people bike shedding over inane topics, and who have never had to really worry about serious survival-level (how am I going to eat today?) issues in their lives.

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#170

Post by simonw »

simonw wrote: post #164
tredre3 wrote: post #163

Earlier quoted context omitted.

Do you have any reasons to believe that granite is more immune to the effects of quantization than other tiny models? Otherwise it seems odd to judge a tiny model true capabilities by using its 4bit quant.

This model is small enough that it might be sensible to try the same prompts against all of the quant sizes to try and spot any differences.

This inspired me to give that a go: https://simonw.github.io/granite-4.1-3b-gguf-pelicans/
Post reply on HN