Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Granite 4.1: IBM's 8B Model Matching 32B MoE

firethering.com

Post reply on HN
Print view View original

201–210 of 223 posts

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#201

Post by agunapal »

aitchnyu wrote: post #108
zozbot234 wrote: post #25

Earlier quoted context omitted.

MoE models will have far more world knowledge than dense models with the same amount of active parameters. MoE is a no-brainer if your inference setup is ultimately limited by compute or memory throughput - not total memory footprint - or alternately if it has fast, high-bandwidth access to lower-tier storage to fetch cold model weights from on demand.

Tangential. I'm a newb, can you name the concept of partitioning weights so we dont need to load whole thing?

Do you mean model sharding?

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#202

Post by agunapal »

vessenes wrote: post #28
agunapal wrote: post #18

If you really think about why MoE came into existence, its to save significant cost during training, I don't think there was any concrete evidence of performance gains for comparable MoE vs dense models. Over the years, I believe all the new techniques being employed in post training have made the models better.

I think you mean inference compute? I believe all expert weights are updated in each backward pass during MoE training. The first benefit was getting a sort of structured pruning of weights through the mechanism of expert selection so that the model didn’t need to go through ‘unnecessary’ parts of the model for a given token. This then let inference use memory more efficiently in memory constrained environments, wher…

Here is a paper from few years ago where they talk about 7x speed increase, which equates to savings.

https://arxiv.org/abs/2101.03961

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#203

Post by mdp2021 »

ibgeek wrote: post #56
mdp2021 wrote: post #2

Wish they also released an embedding model, in the line of their previous: compact (while good)...

They did: https://huggingface.co/collections/ibm-granite/granite-embed... 311M and 97M versions.

Thank you! Yes, after yours I found out that they have produced many NNs in their "Granite 4.1" category:

Granite Vision 4.1; Granite Speech 4.1; Granite Guardian 4.1; Granite Embedding Multilingual R2 - with, of course, the "Small Language Models"

https://research.ibm.com/blog/granite-4-1-ai-foundation-mode...

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#204

Post by mdp2021 »

cubefox wrote: post #91

It's strange that they don't include reasoning training (RLVR). Their justification doesn't sound convincing: > While reasoning models have grown in popularity in recent years, their abilities aren’t always the most efficient way to get a result. In enterprise settings, token costs and speed are often as important as performance. That is why turning to less expensive, non-reasoning models with similar benchmark perfo…

I may have misunderstood: is not reasoning training (RLVR) independent from the use of the "" tags - is it not a method that improves results in reasoning? How do we know that it was not carried out?

Incidentally: I am trying to spend some time researching in the progresses in the area (the jump from parroting, to inconsistent apparent reasoning, to reliable reasoning).

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#206

Post by BoredomIsFun »

v3ss0n wrote: post #51
vessenes wrote: post #32

Earlier quoted context omitted.

Have you tried the Gemma 4 series, out of curiosity? I haven’t run a local model in a while, but the benchmarks look good. I’d take a free local tool-use model if it was relatively consistent.

Qwen 3.6 burns it to the ground. it was not even a challenge. Gemma4 seriously fails at toolcalls and agentic works. It got all messed up after 2-3 turns of Vibecoding.

> Qwen 3.6 burns it to the ground.

Not for creative writing or NLP.

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#207

Post by Tuna-Fish »

zozbot234 wrote: post #192
Tuna-Fish wrote: post #188

Earlier quoted context omitted.

Time for my daily "HBF is coming" comment. The next step for models is to put the weights on flash, connected with a very wide interface to the accelerator. The first users will be datacenters, but it should trickle down to consumer hardware eventually. A single 512GB stack is expected to cost about $200, and provide 1.6TB/s of reads. You still need some fast DRAM for the KV cache and for activations, but weights sho…

Reading from Flash is too power-intensive compared to DRAM, this is why Flash offload isn't used in the data center today. Flash is also prone to wearing out quickly so ephemeral data like the KV-cache can't really be stashed in there. Unless your model has an unprecedented level of sparsity I just don't see how HBF could ever be useful.

Currently available flash is obviously unusable. HBF is not that.

The reason HBF is (about to be) a thing is that flash manufacturers realized that if you heavily optimize flash for read throughput and energy, as opposed to density, you can match DRAM on throughput and get to within 2x on energy, at the cost of half your density. That would make the density still ~50 times better than DRAM, built on a cheap mass-produced process. All manufacturers are chasing this hard right now, with first samples to arrive later this year.

You are correct that it would absolutely not be used for any mutable data, only weights in inference. This is both because there is insufficient endurance (expected to be ~hundreds of drive writes total), but also because it will be very slow to write compared to the read speed. A single HBF stack is expected to provide 1.6TB/s reads, and single-digit GB/s writes. That's why I wrote the last sentence of my post that you replied to.

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#208

Post by Tuna-Fish »

nickpsecurity wrote: post #191
Tuna-Fish wrote: post #188

Earlier quoted context omitted.

Time for my daily "HBF is coming" comment. The next step for models is to put the weights on flash, connected with a very wide interface to the accelerator. The first users will be datacenters, but it should trickle down to consumer hardware eventually. A single 512GB stack is expected to cost about $200, and provide 1.6TB/s of reads. You still need some fast DRAM for the KV cache and for activations, but weights sho…

You're thinking in a provably-useful direction: https://arxiv.org/pdf/2312.11514

HBF is not that. The paper you linked is about how to use flash memory that exists to boost LLM performance, with all kinds of optimization tricks. HBF is about making flash memory that doesn't require any of those tricks, and just has the read throughput that's needed for inference.

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#209

Post by lostmsu »

woadwarrior01 wrote: post #105

The most salient thing about these models is that they're non-reasoning models. This makes then very token efficient and particularly well suited for local inference where decoding is usually slower than with datacenter GPUs. Link to HF collection: https://huggingface.co/collections/ibm-granite/granite-41-la...

Probably worse than Gemma 4 or Qwen 3.6 with thinking off.

Re: Granite 4.1: IBM's 8B Model Matching 32B MoE

#210

Post by jasonlotito »

speedgoose wrote: post #198
jeraldbenny wrote: post #195

Earlier quoted context omitted.

if you are googling you can find so many open source dataset. Also use kaggle, they're also having training datasets which we can use.

I know, but that is not my point.

You never had one. You tried to be clever and failed.
Post reply on HN