Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

91–100 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#91

Post by nzeid »

zdw wrote: post #9

MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.

A few days ago I switched again from Qwen3.6 to Gemma 4 - for personal use I've experienced better average performance with the 26B version of the latter than the 27B of the former.

For someone who's been running local models for a long while, these are very very exciting times.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#93

Post by CharlesW »

franze wrote: post #84
CharlesW wrote: post #79

Earlier quoted context omitted.

LM Studio (for example) is free, can you pitch me on your USP vs. it?

easiness of install (one download), zero configuration, zero online access by design - there will never we websearch, never any kind of tracking, your prompts stay on your device - you can totally put in user data, confident contracts, ... plus over time the harness - coming version has a hotkey for screen capture, next release will have support for native excel, docx export there is value in being offline by design

LM Studio's tagline is literally "local AI on your computer" and has commensurate benefits, as do similar choices like Unsloth Studio and Ollama's desktop app. The differentiators you have planned sound like they'll help you establish a unique value prop. Good luck!

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#94

Post by Klaus23 »

furyofantares wrote: post #75
pu_pe wrote: post #24

So much faster inference with no quality degradation? All that for just some small memory overhead (drafter models are <1B it seems)?

Is it really no quality degradation? I'm curious where my understanding is wrong, but I didn't think you necessarily got the exact same output with how I understand speculative decoding to be used. I thought that if the small model produces tokens that are "good enough", meaning within the top few tokens the larger model produces, they're accepted. I thought it doesn't necessarily have to produce the exact same token…

It really is. This is because LLMs with a single output/user are strongly bandwidth limited. Although the hardware can generate multiple tokens simultaneously, it is slowed down if the tokens depend on each other, as is the case with regular text generation.

The draft model essentially predicts the next token quickly, enabling you to start generating the subsequent token in parallel. If the guess is right, the second generated token is correct. If it is wrong, the second generated token is also potentially wrong, so it must be generated again using the correct prior token obtained through the big model.

A poor draft model will simply slow down the process without affecting the output.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#96

Post by el_isma »

How is this different from the speculative decoding that we had before?

You could pair a big and small model like qwen 32b with qwen 4b and had that same dynamic of the small model generating tokens and the big one "certifiying" them.

The blog says something about re-using the big model's data?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#97

Post by petu »

furyofantares wrote: post #75
pu_pe wrote: post #24

So much faster inference with no quality degradation? All that for just some small memory overhead (drafter models are <1B it seems)?

Is it really no quality degradation? I'm curious where my understanding is wrong, but I didn't think you necessarily got the exact same output with how I understand speculative decoding to be used. I thought that if the small model produces tokens that are "good enough", meaning within the top few tokens the larger model produces, they're accepted. I thought it doesn't necessarily have to produce the exact same token…

Speculative decoding batches multiple completions on all possible outcomes (0/1/2 draft tokens accepted) and sees if big model deviates at any point -- thus verifying each token. So there's no difference in output.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#99

Post by ch_sm »

regexorcist wrote: post #72

Sounds like a game changer if I see that kind of speed up on my hardware. So far I've prefered Qwen 3.6 because of its better tool handling, even though Gemma 4 is faster, but I saw they've updated the model template and that's supposed to be better now. Looking forward to trying this with llama.cpp.

gemma4 has a specific problem with toolcalls that affects most runtimes. fixes for ollama and vllm are being worked on right now

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#100

Post by lelandbatey »

jeffhuys wrote: post #39
skybrian wrote: post #12

Watching the computer write text sort of reminds me of using a modem to call a BBS in the old days. This seems like going from 300 baud to 1200 - a significant improvement, but still pretty slow, and someday we will wonder how we put up with it.

Check chatjimmy.ai

https://chatjimmy.ai being a demo of the "burn the model to an ASIC" approach being sold by Taalas[0], an approach which they use to run Llama 3.1 8B at ~17000 tokens per second.

[0] - https://taalas.com/products/

Post reply on HN