DSpark: Speculative decoding accelerates LLM inference [pdf]

Post by threethirtytwo » Sat, Jun 27, 2026, 5:46 PM UTC

spongebobstoes wrote: post #200

broodbucket wrote: post #48

Earlier quoted context omitted.

Look at how far OpenAI has drifted from their original mission. Everything comes back to greed, so it's ideal for the world if selfish motives happen to coincide with what's good for the world, like advancements in open models

can you elaborate? the original mission was "advance digital intelligence in a way that benefits all of humanity" I don't see an inconsistency. money is pragmatic, the mission needs money

Post by noIdeaTheSecond » Sat, Jun 27, 2026, 5:55 PM UTC

wg0 wrote: post #224

That's why I pay them. Regularly. Without fail. Despite my token usage isn't that much. But I vote for these heroes with my wallet. Just yesterday did again.

Post by SubiculumCode » Sat, Jun 27, 2026, 5:59 PM UTC

kamranjon wrote: post #12

DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.

Post by spijdar » Sat, Jun 27, 2026, 6:14 PM UTC

DiabloD3 wrote: post #252

sigmar wrote: post #241

Earlier quoted context omitted.

>publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Google is still releasing a lot of llm architecture research. They introduced speculative decoding of LLMs in 2022[1], then released the code to perform sceculative decoding for their Gemma 4 model this year[2] [1] https://arxiv.org/abs/2211.17192 [2] https://github.com/google-gemma/cook…

They weren't the first to do MTP like this, and arguably did it wrong: the MTP heads are kept in a separate file and have to be welded in by the inference engine. Qwen 3.6 shipped with working MTP first, and had working MTP in llama.cpp first.

Post by anaisbetts » Sat, Jun 27, 2026, 6:23 PM UTC

spijdar wrote: post #267

DiabloD3 wrote: post #252

Earlier quoted context omitted.

They weren't the first to do MTP like this, and arguably did it wrong: the MTP heads are kept in a separate file and have to be welded in by the inference engine. Qwen 3.6 shipped with working MTP first, and had working MTP in llama.cpp first.

Given the MTP drafter is basically a separate model, keeping it separate makes more sense IMO. It's out of my wheelhouse but it seems like you could adjust the MTP drafter model separately from the main model, too. Ultimately though the real explanation, I think, is Google doesn't care since for their own purposes (in LiteRT-LM), they do bundle them. As far as I know, anyway.