They seem to have taken down the link and I can't find a new one.
Yup. Not just you.
Eagle 3.1: Collaboration Between the EAGLE Team, vLLM Team, and TorchSpec Team
21–25 of 25 posts
Re: Eagle 3.1: Collaboration Between the EAGLE Team, vLLM Team, and TorchSpec Team
#22Earlier quoted context omitted.
Speculative decoding shouldn't actually change the accuracy of the response. The draft model drafts a couple tokens, and the inference framework verifies that the larger model would have picked them. However, I've found that speculative decoders don't help much if you're running a model locally on limited hardware (for instance, my 32GB VRAM M1 Max from 2021). For one, you have to fit both the large and the small dra…
Your experience might be a bit dated, depending on when was the last time you tried it. MTP (which is a flavor of spec decoding) is showing really solid improvements on local models, even on consumer hardware. In fact, as the article mentions, you get the biggest gains at low concurrency (so local should apply), with diminishing returns for higher concurrency (if you think in terms of unit of compute, it's probably b…
Thanks, appreciate the info. For whatever it’s worth regarding recency, I’m testing the main llama-cpp branch that was pulled and built on 2026-05-25 running unsloth/Qwen3.6-35B-A3B-MTP-GGUF:Q4_K_M, my hardware platform is M1 Max 32GB VRAM. Is there a different fork or quant I should be using?
Re: Eagle 3.1: Collaboration Between the EAGLE Team, vLLM Team, and TorchSpec Team
#23I saw EAGLE and thought it's going to be about PCB design. Was left disappointed.
GEDA and then KiCad ate it alive.
Re: Eagle 3.1: Collaboration Between the EAGLE Team, vLLM Team, and TorchSpec Team
#24The EAGLE team traced this fragility to a phenomenon we call ‘attention drift’ Ok that’s downright fascinating. I am one of the world’s foremost experts on the AI psychosis sufferers posting grand theories on Reddit, and ‘drift’ is one of the words that chatbots come back to again and again when told to ponder their own Being (so much so that it even shows up in clearly-unrelated/incorrect contexts — pretty sure I’ve…
When one is talking about two things gradually diverging, isn't "drift" a natural, descriptive verb to reach for? I've heard it often when discussing an implementation diverging from a specification, for example.
Oh of course -- the unusual part is that they apply it to basically everything. Gravity is caused by inertia drift, consciousness is coherence drift, the economy is governed by power drift, etc.
Re: Eagle 3.1: Collaboration Between the EAGLE Team, vLLM Team, and TorchSpec Team
#25Earlier quoted context omitted.
Speculative decoding shouldn't actually change the accuracy of the response. The draft model drafts a couple tokens, and the inference framework verifies that the larger model would have picked them. However, I've found that speculative decoders don't help much if you're running a model locally on limited hardware (for instance, my 32GB VRAM M1 Max from 2021). For one, you have to fit both the large and the small dra…
Your experience might be a bit dated, depending on when was the last time you tried it. MTP (which is a flavor of spec decoding) is showing really solid improvements on local models, even on consumer hardware. In fact, as the article mentions, you get the biggest gains at low concurrency (so local should apply), with diminishing returns for higher concurrency (if you think in terms of unit of compute, it's probably b…
Agree. At high concurrency, you are better off spending the compute budget on parallel requests rather than draft prediction. The challenging part is that most deployment don't have static traffic profiles. A configuration that was right at launch may no longer be correct months later, and there is no signal that tells you when you have crossed the threshold.