Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DSpark: Speculative decoding accelerates LLM inference [pdf]

github.com

Post reply on HN
Print view View original

241–250 of 393 posts

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#241

Post by sigmar »

kamranjon wrote: post #12

DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.

>publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately.

Google is still releasing a lot of llm architecture research. They introduced speculative decoding of LLMs in 2022[1], then released the code to perform sceculative decoding for their Gemma 4 model this year[2]

[1] https://arxiv.org/abs/2211.17192

[2] https://github.com/google-gemma/cookbook/blob/main/docs/mtp/...

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#242

Post by intended »

nozzlegear wrote: post #238
intended wrote: post #80

Earlier quoted context omitted.

The world runs on incentives. Altruism/Self-serving are down stream of that. Wikipedia is altruistic, and serves humanity quite well.

I hate to quote pithy proverbs, but "the road to hell is paved with good intentions." One can have an altruistic goal which ends up harming people too, which is where that proverb comes from. Prohibition and The War on Drugs in the US are two good examples of something that had altruistic origins[†] but ended up doing way more harm than good. [†] Another problem with altruism: we don't all agree on whether a goal is…

[dead]

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#243

Post by kamranjon »

sigmar wrote: post #241
kamranjon wrote: post #12

DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.

>publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Google is still releasing a lot of llm architecture research. They introduced speculative decoding of LLMs in 2022[1], then released the code to perform sceculative decoding for their Gemma 4 model this year[2] [1] https://arxiv.org/abs/2211.17192 [2] https://github.com/google-gemma/cook…

Thanks for the clarification - Google does publish more than others - and I actually really appreciate the work they are doing with the Gemma models, which are truly competitive open models. I do wish they’d publish more in depth papers on their Gemma models but appreciate that they are open weights.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#245

Post by jmyeet »

Der_Einzige wrote: post #229
jmyeet wrote: post #112

Earlier quoted context omitted.

I personally think not owning their own compute is going to be an advantage. There is a meteor headed towards all this AI investment that I don't think has been properly accounted for and that is, what happens to all the existing hardware investments when NVidia's next architecture comes out. Blackwell (H100/H200) is the current generation. Rubin (R100, presumably R200) is the next and arrives soon. Now a lot of the…

Why do people who don't follow the prices of A100 talk like they know things about GPU pricing dynamics? A100s are ~7 years old and going for more than 2 dollars an hour, significantly more expensive than even 2 years ago. This is because anything with 80gb of VRAM or more and made by Nvidia will have economically useful lifespans of like, 10 years. I could see H100s getting 12 years. Micheal Berry doesn't know shit…

So I was curious about how A100s would do running DeepSeek v4. I can't find any instances of running v4 Pro on even an 8xA100 cluster. So you need to run Flash at ~284B params. A100s don't support FP8 so you're running FP16 so you're taking a hit that way. But I see estimates of 30-50tok/s for an 8xA100 cluster. They're drawing 300-400W each so you're looking at probably 3500+ Watts, which is roughly 0.01tok/W.

Now jump ahead 2 years and you seem to have a massive jump in performance [1]. The tokens/Watt goes up by at least 2 orders of magnitude. And the B100 is 3-4x that. And we're about to hit the R100 (Rubin) cliff.

That's what this is going to come down. When hyperscalar DCs are getting to Gigawatt power usage, it all comes down to power efficiency. Those A100s aren't far from being sold for scrap.

I've been looking into how different companies are handling depreciation for this. Amazon seems to be saying the life is 3-4 years, Google 4-5 and Meta is saying 8+, which I think is wildly optimistic.

[1]: https://lambda.ai/inference-models/deepseek-ai/deepseek-v4-f...

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#246

Post by dragonwriter »

nozzlegear wrote: post #238
intended wrote: post #80

Earlier quoted context omitted.

The world runs on incentives. Altruism/Self-serving are down stream of that. Wikipedia is altruistic, and serves humanity quite well.

I hate to quote pithy proverbs, but "the road to hell is paved with good intentions." One can have an altruistic goal which ends up harming people too, which is where that proverb comes from. Prohibition and The War on Drugs in the US are two good examples of something that had altruistic origins[†] but ended up doing way more harm than good. [†] Another problem with altruism: we don't all agree on whether a goal is…

> Prohibition and The War on Drugs in the US are two good examples of something that had altruistic origins

The War on Drugs had the purpose (not just in its origin but in its perpetuation) of inflicting harm on elite-disfavored subsets of the population that could not be openly targeted for Constitutional reasons, which is about as far from an altruistic reason as it possible to get.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#247

Post by cloudfudge »

garn810 wrote: post #127
kamranjon wrote: post #12

DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.

Yep. It's about time western world realized Chinese are not the "very bad guys under dictatorship"

I don't think it's very common to believe the Chinese people are bad guys. It's the government and its control of the people that's the problem. And no, I don't think the US is immune to that sort of problem either.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#248

Post by orbital-decay »

DivingForGold wrote: post #93
kamranjon wrote: post #12

DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.

Sure, in part by "stealing" from American AI companies with Distillation attacks: https://yipzap.com/anthropic-accuses-alibaba-of-largest-ai-d...

Besides "attack" being a ludicrous name for distillation, note how your article says "accuses", also it's mostly about Alibaba, not DeepSeek (although it's mentioned there). Both Dario Amodei and Sam Altman publicly claimed that DS used their outputs to train their models, and knowing the differences between all these models by heart, I believe they're simply lying through their teeth to sway the public opinion and/or the policy. These models are absolutely nothing alike, and distillation necessarily makes student's outputs similar to teacher's. This is very visible in Z.ai models (which were trained on Gemini outputs to the point that they repeated Google's conditional prompt injections in the CoT, and later on Claude where it started repeating their CoT as well) and certain Google models which were trained on Claude's outputs in a roundabout way. Distillation always shows up in the result.

And certainly they have no idea whether these outputs (assuming they ever existed and it wasn't made up) were used for training. The article mentions that DS made 150k requests. This isn't much and might have been just an eval or a benchmark to compare their own model against. It's really hard to believe DeepSeek had any Claude outputs anywhere in their training schedule, since it's just too different. Besides training on random vibecode of course, which is mostly written by Claude.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#249

Post by docfort »

threethirtytwo wrote: post #218
intended wrote: post #80

Earlier quoted context omitted.

The world runs on incentives. Altruism/Self-serving are down stream of that. Wikipedia is altruistic, and serves humanity quite well.

This statement is factually true and you are voted down because many people lack knowledge. Any individual that provides free labor cannot survive off of said free labor. He must work for money to survive or get donations from someone who earned that money from incentive based labor in order to even buy the food he needs to exist as a living human being. Much of the time that labor is actually closed source. This is…

Free labor enables capitalism, especially if you consider labor arbitrage as a mixture of free labor and properly compensated (according to the real value) labor. From literally being born, to family culture, education, and whatever level of broad social cohesion, it’s all free labor. Without that background, money itself loses its value, since an individual cannot have reasonable confidence in trading it for something of actual tangible value. It is abstract stored value, banked into society for free. Indeed, in many cases, the free labor is in the rational self interest of a group. But stability and love and peace aren’t monetized to their true value. Otherwise, markets should be much less stable. Bubbles are only notable for the large impact of a small group of bad actors. Overall, it’s pretty amazing what free labor does. Open source is just another instance of this long and critical tradition.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#250

Post by Stagnant »

edg5000 wrote: post #58
spiderfarmer wrote: post #8

Earlier quoted context omitted.

Is there a way to see how many tokes one does with claude code (pro)?

It's in the JSONs in ~/.claude, but last 30 days only I think. You can have the model analyze history. So for correct history you'd need to run history analysis on a cron job or something. Kinda hacky.

The 30 day limit can be overridden by adding "cleanupPeriodDays": 9999 to .claude/settings.json
Post reply on HN