Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DSpark: Speculative decoding accelerates LLM inference [pdf]

github.com

Post reply on HN
Print view View original

251–260 of 393 posts

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#251

Post by notnullorvoid »

utopiah wrote: post #120
lelanthran wrote: post #68

These companies providing tokens, whether SOTA or not, that want to IPO are so fucked as time goes on. Can't sell their SOTA models, only slightly better than the open source models for the models they can sell, cost 20x to 50x for good models, a TAM that consists almost solely of developers, with no customer of theirs actually boasting increased profits as a result of AI... I fear their time to IPO may have passed.

The question is even, was there EVER a time for an IPO? If the business model requires hundreds of billions to get the required quality (R&D but also infrastructure to collect data and train, either purchased or rented to 3rd party) while "only" dozens of billions can be earned back (as costs still exist to earn, it's not free once models are trained), then maybe there NEVER was nor till be a good time for an IPO in…

> in a rational market.

Unfortunately the market is often not rational in this way.

Hype within retail market means there are suckers willing to buy. Institutional market knows there are suckers when the hype is high. Both would drive the price up, and retail investors the ones left when it falls.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#252

Post by DiabloD3 »

sigmar wrote: post #241
kamranjon wrote: post #12

DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.

>publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Google is still releasing a lot of llm architecture research. They introduced speculative decoding of LLMs in 2022[1], then released the code to perform sceculative decoding for their Gemma 4 model this year[2] [1] https://arxiv.org/abs/2211.17192 [2] https://github.com/google-gemma/cook…

They weren't the first to do MTP like this, and arguably did it wrong: the MTP heads are kept in a separate file and have to be welded in by the inference engine.

Qwen 3.6 shipped with working MTP first, and had working MTP in llama.cpp first.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#253

Post by eagleal »

rsanek wrote: post #94
archerx wrote: post #87

Earlier quoted context omitted.

They are self financed, the company that makes DeepSeek is a finance company that trades on the markets.

The CCP's approach has historically been to subsidize their companies far more than other countries do. Why would LLMs be any different? https://www.oecd.org/en/data/dashboards/magic-database-indus...

Even the latest World Bank report, the defacto neoliberal institution, recognized a couple of months ago that leaving the industries focus be dictaed by purely capital decisions was bad, as in _really_ bad.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#256

Post by otterley »

jimmydoe wrote: post #188
StizzurpXDD wrote: post #146

DeepSeek is, as I feel currently, the sole AI company which is actually trying to innovate rather than top mere benchmarks. Others like OpenAI, Anthropic and Google are mostly just competeing with each rather than keep innovating around the clock.

Besides the founder, the only real external investor for DeepSeek is Chinese govt. there are literally zero revenue pressure compare to O, A & G. To compete in that direction, USG needs to learn from CCP to "seize the means of production", which they are sort of doing, but in such an incompetent way that I'm afraid we will probably end up mixing the worst of both communism and capitalism.

> To compete in that direction, USG needs to learn from CCP to "seize the means of production"

No they don't. The U.S. Government is free to launch their own AI labs if they wish -- and even compete with the private sector -- but that doesn't mean they have to confiscate existing investments and capital. But Congress is unlikely to do that, because we've learned in the course of history that in well-functioning competitive markets, publicly-operated services tend to be worse than private ones across multiple dimensions.

Chinese companies are largely where they are not because they're state funded, but because they operate in ways that would be considered criminal in the U.S. If they didn't constantly trespass on OpenAI and Anthropic to try to achieve product and technological parity, they would be too far behind to produce innovative research.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#258

Post by threethirtytwo »

docfort wrote: post #249
threethirtytwo wrote: post #218

Earlier quoted context omitted.

This statement is factually true and you are voted down because many people lack knowledge. Any individual that provides free labor cannot survive off of said free labor. He must work for money to survive or get donations from someone who earned that money from incentive based labor in order to even buy the food he needs to exist as a living human being. Much of the time that labor is actually closed source. This is…

Free labor enables capitalism, especially if you consider labor arbitrage as a mixture of free labor and properly compensated (according to the real value) labor. From literally being born, to family culture, education, and whatever level of broad social cohesion, it’s all free labor. Without that background, money itself loses its value, since an individual cannot have reasonable confidence in trading it for somethi…

Free labor is derivative to incentivized labor. Your statement here doesn’t disprove or counter what I said. Again, follow the money trail. Everything you said if you follow the origin of the money it comes from paid, incentivized labor. Parents need money to raise kids… where do they get that money?? Our economy is called capitalism for a reason there is literally zero reference to charity or altruism in the vocabulary or even standard models that describe our economy and economic theory.

Put it another away: if we removed your ability to do incentivized labor and all you can do is charity work… you would run out of money and die from starvation. If we did the opposite and we removed your ability to do charity work… you’d be fine.

All of this re-emphasizes the point of this thread: In our objective reality, the world is driven by incentive based work while altruism is a side effect of surplus wealth generated by incentive based work. That is the fundamental reality.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#259

Post by darkoob12 »

kamranjon wrote: post #226
darkoob12 wrote: post #109

Earlier quoted context omitted.

Google and Microsoft publish more than enough and American universities are publishing the science beyond DeepSeek's engineering. That fact that you don't know about them means you're not following the science only reading hacker news.

Google hasn’t published much in depth ML work since T5 (which was hugely influential at the time) - most Gemma releases are 1-3 page model card pdfs these days with no in depth analysis. Even TurboQuant is shaking out to have basically been a rehash of previous work without proper attribution. I do think Microsoft is doing some interesting things with smaller models but haven’t read much research, interested in any r…

Check recent iclr acl icml neurips you will see 10-20 papers from Google Research which are not just simple model cards. they are solid reproducible research.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#260

Post by tiahura »

xnx wrote: post #221

Is this newer/better than the speculative decoding from 2022? https://arxiv.org/abs/2211.17192

Seems like they focus on improving the drafter and the verification policy so speculation keeps producing net speedups rather than wasted verification work at deepseek scale.
Post reply on HN