DSpark: Speculative decoding accelerates LLM inference [pdf] • Hacker News Board

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#231

Post by isoprophlex » Sat, Jun 27, 2026, 2:56 PM UTC

FooBarWidget wrote: post #219

isoprophlex wrote: post #210

Earlier quoted context omitted.

Extremely interesting comment, thank you. Got some links where I can download this source material? I don't read or speak the language, but will try interrogating it with an LLM

The fifth book is on Amazon. https://www.amazon.com/XI-JINPING-GOVERNANCE-CHINA-V/dp/7119... It's already an English translation. For something shorter, you can see Arnaud Bertrand's recent review. https://arnaudbertrand.substack.com/p/the-book-the-west-refu... The review is behind a paywall, but not expensive. If you want to read policy documents directly (primary source), try the State Council / Chinese government…

Thanks again, this is more than enough for a clanker-assisted rabbit hole to disappear into

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#232

Post by yogthos » Sat, Jun 27, 2026, 2:56 PM UTC

jingpostmedia wrote: post #165

c7b wrote: post #144

Earlier quoted context omitted.

The question is also what game they're playing. Deepseek came out of a hedge fund. I think it's no coincidence that their publications tend to have a large impact on AI stock prices. Destroying the growth story of overvalued stocks is an interesting investment strategy. It's not even new. Shortsellers understandably get terrible rep from execs, but their actions are more often in the public interest than you'd think.…

[flagged]

Also, we’re seeing a classic commoditization spiral with open models rapidly closing the gap and driving prices towards the marginal cost of inference. The reality is that models themselves are general commodities and there's just not enough difference between them. A company can get ahead of others by a few months, but then the rest quickly close the gap. It's a really low margin business because there's no way to differentiate yourself.

Chinese companies understand this and they're treating models as shared infrastructure akin to Linux. The money is going to be in customization niches. Companies will charge to tune models for specific use cases and charge support for that. There's also going to be money at the bottom for hardware vendors making chips and memory. But the middle tier of generic LLMs is seeing involution where there's relentless competition driving profits towards the bottom.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#233

Post by lightedman » Sat, Jun 27, 2026, 3:08 PM UTC

Anyone want to bet that much like speculative execution, speculative decoding is going to introduce a whole slew of vulnerabilities in the ways LLMs work?

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#234

Post by nozzlegear » Sat, Jun 27, 2026, 3:08 PM UTC

merelydev wrote: post #156

c7b wrote: post #144

Earlier quoted context omitted.

The question is also what game they're playing. Deepseek came out of a hedge fund. I think it's no coincidence that their publications tend to have a large impact on AI stock prices. Destroying the growth story of overvalued stocks is an interesting investment strategy. It's not even new. Shortsellers understandably get terrible rep from execs, but their actions are more often in the public interest than you'd think.…

> The question is also what game they're playing. Deepseek came out of a hedge fund. I think it's no coincidence that their publications tend to have a large impact on AI stock prices. Its revealing that they always seem to publish after some big announcement by American AI companies. But regardless, this is one of the benefits of a duopoly.

No more revealing than OpenAI, Anthropic and Google always having some new model that just so happens to be waiting in the wings whenever their competitors announce their own model bump.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#235

Post by tiahura » Sat, Jun 27, 2026, 3:08 PM UTC

idiotsecant wrote: post #169

vintermann wrote: post #89

Earlier quoted context omitted.

It used to be the case that NSA hired the majority of all math graduates in the US, and were assumed to be years ahead in cryptography. Yet in the 90s, it became clear that they no longer were that - among other things, the cipher of the notorious Clipper chip was broken, and we can rule out that it was made weak on purpose because the whole point of Clipper was that they had a backdoor. So, despite hiring the cream…

Everyone in this thread is getting distracted by nationalism, but you hit the nail on the head. In this case for whatever reason the Chinese AI industry is collaborative and the American AI industry is not. This will result in the Chinese companies making progress faster. Full stop. This isn't a judgement on the merits of either system, only an observation of likely results.

Hasn't that been the mantra of open source for 40 years. Armies of companies, trillions of valuation, or even just Wayland, suggest that isn't always the case.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#236

Post by tiahura » Sat, Jun 27, 2026, 3:11 PM UTC

idiotsecant wrote: post #172

7speter wrote: post #65

Earlier quoted context omitted.

From what I gather, the Chinese are behind, but a lot of their research amounts to scrappy, clever discoveries in how to use more novel technologies (for Qwen and Deepseek, its mixture of expert models, that can do inference using a portion of the model at a time). The chinese also distill information from American models, so there’s that. The American companies, from my impression don’t involve themselves with such…

The American companies would love to develop these 'hacks' because it would make them more money, something they are in existential need of right now. They don't develop them because they don't collaborate publicly anymore. Where would the whole industry be if Google never allowed publishing the transformers paper? It's not a coincidence that the American AI industry grew fastest in capability when it was the most op…

Why would they collaborate? Why not defect and just keep theirs private and implement the open ones?

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#237

Post by try-working » Sat, Jun 27, 2026, 3:13 PM UTC

jingpostmedia wrote: post #165

c7b wrote: post #144

Earlier quoted context omitted.

The question is also what game they're playing. Deepseek came out of a hedge fund. I think it's no coincidence that their publications tend to have a large impact on AI stock prices. Destroying the growth story of overvalued stocks is an interesting investment strategy. It's not even new. Shortsellers understandably get terrible rep from execs, but their actions are more often in the public interest than you'd think.…

[flagged]

Nope. It is purely a marketing and distribution strategy. Without open sourcing their models, their businesses would have never gotten off the ground. I've written about this here: https://try.works/writing-1#why-chinese-ai-labs-went-open-an...

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#238

Post by nozzlegear » Sat, Jun 27, 2026, 3:21 PM UTC

intended wrote: post #80

oefrha wrote: post #30

Earlier quoted context omitted.

Which is a good thing. Self-serving motives are more reliable than altruistic ones.

The world runs on incentives. Altruism/Self-serving are down stream of that. Wikipedia is altruistic, and serves humanity quite well.

I hate to quote pithy proverbs, but "the road to hell is paved with good intentions." One can have an altruistic goal which ends up harming people too, which is where that proverb comes from. Prohibition and The War on Drugs in the US are two good examples of something that had altruistic origins[†] but ended up doing way more harm than good.

[†] Another problem with altruism: we don't all agree on whether a goal is altruistic, and what's altruistic in the enactor's eyes might not be in yours. Curating a fountain of human knowledge like Wikipedia? Probably altruistic. Protecting humanity from itself by installing your company as the stewards of frontier LLMs? Not so altruistic in my view.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#239

Post by peheje » Sat, Jun 27, 2026, 3:24 PM UTC

fer wrote: post #223

piterrro wrote: post #6

I’ve been using DeepSeek v4 pro for a month now in Kilo Code and its great. Fast, reliable, large context window and cheap as… Did 1,5B tokens this month and cost me 40usd (majority cached, but still).

Which provider? I went through 40 bucks on it on openrouter. It was not a lot of back and forth, context ended at around 300k, 15kloc output. I was using opencode, unsure if I can make the total token count visible.

OpenRouter sometimes chooses a very expensive provider. Try the floor slug or choose directly the provider. I moved to just putting 5 dollars directly on deepseek instead of going through OR.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#240

Post by otterley » Sat, Jun 27, 2026, 3:29 PM UTC

StizzurpXDD wrote: post #146

DeepSeek is, as I feel currently, the sole AI company which is actually trying to innovate rather than top mere benchmarks. Others like OpenAI, Anthropic and Google are mostly just competeing with each rather than keep innovating around the clock.

> Others like OpenAI, Anthropic and Google are mostly just competeing with each rather than keep innovating around the clock.

They compete with each other by innovating. The innovations result in more utility for the customer, but the technology isn't made public. Trade secrets are secret for a reason.

The reason people may think that DeepSeek is the "most innovative" is because of what they can observe from the outside, much like people may mistakenly conclude models are the "prettiest of the population" because not everyone is photographed for public consumption.