Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DSpark: Speculative decoding accelerates LLM inference [pdf]

github.com

Post reply on HN
Print view View original

31–40 of 393 posts

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#31

Post by budsniffer952 »

tw1984 wrote: post #24
lwansbrough wrote: post #20

Earlier quoted context omitted.

Chinese labs are also still behind, so they’re incentivized to collaborate and have no reason to do it in private. I suspect their tune will change if they ever take the lead..

> Chinese labs are also still behind, so they’re incentivized to collaborate and have no reason to do it in private. US labs in Google, Meta and SpaceX are not leading, none of them managed to build something on par with GLM 5.2. Care to explain to me why they still don't collaborate and still choose to do it in private?

Wait, are you claiming that these companies haven't contributed to the ecosystem via research and open source?

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#33

Post by vidarh »

tw1984 wrote: post #24
lwansbrough wrote: post #20

Earlier quoted context omitted.

Chinese labs are also still behind, so they’re incentivized to collaborate and have no reason to do it in private. I suspect their tune will change if they ever take the lead..

> Chinese labs are also still behind, so they’re incentivized to collaborate and have no reason to do it in private. US labs in Google, Meta and SpaceX are not leading, none of them managed to build something on par with GLM 5.2. Care to explain to me why they still don't collaborate and still choose to do it in private?

I'm not sure I'd put Google in that list, but either way: Because they think they have enough capital that they can catch up and don't need the reputational boost of this.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#34

Post by abc123abc123 »

colordrops wrote: post #23
lwansbrough wrote: post #20

Earlier quoted context omitted.

Chinese labs are also still behind, so they’re incentivized to collaborate and have no reason to do it in private. I suspect their tune will change if they ever take the lead..

So the marketplace is working.

This is the way! Open source models will benefit, and once open source models reach the state of "good enough" the hyped up US AI companies will fear, since the availability of free, good enough, AI models will set the ceiling for how much they can charge. Then the bubble will pop.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#35

Post by abc123abc123 »

budsniffer952 wrote: post #27
tomalaci wrote: post #16

Earlier quoted context omitted.

Probably because American AI companies are on the hook for quite a lot of investment money. I think they are trying to find the magical moat to justify their valuation. Revealing optimizations similar to these would pretty much reduce their competitive position.

[flagged]

This is incorrect binary thinking. Them releasing open source can be good, but that does not commit you to think that china or chinese companies are saints. There are many shades of grey here and one does not exclude the other (nor include it).

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#36

Post by vidarh »

rvz wrote: post #19
kamranjon wrote: post #12

DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.

Exactly. They did not have to open up their research up and this is what happens when smart researchers are forced to squeeze performance gains out of existing hardware. They don't have TPUs or access to the latest Vera Rubin GPUs either to get performance gains for free. All of the optimizations Deepseek have done are in software and it goes down to the PTX assembly level. Compared to Anthropic who are celebrating i…

> Compared to Anthropic who are celebrating in fixing a flickering issue in a terminal app which took months to fix.

It's funny, because if you ran Claude Code on a slow terminal, the cause of the flicker was obvious: They kept dumping the entire history of the chat back into the terminal in a number of situations, and relied on the terminal to them end up in the correct state.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#37

Post by vidarh »

pokot0 wrote: post #29

I am wondering if this is why they can offer their pro model at ~1/4th of the price compared to the other providers offering the same model, and if other providers will be able to do the same in a short timeframe.

It'd presumably help a lot, but also when you use their endpoint they get more training data.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#38

Post by CuriouslyC »

vidarh wrote: post #33
tw1984 wrote: post #24

Earlier quoted context omitted.

> Chinese labs are also still behind, so they’re incentivized to collaborate and have no reason to do it in private. US labs in Google, Meta and SpaceX are not leading, none of them managed to build something on par with GLM 5.2. Care to explain to me why they still don't collaborate and still choose to do it in private?

I'm not sure I'd put Google in that list, but either way: Because they think they have enough capital that they can catch up and don't need the reputational boost of this.

As good as Gemini's visual intelligence is, it's a terrible agent.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#39

Post by nubg »

oefrha wrote: post #30
lwansbrough wrote: post #20

Earlier quoted context omitted.

Chinese labs are also still behind, so they’re incentivized to collaborate and have no reason to do it in private. I suspect their tune will change if they ever take the lead..

Which is a good thing. Self-serving motives are more reliable than altruistic ones.

Very interesting take

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#40

Post by nicce »

vidarh wrote: post #37
pokot0 wrote: post #29

I am wondering if this is why they can offer their pro model at ~1/4th of the price compared to the other providers offering the same model, and if other providers will be able to do the same in a short timeframe.

It'd presumably help a lot, but also when you use their endpoint they get more training data.

This applies to every provider. OpenAI seems to be the worst hoarder.
Post reply on HN