I’ve been using DeepSeek v4 pro for a month now in Kilo Code and its great. Fast, reliable, large context window and cheap as… Did 1,5B tokens this month and cost me 40usd (majority cached, but still).
Is there a way to see how many tokes one does with claude code (pro)?
DSpark: Speculative decoding accelerates LLM inference [pdf]
11–20 of 393 posts
Re: DSpark: Speculative decoding accelerates LLM inference [pdf]
#12DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.
Re: DSpark: Speculative decoding accelerates LLM inference [pdf]
#13I see a world soon where there’s an extremely wide variety of small models for speculative decoding, unique to use cases, companies, and even individuals.
yes, heavily constrained by sophisticated guardrails.
this is definitely where things are going. the enormous "eat the world" models have extreme diminishing returns by comparison.
Re: DSpark: Speculative decoding accelerates LLM inference [pdf]
#14DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.
Publishing by necessity I wonder? American labs on the cutting edge pioneering the way forward, so Deepseek open sourcing what they’ve got is to help even the playing field.
Hopefully the experts here can offer insight. The above is just my hunch and I’m not a specialist in this field.
Re: DSpark: Speculative decoding accelerates LLM inference [pdf]
#15DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.
Publishing by necessity I wonder? American labs on the cutting edge pioneering the way forward, so Deepseek open sourcing what they’ve got is to help even the playing field. Hopefully the experts here can offer insight. The above is just my hunch and I’m not a specialist in this field.
Wouldn’t that just help the American labs anyway though? Or do they assume they’ve actually already figured this stuff out and kept it secret?
Re: DSpark: Speculative decoding accelerates LLM inference [pdf]
#16DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.
Probably because American AI companies are on the hook for quite a lot of investment money. I think they are trying to find the magical moat to justify their valuation.
Revealing optimizations similar to these would pretty much reduce their competitive position.
Re: DSpark: Speculative decoding accelerates LLM inference [pdf]
#17Presumably this has been in production for a while, and is one of the reasons they were able to dramatically lower prices a month ago?
Lookahead Sparse Attention should be playing a big role as well, as it dramatically slashes memory consumption.
Re: DSpark: Speculative decoding accelerates LLM inference [pdf]
#18Must be wonderful to be on the board of OpenAi et al & their PE investors whilst China keeps blowing up these mines under their feet lmao. Luckily Korean pension funds will buy all the trash as usual but goddamn you gotta start moving quick or you are gonna need some serious AGI to show you how to offload those bonds
"We will build the machine-god and pray for it to pay for itself."
Re: DSpark: Speculative decoding accelerates LLM inference [pdf]
#19DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.
Exactly. They did not have to open up their research up and this is what happens when smart researchers are forced to squeeze performance gains out of existing hardware.
They don't have TPUs or access to the latest Vera Rubin GPUs either to get performance gains for free. All of the optimizations Deepseek have done are in software and it goes down to the PTX assembly level.
Compared to Anthropic who are celebrating in fixing a flickering issue in a terminal app which took months to fix.
Re: DSpark: Speculative decoding accelerates LLM inference [pdf]
#20DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.
Probably because American AI companies are on the hook for quite a lot of investment money. I think they are trying to find the magical moat to justify their valuation. Revealing optimizations similar to these would pretty much reduce their competitive position.
Chinese labs are also still behind, so they’re incentivized to collaborate and have no reason to do it in private.
I suspect their tune will change if they ever take the lead..