Live data from Hacker News

DSpark: Speculative decoding accelerates LLM inference [pdf]

github.com

91–100 of 393 posts

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#91

These companies providing tokens, whether SOTA or not, that want to IPO are so fucked as time goes on. Can't sell their SOTA models, only slightly better than the open source models for the models they can sell, cost 20x to 50x for good models, a TAM that consists almost solely of developers, with no customer of theirs actually boasting increased profits as a result of AI... I fear their time to IPO may have passed.

[deleted]

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#92
post #67
post #60

Earlier quoted context omitted.

actually you can buy inference on third party providers that serve deepseek v4 pro with zero data retention (ZDR).

Only reliable way to have zero data retention is to self-host.

True. But at some point you got to close your eyes and take a step forward.

It’s like with VPN providers. Is Mullvad actually collaborating with law enforcement? They very well could be. It is a calculated risk.

Is DeepInfra actually logging and training or selling the logs? They could be.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#93

DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.

Sure, in part by "stealing" from American AI companies with Distillation attacks:

https://yipzap.com/anthropic-accuses-alibaba-of-largest-ai-d...

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#94
post #87
post #85

Earlier quoted context omitted.

Who is financing DeepSeek and what are they expecting in return?

They are self financed, the company that makes DeepSeek is a finance company that trades on the markets.

The CCP's approach has historically been to subsidize their companies far more than other countries do. Why would LLMs be any different?

https://www.oecd.org/en/data/dashboards/magic-database-indus...

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#95

DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.

Probably because American AI companies are on the hook for quite a lot of investment money. I think they are trying to find the magical moat to justify their valuation. Revealing optimizations similar to these would pretty much reduce their competitive position.

[dead]

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#96

DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.

Its because our culture worships pieces of paper the government tells us is worth something.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#97
post #63

Earlier quoted context omitted.

Projection is a funny thing. It causes people to misread situations all the time. Southern slaveowners feared violent retribution from freed slaves, for example [1]. It was pure projection and said more about the South than it did the slaves. The reality was there was no violent retribution. It was the opposite where the former slaveowners continued to inflict violence on the formerly enslaved. I say this because we…

It's even worse than that. China publishes stacks upon stacks of policy documents in which they explain clearly what they will do and why. This includes why they do poverty alleviation and why they believe big monopolies that own everything are bad. But almost no western observers care to read those documents. Instead, western observers, including HN, speculate endlessly about China's intentions, and "it would be nai…

I 100% agree with you and want to add something.

If you simply take what the Chinese government says at face value, you will be correct way more often than 95% of Western policy wonks, media talking heads, "analysts" and so forth. Because, like you say, they tell you everything they're doing.

In the recent US-China summit, Xi Jinping just came out and used the Thucydides Trap metaphor, which tells you everything about where China thinks it is and where it sees the US going, which is to become increasingly belligerent as their power declines. Now whether or not you agree with that assessment (I do agree), it still tells you China wants to avoid open hostilities, it sees itself as continuing to rise and it fears what a declining US might do.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#98
post #96

DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.

Its because our culture worships pieces of paper the government tells us is worth something.

Money is just a physical representation of the ability to get what you want. The problem is not money. It’s the fact that we live in a “me” society.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#99
post #69

At this point why can't someone produce a fridge or container-sized AI appliance based on legacy chips (12nm)? I imagine this would cover 80% of corporate use cases where you need to "google-in-a-box" functionality. The state-of-the-art nanometer are impossible to achieve but if you have infinite solar energy during business hours does it really matter? Every company has a parking spot so this ASIC-like appliance cou…

See "exabox" from George Hotz: https://tinycorp.myshopify.com/products/exabox-preorder

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#100
post #37
post #29

I am wondering if this is why they can offer their pro model at ~1/4th of the price compared to the other providers offering the same model, and if other providers will be able to do the same in a short timeframe.

It'd presumably help a lot, but also when you use their endpoint they get more training data.

US labs are the biggest data broker in the current history. They collect everything, dumb fuck.
Post reply on HN