Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DSpark: Speculative decoding accelerates LLM inference [pdf]

github.com

Post reply on HN
Print view View original

301–310 of 393 posts

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#301

Post by stymaar »

nicce wrote: post #203
StizzurpXDD wrote: post #146

DeepSeek is, as I feel currently, the sole AI company which is actually trying to innovate rather than top mere benchmarks. Others like OpenAI, Anthropic and Google are mostly just competeing with each rather than keep innovating around the clock.

> Others like OpenAI, Anthropic and Google are mostly just competeing with each rather than keep innovating around the clock. The strategy for the most companies in the US has been for a long time to capture the social audience, whatever the mean is. Quality and innovation is the second factor. Capture the market, lock in the users, influence regulation and lobbying to keep the power.

> Capture the market, lock in the users, influence regulation and lobbying to keep the power.

“Buy every new players threatening their business” should be at #3 in your list.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#302

Post by [deleted] »

threethirtytwo wrote: post #299
operatingthetan wrote: post #296

Earlier quoted context omitted.

>Talking like this is not only against the rules here but it is some of the most vile and direct insults I’ve ever fucking read. And it doesn’t even stem from us disagreeing. It stems from you misunderstanding what was said. Why don’t you read over what I wrote and my explanation before making such a stupid comment. Let me be clear. You’re not stupid, but your reply is stupid. And your reply is stupid because of a mi…

I am self aware. I’m fully aware of the potential feelings that what I said could evoke but reality is reality and a forum is one of the few places we can still talk about reality without cancel culture or feelings muddling everything up. I’d rather speak the truth and what I believe in rather than cater to the feelings of people who cannot face objective reality. And I didn’t openly or directly insult anyone. I crit…

[deleted]

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#303

Post by mistercheph »

tiahura wrote: post #236
idiotsecant wrote: post #172

Earlier quoted context omitted.

The American companies would love to develop these 'hacks' because it would make them more money, something they are in existential need of right now. They don't develop them because they don't collaborate publicly anymore. Where would the whole industry be if Google never allowed publishing the transformers paper? It's not a coincidence that the American AI industry grew fastest in capability when it was the most op…

Why would they collaborate? Why not defect and just keep theirs private and implement the open ones?

this is not an effective long term strategy in a collaborative environment that is advancing for the same reason that having a private secret fork of the linux kernel with a few proprietary improvements is not an effective strategy.

integrating your own work with the latest public advances takes resources. For one or two small changes this is manageable, but the further you diverge from the public, the cost of maintenance rises exponentially if you want to continue to integrate public advances. when you publish your meaningful advance, you offload the maintenance burden onto everyone else (and they only have to pay a linear cost rather than an exponential one) as it's integrated by default in new work.

In most cases, the (exponential) maintenance cost of integrating public advances with secret ones exceeds the value of the public advances, so most that undertake this strategy of advancing the open frontier in secret don't attempt to integrate continually, but instead try to make a breakaway sprint in isolation to grab a few sticky customers before the unstoppable wave of the public frontier catches up.

This is a pattern commonly seen in university research departments when researchers switch into product development mode, most of these projects are a sprint to advance away from the public frontier once a good idea is found and they do good work and find a few customers for a little while. But if you check back in a few years you won't find an advanced research department but a zombie IP company that brings in a steady income via IP enforcement and a small number of customers for whom switching is too expensive.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#304

Post by unknownfuture »

threethirtytwo wrote: post #295
unknownfuture wrote: post #292

Earlier quoted context omitted.

> If they exclusively work on open source stuff where are they getting money from to survive? These are orthogonal. One can have a paid job while contributing to open source for entirely altruistic reasons. > Follow the money trail… even a donation… eventually it leads to an incentive based source or action. BS. Humans do things for altruistic reasons devoid of individual reward all the time. I, myself, maintain mult…

[flagged]

I'm quite certain I was criticising a set of ideas, not you personally.

That I misunderstood your point in context is a different issue, in which case, yup, my mistake.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#305

Post by foltik »

jingpostmedia wrote: post #165
c7b wrote: post #144

Earlier quoted context omitted.

The question is also what game they're playing. Deepseek came out of a hedge fund. I think it's no coincidence that their publications tend to have a large impact on AI stock prices. Destroying the growth story of overvalued stocks is an interesting investment strategy. It's not even new. Shortsellers understandably get terrible rep from execs, but their actions are more often in the public interest than you'd think.…

[flagged]

[dead]

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#306

Post by threethirtytwo »

threethirtytwo wrote: post #299
operatingthetan wrote: post #296

Earlier quoted context omitted.

>Talking like this is not only against the rules here but it is some of the most vile and direct insults I’ve ever fucking read. And it doesn’t even stem from us disagreeing. It stems from you misunderstanding what was said. Why don’t you read over what I wrote and my explanation before making such a stupid comment. Let me be clear. You’re not stupid, but your reply is stupid. And your reply is stupid because of a mi…

I am self aware. I’m fully aware of the potential feelings that what I said could evoke but reality is reality and a forum is one of the few places we can still talk about reality without cancel culture or feelings muddling everything up. I’d rather speak the truth and what I believe in rather than cater to the feelings of people who cannot face objective reality. And I didn’t openly or directly insult anyone. I crit…

Responding to the operatingthetan sibling reply because for some reason it’s dead (he just deleted it):

> Your comment is highly uncivil because it relies on aggressive profanity, condescending directives, and absurdist hyperbole to attack the other person. It lacks self-awareness because you hypocritically engage in the exact same hostile, rule-breaking behavior you are condemning.

From my interpretation here he attacked me. A defensive response is appropriate, but one where I do not attack his character directly but one where his unjust actions are criticized honestly. Unfortunately criticism is always somewhat condescending and for people to even be receptive of it you need to match the tone.

1. Profanity helps illustrate my point and I never directed at him.

2. I disagree with hyperbole. I think you’re outright lying to me here. So this is on you. You’re exaggerating things. He literally associated me with psychopaths and I called what he did a “misunderstanding”. So really you need to be more self aware, not me.

3. I never commented on my own rule breaking behavior and my stance on the rules of HN. I just commented ON his rule breaking behavior just as you are commenting on my behavior while ignoring your own.

I think this is enough. I’m not perfect with the rules here but this is becoming a bit of flame war and I’m fucking done. I’m ejecting myself from this thread.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#307

Post by [deleted] »

threethirtytwo wrote: post #306
threethirtytwo wrote: post #299

Earlier quoted context omitted.

I am self aware. I’m fully aware of the potential feelings that what I said could evoke but reality is reality and a forum is one of the few places we can still talk about reality without cancel culture or feelings muddling everything up. I’d rather speak the truth and what I believe in rather than cater to the feelings of people who cannot face objective reality. And I didn’t openly or directly insult anyone. I crit…

Responding to the operatingthetan sibling reply because for some reason it’s dead (he just deleted it): > Your comment is highly uncivil because it relies on aggressive profanity, condescending directives, and absurdist hyperbole to attack the other person. It lacks self-awareness because you hypocritically engage in the exact same hostile, rule-breaking behavior you are condemning. From my interpretation here he att…

[deleted]

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#308

Post by krackers »

FooBarWidget wrote: post #219
isoprophlex wrote: post #210

Earlier quoted context omitted.

Extremely interesting comment, thank you. Got some links where I can download this source material? I don't read or speak the language, but will try interrogating it with an LLM

The fifth book is on Amazon. https://www.amazon.com/XI-JINPING-GOVERNANCE-CHINA-V/dp/7119... It's already an English translation. For something shorter, you can see Arnaud Bertrand's recent review. https://arnaudbertrand.substack.com/p/the-book-the-west-refu... The review is behind a paywall, but not expensive. If you want to read policy documents directly (primary source), try the State Council / Chinese government…

> The review is behind a paywall, but not expensive.

I think the author wrote a twitter post with a summary of the content, and someone on twitter who had read the original Chinese source also chimed in with a summary

https://x.com/HaraldinChina/status/2070022115529740512

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#309

Post by DANmode »

lwansbrough wrote: post #20
tomalaci wrote: post #16

Earlier quoted context omitted.

Probably because American AI companies are on the hook for quite a lot of investment money. I think they are trying to find the magical moat to justify their valuation. Revealing optimizations similar to these would pretty much reduce their competitive position.

Chinese labs are also still behind, so they’re incentivized to collaborate and have no reason to do it in private. I suspect their tune will change if they ever take the lead..

Are they behind in models, or behind in VC money to burn on subsidized compute offered to the public and early customers?

Genuine question.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#310

Post by porphyra »

I thought this had something to do with the DGX Spark at first from the name haha. (Incidentally, a lot of recent work has gone into making the DGX Spark better at inference, like MTP yielded a 50-100% speedup, so DSpark will likely be very helpful to that end as well)
Post reply on HN