Live data from Hacker News

DSpark: Speculative decoding accelerates LLM inference [pdf]

github.com

381–390 of 393 posts

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#381

Earlier quoted context omitted.

You know this from? Any sources? I'd love to learn more because it would be one of the very few industries that still write assembly by hand extensively enough to warrant hiring experts on just that.

My source is that I work on this at a non-frontier lab and also I interviewed with that team

Okay that's fascinating. Can you share what kind of things require this? Where are compilers and extensive profiling not enough? Is it just very hot right loops, or larger routines? Is it for CPU or GPU?

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#382

Earlier quoted context omitted.

Aren't they only open weights, not true open source?

The concept of open source doesn't really apply to AI models since their behavior is mostly controlled by the data they were trained on and the complex ways they are trained. Having the source code of the model by itself wouldn't help you. From a practical POV having all the training data, training infrastructure, and training know-how wouldn't help you either unless you could afford to spend the millions of dollars…

Publishing RL/SFT/self-distillation harnesses would be very impactful even without the data.

Particularly when it comes to tool use w/ self-distillation it can be done without any data... have a tool the model doesn't know? a teacher model RTFMs and the source code, and helps the student learn to get it right.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#383

Earlier quoted context omitted.

My source is that I work on this at a non-frontier lab and also I interviewed with that team

Okay that's fascinating. Can you share what kind of things require this? Where are compilers and extensive profiling not enough? Is it just very hot right loops, or larger routines? Is it for CPU or GPU?

Taking a step back: I think a lot of people have a misunderstanding of this space. Despite what the "coolest baddest hackers" on social media might have you believe, performance engineers are not thinking about assembly in that they are writing assembly by hand all day. They most certainly know how to do so, and sometimes they end up having to do it themselves, but the goal is always specific workloads and how to make them run as fast as possible, with as little work as possible. If I could have Claude take my model and spit out a perfectly fused kernel for it that I knew was correct and hit 99% MFU I would just use that (well, actually I would probably retire at that point).

Until that happens this remains an unsolved problem, so my job is to take the description of what needs to be done and find which code is on the cold setup path and can just be some PyTorch or whatever the ML researchers can write themselves, and also which part of the algorithm is where all the FLOPs are. As things get more performance critical and run more, I look at the code closer and closer. In the core of the hottest kernels, where most of the work happens, I might be placing individual instructions by hand, or going even below that and thinking about cache behavior or power characteristics.

A good performance engineer is capable of doing this while also being able to find places where they can automate this process. And there are a lot of things you can automate: layouts, schedules, pipelines. There's a lot of work going for compilers and profilers for all kinds of accelerators. Some of these operate on the "assembly" but there are all kinds of assembly. Some of these tools do almost everything for you; some are a very thin layer over the code they generate. You can see this in the interview that was linked above: it's an assembly optimization task, but you will get better results (in the time provided, at least) if you do compiler-like things. IIRC the assembler already operates on named values and in my submission I had extended the instruction selection algorithm to pack bundles based on hazards.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#384

Anyone want to bet that much like speculative execution, speculative decoding is going to introduce a whole slew of vulnerabilities in the ways LLMs work?

Don't think so because all tokens predicted speculatively are still validated against the main model (which is faster than predicting them from scratch) and only accepted if they match exactly.

If your main model is inherently-busted does validation actually matter?

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#385
post #361

Earlier quoted context omitted.

Peace for who? Like just look at the last 50-70 years of US intervention in LatAm. Backing a series of coups and extremely violent right wing dictatorships. The issue is one of incentives. The US needs cheap foreign labor because of deindustrialization policies in the 60s and 70s. These were arguably passed as a check on labor power since socialism was still looking potentially ascendant at the time. Whatever the rea…

> Peace for who? Like just look at the last 50-70 years of US intervention in LatAm. Backing a series of coups and extremely violent right wing dictatorships. For the world. Compare the those 70 years to the previous 70 years. Regime change and intervention is significantly better than full scale invasion, total war and colonization of other people. > China's play is more mercantile. Because they are held in check by…

Eh a death squad is a death squad tbh.

I mean maybe, but China's industrial strategy isn't well served by war. I just don't see the structural incentive.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#386

Earlier quoted context omitted.

I am also a fan of giving credit where it is due, and not giving it where it is not due. China is not an innovator. Perhaps they are only beginning to be, but historically, this is simply not the case, and yet distillation still falls squarely under "not innovating".

> historically, this is simply not the case I wish people knew more history before using the term “historically” The Chinese civilisation has been one of the most long living ones with plenty of “innovation” throughout history, both long ago and recently.

Long ago, OK.

Recently, maybe. I see a lot more copycat behavior however.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#387
post #6

I’ve been using DeepSeek v4 pro for a month now in Kilo Code and its great. Fast, reliable, large context window and cheap as… Did 1,5B tokens this month and cost me 40usd (majority cached, but still).

I've been preferring Mimo recently. Same price as deekseek, more reliable tool calling (subjectively), and has some nice qualities in terms of prose, etc.

I've heard others say that Deepseek tends to be smarter on specific problems but that Mimo tends to more well-rounded.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#388

Earlier quoted context omitted.

Don't think so because all tokens predicted speculatively are still validated against the main model (which is faster than predicting them from scratch) and only accepted if they match exactly.

If your main model is inherently-busted does validation actually matter?

But then it's not a problem of speculative decoding, fix your dam main model.

BTW, in case there is confusion, we are not talking about CPU speculative execution affecting model inference at all, just about this specific technique: predict tokens via a smaller drafter model, then validate them against the main model in batch.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#389
post #144

Earlier quoted context omitted.

The question is also what game they're playing. Deepseek came out of a hedge fund. I think it's no coincidence that their publications tend to have a large impact on AI stock prices. Destroying the growth story of overvalued stocks is an interesting investment strategy. It's not even new. Shortsellers understandably get terrible rep from execs, but their actions are more often in the public interest than you'd think.…

I always see these malicious speculations without evidence, simply because these companies are from China.

xenophobia can be profitable, it's not hard to follow the financial incentives and explain the rhetoric.

Re: DSpark: Speculative decoding accelerates LLM inference [pdf]

#390
post #184

Earlier quoted context omitted.

Yeah can definitely see a world where china pivots and we're stuck with closed/closed Mistral...don't fumble this

What are some things that China has pivoted on in history?

IIRC Mao was surprisingly quite okay with male homosexuality.
Post reply on HN