Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two different tricks for fast LLM inference

seangoedecke.com

Post reply on HN
Print view View original

61–70 of 82 posts

Re: Two different tricks for fast LLM inference

#61

Post by ankit219 »

People are misunderstanding Anthropic's fast mode because they chose to name it that way. The hints all point to a specific thing they did. The setup is costlier, its also smarter and better on tougher problems which is unheard of in terms of speed. This paper[1] fits perfectly:

The setup is parallel distill and refine. You start with parallel trajectories instead of one, then distill from them, and refine that to get to an answer. Instead of taking all trajectories to completion, they distill it quickly and refine so it gives outputs fast and yet smarter.

- paper came out in nov 2025

- three months is a good research to production pipeline

- one of the authors is at anthropic

- this approach will definitely burn more tokens than a usual simple run.

- > Anthropic explicitly warns that time to first token might still be slow (or even slower)

To what people are saying, speculative decoding wont be smarter or make any difference. Batching could be faster, but then not as costly.

Gemini Deepthink and gpt-5.2-pro use the same underlying parallel test time compute but they take each trajectory to completion before distilling and refining for the user.

[1]: https://arxiv.org/abs/2510.01123

Re: Two different tricks for fast LLM inference

#62

Post by kristjansson »

yorwba wrote: post #12

> The idea is to have a chip with SRAM large enough to fit the entire model , so inference can happen entirely in-memory. [...] So how much internal memory does the latest Cerebras chip have? 44GB. This puts OpenAI in kind of an awkward position. 44GB is enough to fit a small model (~20B params at fp16, ~40B params at int8 quantization), but clearly not enough to fit GPT-5.3-Codex. You don't really need to fit the en…

> Given that Cerebras offers models much larger than 40B at faster speeds

This fact really should have given the author pause. It’s hard to take his any of his claims seriously in its face.

Re: Two different tricks for fast LLM inference

#63

Post by ankit219 »

> Batching multiple users up thus increases overall throughput at the cost of making users wait for the batch to be full.

writer has not heard of continuous batching. this is no longer an issue. this is what makes claude code that affordable. https://huggingface.co/blog/continuous_batching

Re: Two different tricks for fast LLM inference

#64

Post by x_may »

anonymous908213 wrote: post #51
x_may wrote: post #38

Earlier quoted context omitted.

Wait sorry how did you use and expose seeds? That’s the most interesting part of your post

We were not a ChatGPT wrapper; we used a finetuned open-source model running on our own hardware, so we naturally had full control of the input parameters. I apologize if my language was ambiguous, but by "expose seeds" I simply meant users can see the seed used for each prompt and input their own in the UI, rather than "exposing secrets" of the frontier LLM APIs, if that's what you took it to mean.

I just wanted deterministic outputs and was curious how you were doing it. Sounds like probably temp = 0, which major providers no longer offer. Thanks for your response.

Re: Two different tricks for fast LLM inference

#65

Post by altcunn »

woeirua wrote: post #55

Article closes with: >The usefulness of AI agents is dominated by how few mistakes they make, not by their raw speed. Buying 6x the speed at the cost of 20% more mistakes is a bad bargain, because most of the user’s time is spent handling mistakes instead of waiting for the model6. That might be true today. I think the OpenAI-Cerebras partnership ultimately is going to lead to a paradigm shift because it will be poss…

[dead]

Re: Two different tricks for fast LLM inference

#66

Post by kristjansson »

BitsAndObjects wrote: post #11

This author thinks Cerebras chips were deployed at scale to serve users worldwide in just one month since the partnership announcement? Seems like nonsense to me.

Cerebras has been serving their own inference users for sometime. Not unreasonable to deploy a turnkey product as-is to start a partnership and then iterate from there?

Re: Two different tricks for fast LLM inference

#67

Post by anonymous908213 »

x_may wrote: post #64
anonymous908213 wrote: post #51

Earlier quoted context omitted.

We were not a ChatGPT wrapper; we used a finetuned open-source model running on our own hardware, so we naturally had full control of the input parameters. I apologize if my language was ambiguous, but by "expose seeds" I simply meant users can see the seed used for each prompt and input their own in the UI, rather than "exposing secrets" of the frontier LLM APIs, if that's what you took it to mean.

I just wanted deterministic outputs and was curious how you were doing it. Sounds like probably temp = 0, which major providers no longer offer. Thanks for your response.

No, seed and temperature are separate parameters accepted by the inference engine. You can still get deterministic outputs with high temp if you're using the same seed, provided the inference engine itself operates in a deterministic manner, and the hardware is deterministic (in testing, we did observe small non-deterministic variations when running the same prompt on the same stack but a different model of GPU).

Re: Two different tricks for fast LLM inference

#69

Post by xcodevn »

ankit219 wrote: post #61

People are misunderstanding Anthropic's fast mode because they chose to name it that way. The hints all point to a specific thing they did. The setup is costlier, its also smarter and better on tougher problems which is unheard of in terms of speed. This paper[1] fits perfectly: The setup is parallel distill and refine. You start with parallel trajectories instead of one, then distill from them, and refine that to ge…

The official document from Anthropic:

> Fast mode is not a different model. It uses the same Opus 4.6 with a different API configuration that prioritizes speed over cost efficiency. You get identical quality and capabilities, just faster responses.

Post reply on HN