Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two different tricks for fast LLM inference

seangoedecke.com

Post reply on HN
Print view View original

51–60 of 82 posts

Re: Two different tricks for fast LLM inference

#51

Post by anonymous908213 »

x_may wrote: post #38
anonymous908213 wrote: post #8

Earlier quoted context omitted.

> especially if quality degrades at all It is worth noting that consumers are completely and totally incapable of detecting quality degradation with any accuracy. Which is a given since the models are already effectively random, but there is a strong bent to hallucinate degradations. Having done frontend work for an AI startup, complaints of degrading the model were by far the most common, despite the fact that not o…

Wait sorry how did you use and expose seeds? That’s the most interesting part of your post

We were not a ChatGPT wrapper; we used a finetuned open-source model running on our own hardware, so we naturally had full control of the input parameters. I apologize if my language was ambiguous, but by "expose seeds" I simply meant users can see the seed used for each prompt and input their own in the UI, rather than "exposing secrets" of the frontier LLM APIs, if that's what you took it to mean.

Re: Two different tricks for fast LLM inference

#52

Post by tasuki »

> A good analogy is a bus system. If you had zero batching for passengers - if, whenever someone got on a bus, the bus departed immediately - commutes would be much faster for the people who managed to get on a bus.

A good analogy? I wonder... how do buses work at your place? Do they wait to be at least half-full before departing? I used to do that in the Simutrans game!

Where I'm from, buses usually depart on schedule, whether you get on the bus or not...

[Edit:] Otherwise an insightful article I guess.

Re: Two different tricks for fast LLM inference

#53

Post by foobar10000 »

zozbot234 wrote: post #45
gchadwick wrote: post #13

Earlier quoted context omitted.

> If copying user tokens was the bottle neck, batching would not achieve any speed up. Reality is more complex. As context length grows your KV cache becomes large and will begin to dominate your total FLOPs (and hence bytes loaded). The issue with KV cache is you cannot batch it because only one user can use it, unlike static layer weights where you can reuse them across multiple users. Emerging sparse attention tec…

> The issue with KV cache is you cannot batch it because only one user can use it This is not really correct given how input token caching works and the reality of subagent workloads. You could launch many parallel subagents sharing some portion of their input tokens and use batching for that task.

2 things:

1. Parallel investigation : the payoff form that is relatively small - starting K subagents assumes you have K independent avenues of investigation - and quite often that is not true. Somewhat similar to next-turn prediction using a speculative model - works well enough for 1 or 2 turns, but fails after.

2. Input caching is pretty much fixes prefill - not decode. And if you look at frontier models - for example open-weight models that can do reasoning - you are looking at longer and longer reasoning chains for heavy tool-using models. And reasoning chains will diverge very vey quickly even from the same input assuming a non-0 temp.

Re: Two different tricks for fast LLM inference

#55

Post by woeirua »

Article closes with:

>The usefulness of AI agents is dominated by how few mistakes they make, not by their raw speed. Buying 6x the speed at the cost of 20% more mistakes is a bad bargain, because most of the user’s time is spent handling mistakes instead of waiting for the model6.

That might be true today. I think the OpenAI-Cerebras partnership ultimately is going to lead to a paradigm shift because it will be possible to scale these chips up to the point where a model like full Codex-5.3 can run on them and then you'll have a super fast model that makes relatively few errors. A Codex-5.3 model running at these speeds is more than sufficient to actually start replacing customer facing jobs.

Re: Two different tricks for fast LLM inference

#56

Post by croes »

woeirua wrote: post #55

Article closes with: >The usefulness of AI agents is dominated by how few mistakes they make, not by their raw speed. Buying 6x the speed at the cost of 20% more mistakes is a bad bargain, because most of the user’s time is spent handling mistakes instead of waiting for the model6. That might be true today. I think the OpenAI-Cerebras partnership ultimately is going to lead to a paradigm shift because it will be poss…

Is the problem solved that training on AI generated data makes the model worse?

If not then updates to the current models will become harder and harder

Re: Two different tricks for fast LLM inference

#57

Post by martinald »

criemen wrote: post #3

One other thing I'd assume Anthropic is doing is routing all fast requests to the latest-gen hardware. They most certainly have a diverse fleet of inference hardware (TPUs, GPUs of different generations), and fast will be only served by whatever is fastest, whereas the general inference workload will be more spread out.

This was my assumption - GB200 memory bandwidth is 2.4x faster than H100, so I think personally that's all it is. Doesn't really make sense otherwise as yes there are tricks to get faster time to first token but not really for the same model in throughput terms (speculative decoding etc, but they already use that).

I'm happy to be wrong but I don't think it's batching improvements.

Re: Two different tricks for fast LLM inference

#58

Post by olivermuty »

woeirua wrote: post #55

Article closes with: >The usefulness of AI agents is dominated by how few mistakes they make, not by their raw speed. Buying 6x the speed at the cost of 20% more mistakes is a bad bargain, because most of the user’s time is spent handling mistakes instead of waiting for the model6. That might be true today. I think the OpenAI-Cerebras partnership ultimately is going to lead to a paradigm shift because it will be poss…

At 40gb and a rumoured 5 to 7 TB size of the proprietary flagships you are looking at several megawatts to run one single model instance. Cerebras is insanely power hungry. It is funny how they are essentially a parallell happenstance (chips being made for other compute stuff also works for LLMs) to gaming processors accidentally being good for LLMs.

The world will be much more interesting when real bespoke hardware built for actual LLM usage comes to market. This means silicon of the SIMD flavour or other variants, but using DRAM so you can pack more tightly.

Re: Two different tricks for fast LLM inference

#59

Post by nmilo »

I don’t really get the bus analogy. It seems like it massively increases latency but as soon as you’re “on the bus” throughput is normal? When in reality (if I understand correctly) opus-fast is just giving you a bigger portion of the batch so increasing throughput with little affect on latency? (I’m assuming anthropic gets enough volume that these batches fill up pretty much instantly)
Post reply on HN