Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How fast is N tokens per second really?

mikeveerman.github.io

Post reply on HN
Print view View original

81–90 of 105 posts

Re: How fast is N tokens per second really?

#81

Post by colechristensen »

c7b wrote: post #61
colechristensen wrote: post #31

Earlier quoted context omitted.

Branching strategies, do 10 things in parallel and evaluate for the best at the end or something along the lines of an evolutionary algorithms. Turn up the temperature on an LLM and have a survival mechanism, and generate solutions to the same problem over and over.

Regarding the first, parallel requests to the same loaded model seem to work pretty well, I'm trying to find time to look more into it myself, but this may be something that might already be within reach for local models.

Sure, it's possible, but you'd start to use it much more and in more advanced ways. Like "thinking hard" would consist of spawning a dozen different inferences from the same cached point and then picking the best one.

Re: How fast is N tokens per second really?

#82

Post by calmingsolitude »

sig_kill wrote: post #67
SXX wrote: post #29

I think your demo need more realistic thinking logs because thinking usually burns at least 2x to 3x of tokens of the code and for harder tasks much more.

[dead]

You've advertised twice on this thread about your site, but I personally find the UX to be worse, not to mention the phantom scroll issues and (imo) the tasteless animations.

Re: How fast is N tokens per second really?

#85

Post by blobbers »

stingraycharles wrote: post #74
blobbers wrote: post #72

Isn't the thinking part the part that burns the tokens? You're just outputting tokens.

Totally depends, but I think this is mostly just an illustration of overall speed, regardless of the content.

Okay, but I think the realistic thing is * burns 18000 tokens thinking of the solution * outputs 1000 tokens of code

So you can easily follow the 1000 tokens of code, and the 18000 tokens of thinking is you sitting around waiting for your GPU to process the LLM.

Re: How fast is N tokens per second really?

#86

Post by j_maffe »

ricardobeat wrote: post #40

It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent. On the other hand, I've been using Mimo and Minimax a lot recently. They routinely reach 100-150 tokens per second and that feels too fast , to the point where it's hard to keep up with what it's actually doing. Great for subagents though.

I think the metric should be reading speed, not writing speed. At the very least it should be speech speed.

Re: How fast is N tokens per second really?

#87

Post by jamesdeakee »

antirez wrote: post #20

Token/sec only makes sense once you tell me three four things: 1. decoding t/s, that is, when the model is generating text in the autoregressive fashion. 2. prefill t/s, that is, prompt processing speed. 3. What is the slope of those two numbers as the context size increases. An implementation that decodes at 50t/s with 2k context but decodes at 7t/s at 100k context is going to be a lot less useful that it seems at a…

[flagged]

Re: How fast is N tokens per second really?

#89

Post by kqr »

ricardobeat wrote: post #40

It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent. On the other hand, I've been using Mimo and Minimax a lot recently. They routinely reach 100-150 tokens per second and that feels too fast , to the point where it's hard to keep up with what it's actually doing. Great for subagents though.

> It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent.

Calling the token rate the rate at which they "type" is a bit misleading. They also do virtually all of their more complex reasoning in tokens, so 5 tokens per second is also their thinking speed. And thinking at 5 tokens per second is glacially slow.

This is why faster versions of strong models do so well on reasoning tasks like playing text adventure games[1]. Their output isn't better on a token-for-token basis, but they get so much more thinking in during a given time window, they get more opportunities to find the right conclusion.

[1]: https://entropicthoughts.com/updated-llm-benchmark

Post reply on HN