Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How fast is N tokens per second really?

mikeveerman.github.io

Post reply on HN
Print view View original

61–70 of 105 posts

Re: How fast is N tokens per second really?

#61

Post by c7b »

colechristensen wrote: post #31
c7b wrote: post #24

Earlier quoted context omitted.

Do you have ideas/suggestions for agentic workflows that only start making sense at such speeds?

Branching strategies, do 10 things in parallel and evaluate for the best at the end or something along the lines of an evolutionary algorithms. Turn up the temperature on an LLM and have a survival mechanism, and generate solutions to the same problem over and over.

Regarding the first, parallel requests to the same loaded model seem to work pretty well, I'm trying to find time to look more into it myself, but this may be something that might already be within reach for local models.

Re: How fast is N tokens per second really?

#62

Post by metalliqaz »

ricardobeat wrote: post #40

It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent. On the other hand, I've been using Mimo and Minimax a lot recently. They routinely reach 100-150 tokens per second and that feels too fast , to the point where it's hard to keep up with what it's actually doing. Great for subagents though.

I run models in the ~120B class on my old server (96GB DDR4) and it manages about 3-3.5 tok/sec. It is indeed painfully slow to watch, but I find if I walk away or bury the window and do something else, it always seems to be done when I check back

Re: How fast is N tokens per second really?

#63

Post by flockonus »

sedatk wrote: post #51
flockonus wrote: post #46

Curious about the other way around, how many tokens per second a productive developer codes in a day?

Much more, given that you need to incorporate the dev's thought process too.

Interesting point, i didn't consider the thought process as tokens.

Re: How fast is N tokens per second really?

#64

Post by travisjungroth »

danbruc wrote: post #45
ricardobeat wrote: post #40

It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent. On the other hand, I've been using Mimo and Minimax a lot recently. They routinely reach 100-150 tokens per second and that feels too fast , to the point where it's hard to keep up with what it's actually doing. Great for subagents though.

They routinely reach 100-150 tokens per second and that feels too fast, to the point where it's hard to keep up with what it's actually doing. There is no way you can follow what is going on even at 30 tokens per second. Maybe you can maintain a rough idea of what is going on for some tens of seconds but that is probably about it. Follow it in any detail, no chance. Reason about what you read, absolutely no chance. 8…

On top of the other comments, this reads like a half-joke.

Re: How fast is N tokens per second really?

#65

Post by danbruc »

moralestapia wrote: post #59
danbruc wrote: post #45

Earlier quoted context omitted.

They routinely reach 100-150 tokens per second and that feels too fast, to the point where it's hard to keep up with what it's actually doing. There is no way you can follow what is going on even at 30 tokens per second. Maybe you can maintain a rough idea of what is going on for some tens of seconds but that is probably about it. Follow it in any detail, no chance. Reason about what you read, absolutely no chance. 8…

>I do not understand why they say this. Click on 800. Try to read the text. You'll understand.

Because it is scrolling. If they would show one page of text while filling the next one in the background, the result would probably be somewhat like flicking through a book at one page per second. You still can not read one page per second but you would not be limited by your eyes being unable to recognizing the quickly scrolling text.

EDIT: As others have pointed out and I now did some reading on, it is an illusion that you can see all the text on a page at once, that is beyond the resolution limit of the human eye. To actually see all the words, you have to scan the page and that takes several seconds. From the numbers I have seen, it seems that the ultimate limit is probably below 30 tokens per second, no matter what, even using rapid serial visual presentation to cut out eye movements. Even 10 to 20 tokens per second is probably pushing it and unsustainable for many, if not most, people.

Re: How fast is N tokens per second really?

#68

Post by sig_kill »

antirez wrote: post #20

Token/sec only makes sense once you tell me three four things: 1. decoding t/s, that is, when the model is generating text in the autoregressive fashion. 2. prefill t/s, that is, prompt processing speed. 3. What is the slope of those two numbers as the context size increases. An implementation that decodes at 50t/s with 2k context but decodes at 7t/s at 100k context is going to be a lot less useful that it seems at a…

[dead]

Re: How fast is N tokens per second really?

#70

Post by SpyCoder77 »

danbruc wrote: post #65
moralestapia wrote: post #59

Earlier quoted context omitted.

>I do not understand why they say this. Click on 800. Try to read the text. You'll understand.

Because it is scrolling. If they would show one page of text while filling the next one in the background, the result would probably be somewhat like flicking through a book at one page per second. You still can not read one page per second but you would not be limited by your eyes being unable to recognizing the quickly scrolling text. EDIT: As others have pointed out and I now did some reading on, it is an illusion…

Did someone say rapid serial visual presentation? I made a tool for that! Https://wordflashreader.vercel.app
Post reply on HN