Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How fast is N tokens per second really?

mikeveerman.github.io

Post reply on HN
Print view View original

91–100 of 105 posts

Re: How fast is N tokens per second really?

#91

Post by michibertel »

kqr wrote: post #89
ricardobeat wrote: post #40

It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent. On the other hand, I've been using Mimo and Minimax a lot recently. They routinely reach 100-150 tokens per second and that feels too fast , to the point where it's hard to keep up with what it's actually doing. Great for subagents though.

> It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent. Calling the token rate the rate at which they "type" is a bit misleading. They also do virtually all of their more complex reasoning in tokens, so 5 tokens per second is also their thinking speed. And thinking at 5 tokens per second is glacially slow. This is why faster versions of strong models…

How many tok/s does an average human think?

Re: How fast is N tokens per second really?

#92

Post by madwolf »

michibertel wrote: post #91
kqr wrote: post #89

Earlier quoted context omitted.

> It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent. Calling the token rate the rate at which they "type" is a bit misleading. They also do virtually all of their more complex reasoning in tokens, so 5 tokens per second is also their thinking speed. And thinking at 5 tokens per second is glacially slow. This is why faster versions of strong models…

How many tok/s does an average human think?

Most of my thinking is non-verbal. I don't think in sentences. I CAN think in sentences and internally rationalize my actions and explain them and sometimes that's beneficial (rubber duck debuggin, sometimes it's good to verbalize and explain something) but usually I don't do it

Re: How fast is N tokens per second really?

#93

Post by goldenarm »

__natty__ wrote: post #90

I wonder when we reach speed of 1000 tps with high quality models. 5 years? 10 years?

We technically can (check Cerebras grok and Gemini diffusion), but it's not economically viable and not a priority for product managers.

Maybe when intelligence plateaus it could become a main differentiating factor, like smartphones and battery life.

Re: How fast is N tokens per second really?

#94

Post by buzzin_ »

__natty__ wrote: post #90

I wonder when we reach speed of 1000 tps with high quality models. 5 years? 10 years?

Don't set your goals so low. We already reached 17k on a small models.

Since the whole goal of software architecture schemes it to allow the rest of us non-geniuses to still understand it and modify it, perhaps the same could be true of llms.

Perhaps a million-per-second hypothetical (small) model can be more useful than a state of the art big one.

Re: How fast is N tokens per second really?

#96

Post by jjbinx007 »

nl wrote: post #95

I always wonder what the world is going to be like when these are 2 orders of magnitude faster. Something like https://chatjimmy.ai/ will do 14,000 tokens per second, and it's a completely different experience to what we see now. It's more like a page load than a conversation.

That's really impressive from a speed point of view but it hallucinates like it's on drugs.

I'd much rather trade speed for accuracy.

Re: How fast is N tokens per second really?

#97

Post by gunalx »

jjbinx007 wrote: post #96
nl wrote: post #95

I always wonder what the world is going to be like when these are 2 orders of magnitude faster. Something like https://chatjimmy.ai/ will do 14,000 tokens per second, and it's a completely different experience to what we see now. It's more like a page load than a conversation.

That's really impressive from a speed point of view but it hallucinates like it's on drugs. I'd much rather trade speed for accuracy.

Its llama 3.1 8b. So not a particular good model, mostly as a tech demo of the speed possible.

Re: How fast is N tokens per second really?

#99

Post by kqr »

michibertel wrote: post #91
kqr wrote: post #89

Earlier quoted context omitted.

> It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent. Calling the token rate the rate at which they "type" is a bit misleading. They also do virtually all of their more complex reasoning in tokens, so 5 tokens per second is also their thinking speed. And thinking at 5 tokens per second is glacially slow. This is why faster versions of strong models…

How many tok/s does an average human think?

This question gets into information theory way beyond me, but I suspect it depends a lot on the task at hand. Human brains aren't very effective at combining sources of statistical variation, but they're great at other things. I'm personally most impressed by the cerebellum. It is highly trainable, yet if we tried to translate the things it does to maintain locomotion, proprioception, coordination of movement, etc. into tokens would probably result in a high token rate.
Post reply on HN