How fast is N tokens per second really?
mikeveerman.github.io
How fast is N tokens per second really?
1–10 of 105 posts
Re: How fast is N tokens per second really?
#2Thank you for this great utility. I love the "gut feel" calibration utilities like this one!
Re: How fast is N tokens per second really?
#3Neat visual. 5 tok/s is still faster than me!
Re: How fast is N tokens per second really?
#4Neat website, the visualization is great. I had a hard time wrapping my head around the tokens/s thing but this made it easy.
Re: How fast is N tokens per second really?
#5Interesting. It seems to me that with that speed (20-30) on local hardware the real issue is quality of output, not tokens per sec.
Re: How fast is N tokens per second really?
#6This is cool, thanks for making it.
Re: How fast is N tokens per second really?
#7Neat visual. 5 tok/s is still faster than me!
I had the opposite reaction, 5tok/s is so slow that when you include all the reasoning and thinking + warmup it is far slower than me.
Re: How fast is N tokens per second really?
#8Interesting. It seems to me that with that speed (20-30) on local hardware the real issue is quality of output, not tokens per sec.
It really depends. With the new "thinking" models they usually spend some time before writing the final answer. If they "think" for 1k tokens, that's a minute of spinning wheel you're gonna see for each question. Add that to the prompt processing, and diminishing speeds as context increases, and it becomes really slow for longer sessions.
Re: How fast is N tokens per second really?
#9On avg 1 token = 4 chars
So 75 tokens/s is ~ 300 chars per second which is the speed you'd get with a 2400 baud modem
Re: How fast is N tokens per second really?
#10super cool, thanks