The non-linear scaling on the slider is an excellent UX.
How fast is N tokens per second really?
41–50 of 105 posts
Re: How fast is N tokens per second really?
#42Cool visualization, but most of the token generation in my sessions doesn't go to output code or even the text I see. Reasoning tokens make up most of the output. That can only occur after processing the input files and context.
For non-trivial work I go through hundreds of thousands of tokens (combined prefill + tg of course) before even getting to some useful text output.
I mostly use LLMs for exploration and studies, rarely code generation. Prefill matters heavily for this. Even in the high hundreds or low thousands prefill rate I spend a lot of time waiting on the LLM (doing other things, not twiddling thumbs)
Re: How fast is N tokens per second really?
#43Nice, I always thought 15 tok/sec is too "slow"
Re: How fast is N tokens per second really?
#44[dead]
Re: How fast is N tokens per second really?
#45It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent. On the other hand, I've been using Mimo and Minimax a lot recently. They routinely reach 100-150 tokens per second and that feels too fast , to the point where it's hard to keep up with what it's actually doing. Great for subagents though.
They routinely reach 100-150 tokens per second and that feels too fast, to the point where it's hard to keep up with what it's actually doing.
There is no way you can follow what is going on even at 30 tokens per second. Maybe you can maintain a rough idea of what is going on for some tens of seconds but that is probably about it. Follow it in any detail, no chance. Reason about what you read, absolutely no chance.
800 tok/s — Cerebras-class, where the bottleneck is your eyeballs
I do not understand why they say this. I am not sure if it is even true. 800 tokens sounds like a page of text and I would assume you can look at one page per second without hitting any limitation of your eyes. Or is the resolution of the human not good enough to see an entire page at once and you have to scan it with the fovea? Scrolling text might of course hit the temporal resolution limit. But why does this even matter, your brain can not process anything close to the amount of information your eyes can take in.
Re: How fast is N tokens per second really?
#46Curious about the other way around, how many tokens per second a productive developer codes in a day?
Re: How fast is N tokens per second really?
#47This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
Do you have ideas/suggestions for agentic workflows that only start making sense at such speeds?
Obviously things will get expensive quick, but the main thing for me would be not dealing with the context switch every time I leave the agent to do stuff on it's own.
Feedback loops for prototyping could become even quicker.
Re: How fast is N tokens per second really?
#48Earlier quoted context omitted.
If you have a Cerebras Code subscription you can experience it right now. Indeed, a very different experience.
Used them for a while! They didn't seem to have prompt caching so I burnt through the daily 24M token limitations really quickly when doing large scale changes on a codebase (essentially a team's worth of menial migration/refactoring work). A lot of it was okay, but plenty had to be re-done and I still spotted some issues months down the line, in part I blame their model catalogue which did get an update to GLM 4.7 s…
Have you tried Codex? If you have, how does it compare to Opus?
Re: How fast is N tokens per second really?
#49I think your demo need more realistic thinking logs because thinking usually burns at least 2x to 3x of tokens of the code and for harder tasks much more.
Indeed, at 30tok/s make it pause for 20 seconds while "thinking" is streaming (and hidden); that's the real experience.
[deleted]
Re: How fast is N tokens per second really?
#50Earlier quoted context omitted.
People seem to use these tools very differently from each other. I value intelligence over speed any day. My programs are written in Haskell, so there are rarely any tasks which require thousands and thousands of lines to solve. Just intelligence. If there are rote tasks, I want the LLM to help me find intelligent ways of automating it: the right abstraction, the right meta-programming technique. I constantly push Op…
Why do you use Haskell? Why not something that produces a more predictable memory use at runtime? (I’m asking earnestly as a former Haskeller turned Rustacean who sees the value in “Boring Haskell”, but favours strictness for anything internet-facing and many things that aren’t compilers.)
I use Haskell because purity and strong typing gives so much control over what each part of the program does. This has huge benefits when it comes to security, and just general lack of bugs. Also, it makes the code easy to write, once the types are in place.
I use Haskell because I find laziness to be a super power. I can solve so many problems in the most straightforward way, and then laziness saves my butt w.r.t. performance.
I use Haskell because it is a better C than C is. The foreign function interface is brilliant, and I can take C primitives and apply all the abstraction mechanisms from Haskell to them. My latest project has been OpenGL based, so lots of caring about byte alignments and shovelling data to the GPU. But all this can be automated with clever use of type classes and Generics (Haskells super cool meta system of data types.)
I use Haskell because I love applying abstractions to make code which describes the problem, and then the compiler finds the solution.
I don’t do programming for embedded, so I am rarely memory constrained. I also understand Haskell memory usage quite well, and can get myself out of trouble.