This is great.
How fast is N tokens per second really?
11–20 of 105 posts
Re: How fast is N tokens per second really?
#12This is great.
Agentic coding at 600+ tokens/sec is going to be a radically different beast.
Coming soon-ish?
Re: How fast is N tokens per second really?
#13Neat visual. 5 tok/s is still faster than me!
yeah 3t/s seems human. only that i never wrote code perfectly top to bottom.
Re: How fast is N tokens per second really?
#14This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
If you have a Cerebras Code subscription you can experience it right now. Indeed, a very different experience.
Re: How fast is N tokens per second really?
#15This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
For small enough tasks with tight enough workflows, you can have it right now. Ie if you can constrain the task to work well with GPT OSS 120B/llama 3.3/qwen 3, then you can get upwards of 600 TPS on groq and up to 3k TPS on Cerebras.
Those models aren’t comparable to Opus, or even weaker models like MiniMax, but for certain task (focused context and prompts, strict workflows, single purpose requests) you absolutely can use these models and get insane speeds.
Re: How fast is N tokens per second really?
#16Neat visual. 5 tok/s is still faster than me!
I had the opposite reaction, 5tok/s is so slow that when you include all the reasoning and thinking + warmup it is far slower than me.
The sweet spot for being just fast enough to not irritate you is 10tok/s. Still slow but faster than you can sustain at typing and thinking. Just interesting to observe.
Re: How fast is N tokens per second really?
#17This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
If you have a Cerebras Code subscription you can experience it right now. Indeed, a very different experience.
It’s GLM 4.7, GPT OSS 120B, or llama 3.1 8B so not exactly the latest or best models.
But GLM is good enough for many small tasks, certainly enough to get a taste for Cerebras’ high speeds!
[edit: actually that’s just their general models, I can’t see what Cerebras code offers. It was Qwen-coder when it launched but I don’t know what it is now. I think GLM 4.7 but I’m not completely sure]
Re: How fast is N tokens per second really?
#18This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
i really want a qwen on one of these chips: https://chatjimmy.ai
15k tokens/s would get me feeling like its actually worth splitting out worktrees to try several approaches to a problem
Re: How fast is N tokens per second really?
#19This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
If you have a Cerebras Code subscription you can experience it right now. Indeed, a very different experience.
Used them for a while! They didn't seem to have prompt caching so I burnt through the daily 24M token limitations really quickly when doing large scale changes on a codebase (essentially a team's worth of menial migration/refactoring work). A lot of it was okay, but plenty had to be re-done and I still spotted some issues months down the line, in part I blame their model catalogue which did get an update to GLM 4.7 sometime way back, but definitely is showing its age: https://inference-docs.cerebras.ai/models/overview
Quality wise, Anthropic gives me the best results (Opus for almost everything, I make sub-agents with fresh context review its work, after 2-10 loops, usually finds most issues). Token amount wise for agentic work, DeepSeek V4 is up there. What Cerebras is doing pretty cool though, apparently they even have prompt caching now like the other big providers: https://inference-docs.cerebras.ai/capabilities/prompt-cachi... At the same time, producing bad code faster was annoying in a uniquely new way.
Wish they'd update the models with their subscription, it could genuinely be great with the proper harness. Like if they can run GLM 4.7, surely they could at least get DeepSeek V4 Flash with a big context window going as a starting point. How can you have so much money to make your own chips, but can't run modern models that you can get for free? It's like they don't want people to use their subscription.
Re: How fast is N tokens per second really?
#20Token/sec only makes sense once you tell me three four things:
1. decoding t/s, that is, when the model is generating text in the autoregressive fashion.
2. prefill t/s, that is, prompt processing speed.
3. What is the slope of those two numbers as the context size increases. An implementation that decodes at 50t/s with 2k context but decodes at 7t/s at 100k context is going to be a lot less useful that it seems at a first glance for a big number of real world use cases.
4. What's your use case? Reading a huge text and then having a small output like, fraud probability=12%? Or Reading a small question and generating a lot of text? This changes substantially if a model is usable based on its prefill/decoding speed.
For instance my DS4F inference on the DGX Spark does prefill at 350 t/s and at 200 t/s on already large contexts. But decodes at 13 t/s.
On the Mac Ultra the prefill is like 400 t/s and decoding 35 t/s.
The two systems can perform dramatically differently or almost the same based on the use case. In general for local inference to be acceptable, even if slow, you want at least 100 t/s prefill, at least 10 t/s generation. To be ok-ish from 200 to 400 t/s prefill, 15-25 t/s generation. To be a wonderful experience thousands of t/s prefill, 100 t/s generation.