I just looked up what my computer is capable of (m2 MacBook Air) and it says 15-35 tokens per second. I could live with that writing code with a local model.
How fast is N tokens per second really?
21–30 of 105 posts
Re: How fast is N tokens per second really?
#22This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
i really want a qwen on one of these chips: https://chatjimmy.ai 15k tokens/s would get me feeling like its actually worth splitting out worktrees to try several approaches to a problem
Why is that? It seems the other direction? I want to be sure I can complete a task in a certain amount of wall clock time. If the tokens per second are slow, then I am risking more by running a single approach at a time, and then have an incentive to try to multiplex my attention between separate work-streams. If the generation is fast enough to occupy my attention then there is no more available improvement by having parallel threads.
Re: How fast is N tokens per second really?
#23[flagged]
Re: How fast is N tokens per second really?
#24This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
Do you have ideas/suggestions for agentic workflows that only start making sense at such speeds?
Re: How fast is N tokens per second really?
#25We truly are in the dial up era of GenAI.
Re: How fast is N tokens per second really?
#26> Now switch between c and t at the same rate. The difference is striking — and intentional.
I don't see a big difference.
Re: How fast is N tokens per second really?
#27This is awesome!! I use Cursor and I've been trending towards medium thinking models as much as possible - I don't like the dev cadence with something like opus 4.7 (thinking: very high) (great for some tasks, like complex plans). Eventually I'd like to make my way to open models and open harness, and this tool or something like it could help me understand what performance I'd need for productive work - bookmarked!
Re: How fast is N tokens per second really?
#28This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
People seem to use these tools very differently from each other. I value intelligence over speed any day. My programs are written in Haskell, so there are rarely any tasks which require thousands and thousands of lines to solve. Just intelligence. If there are rote tasks, I want the LLM to help me find intelligent ways of automating it: the right abstraction, the right meta-programming technique.
I constantly push Opus and GPT, and they are getting better. But still have to do the hardest parts myself. I would not mind waiting 10-15 minutes for the right 20 lines of code!
Re: How fast is N tokens per second really?
#29I think your demo need more realistic thinking logs because thinking usually burns at least 2x to 3x of tokens of the code and for harder tasks much more.
Re: How fast is N tokens per second really?
#30This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
Google's 3.5 Flash – which came out yesterday – is 200-300 tokens/second (albeit purportedly inefficient in its use of reasoning tokens) and according to Google, 800-1500+ tokens/second on their 8i TPUs when they're out!
It's... suboptimal, but hopefully that's a reason to hope... if Google get themselves together for 3.5 Pro / the next Flash.