31–40 of 105 posts
This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
Do you have ideas/suggestions for agentic workflows that only start making sense at such speeds?
Interesting. It seems to me that with that speed (20-30) on local hardware the real issue is quality of output, not tokens per sec.
It really depends. With the new "thinking" models they usually spend some time before writing the final answer. If they "think" for 1k tokens, that's a minute of spinning wheel you're gonna see for each question. Add that to the prompt processing, and diminishing speeds as context increases, and it becomes really slow for longer sessions.
Token/sec only makes sense once you tell me three four things: 1. decoding t/s, that is, when the model is generating text in the autoregressive fashion. 2. prefill t/s, that is, prompt processing speed. 3. What is the slope of those two numbers as the context size increases. An implementation that decodes at 50t/s with 2k context but decodes at 7t/s at 100k context is going to be a lot less useful that it seems at a…
This is great. Agentic coding at 600+ tokens/sec is going to be a radically different beast. Coming soon-ish?
People seem to use these tools very differently from each other. I value intelligence over speed any day. My programs are written in Haskell, so there are rarely any tasks which require thousands and thousands of lines to solve. Just intelligence. If there are rote tasks, I want the LLM to help me find intelligent ways of automating it: the right abstraction, the right meta-programming technique. I constantly push Op…
I think your demo need more realistic thinking logs because thinking usually burns at least 2x to 3x of tokens of the code and for harder tasks much more.
On the other hand, I've been using Mimo and Minimax a lot recently. They routinely reach 100-150 tokens per second and that feels too fast, to the point where it's hard to keep up with what it's actually doing. Great for subagents though.