Gemini Flash 2.5 lite does 400 tokens/sec. Is there benefit to going faster than a person can read?
Agents already bypass human inference time, if it can input-output instantly it can also loop it generating near instantly long cached tasks
The path to ubiquitous AI (17k tokens/sec)
281–290 of 471 posts
Re: The path to ubiquitous AI (17k tokens/sec)
#282This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…
In 20$ a die, they could sell Gameboy style cartridges for different models.
Make them shaped like floppy disks to confuse the younger generations.
Re: The path to ubiquitous AI (17k tokens/sec)
#283Earlier quoted context omitted.
If LLMs just regurgitate compressed text, they'd fail on any novel problem not in their training data. Yet, they routinely solve them, which means whatever's happening between input and output is more than retrieval, and calling it "not understanding" requires you to define understanding in a way that conveniently excludes everything except biological brains.
Yes there are some fascinating emergent properties at play, but when they fail it's blatantly obvious that there's no actual intelligence nor understanding. They are very cool and very useful tools, I use them on a daily basis now and the way I can just paste a vague screenshot with some vague text and they get it and give a useful response blows my mind every time. But it's very clear that it's all just smoke and mi…
When humans fail a task, it’s obvious there is no actual intelligence nor understanding.
Intelligence is not as cool as you think it is.
Re: The path to ubiquitous AI (17k tokens/sec)
#284So they create a new chip for every model they want to support, is that right? Looking at that from 2026, when new large models are coming out every week, that seems troubling, but that's also a surface take. As many people here know better than I that a lot of the new models the big guys release are just incremental changes with little optimization going into how they're used, maybe there's plenty of room for a model-as-hardware model.
Which brings me to my second thing. We mostly pitch the AI wars as OpenAI vs Meta vs Claude vs Google vs etc. But another take is the war between open, locally run models and SaaS models, which really is about the war for general computing. Maybe a business model like this is a great tool to help keep general computing in the fight.
Re: The path to ubiquitous AI (17k tokens/sec)
#285I tried the chatbot. jarring to see a large response come back instantly at over 15k tok/sec I'll take one with a frontier model please, for my local coding and home ai needs..
Yeah, feeding that speed into a reasoning loop or a coding harness is going to revolutionize AI.
Re: The path to ubiquitous AI (17k tokens/sec)
#286Earlier quoted context omitted.
Yes there are some fascinating emergent properties at play, but when they fail it's blatantly obvious that there's no actual intelligence nor understanding. They are very cool and very useful tools, I use them on a daily basis now and the way I can just paste a vague screenshot with some vague text and they get it and give a useful response blows my mind every time. But it's very clear that it's all just smoke and mi…
When humans fail a task, it’s obvious there is no actual intelligence nor understanding. Intelligence is not as cool as you think it is.
I assure you, intelligence is very cool.
Re: The path to ubiquitous AI (17k tokens/sec)
#287Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
Fast, but stupid. Me: "How many r's in strawberry?" Jimmy: There are 2 r's in "strawberry". Generated in 0.001s • 17,825 tok/s The question is not about how fast it is. The real question(s) are: 1. How is this worth it over diffusion LLMs (No mention of diffusion LLMs at all in this thread) (This also assumes that diffusion LLMs will get faster) 2. Will Talaas also work with reasoning models, especially those that ar…
[dead]
Re: The path to ubiquitous AI (17k tokens/sec)
#288My concept was to do this with two pieces:
1. Generic, mask layers and board to handle what's common across models. Especially memory and interface.
2. Specific layers for the model implementation.
Masks are the most expensive part of ASIC design. So, keeping the custom part small with the rest pre-proven in silicon, even shared across companies, would drop the costs significantly. This is already done in hardware industry in many ways but not model acceleration.
Then, do 8B, 30-40B, 70B, and 405B models in hardware. Make sure they're RLHF-tuned well since changes will be impossible or limited. Prompts will drive most useful functionality. Keep cranking out chips. There's maybe a chance to keep the weights changeable on-chip but it should still be useful if only inputs can change.
The other concept is to use analog, neural networks with the analog layers on older, cheaper nodes. We only have to customize that per model. The rest is pre-built digital with standard interfaces on a modern node. Given the chips would be distributed, one might get away with 28nm for the shared part and develop it eith shuttle runs.
Re: The path to ubiquitous AI (17k tokens/sec)
#289Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
With this speed, you can keep looping and generating code until it passes all tests. If you have tests. Generate lots of solutions and mix and match. This allows a new way to look at LLMs.
Agreed, this is exciting, and has me thinking about completely different orchestrator patterns. You could begin to approach the solution space much more like a traditional optimization strategy such as CMA-ES. Rather than expect the first answer to be correct, you diverge wildly before converging.
Re: The path to ubiquitous AI (17k tokens/sec)
#290wow that is fast!