try here, I hate llms but this is crazy fast. https://chatjimmy.ai/
"447 / 6144 tokens"
"Generated in 0.026s • 15,718 tok/s"
The path to ubiquitous AI (17k tokens/sec)
31–40 of 471 posts
Re: The path to ubiquitous AI (17k tokens/sec)
#32This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…
Don’t forget that the 8B model requires 10 of said chips to run.
And it’s a 3bit quant. So 3GB ram requirement.
If they run 8B using native 16bit quant, it will use 60 H100 sized chips.
Re: The path to ubiquitous AI (17k tokens/sec)
#33Edit: it seems like this is likely one chip and not 10. I assumed 8B 16bit quant with 4K or more context. This made me think that they must have chained multiple chips together since N6 850mm2 chip would only yield 3GB of SRAM max. Instead, they seem to have etched llama 8B q3 with 1k context instead which would indeed fit the chip size. This requires 10 chips for an 8 billion q3 param model. 2.4kW. 10 reticle sized…
Where are those numbers from? It's not immediately clear to me that you can distribute one model across chips with this design.
> Model is etched onto the silicon chip. So can’t change anything about the model after the chip has been designed and manufactured.
Subtle detail here: the fastest turnaround that one could reasonably expect on that process is about six months. This might eventually be useful, but at the moment it seems like the model churn is huge and people insist you use this week's model for best results.
Re: The path to ubiquitous AI (17k tokens/sec)
#34Edit: it seems like this is likely one chip and not 10. I assumed 8B 16bit quant with 4K or more context. This made me think that they must have chained multiple chips together since N6 850mm2 chip would only yield 3GB of SRAM max. Instead, they seem to have etched llama 8B q3 with 1k context instead which would indeed fit the chip size. This requires 10 chips for an 8 billion q3 param model. 2.4kW. 10 reticle sized…
ceo
Re: The path to ubiquitous AI (17k tokens/sec)
#35It's crazily fast. But 8B model is pretty much useless.
Anyway VCs will dump money onto them, and we'll see if the approach can scale to bigger models soon.
Re: The path to ubiquitous AI (17k tokens/sec)
#36try here, I hate llms but this is crazy fast. https://chatjimmy.ai/
The full answer pops in milliseconds, it's impressive and feels like a completely different technology just by foregoing the need to stream the output.
Re: The path to ubiquitous AI (17k tokens/sec)
#37This would be killer for exploring simultaneous thinking paths and council-style decision taking. Even with Qwen3-Coder-Next 80B if you could achieve a 10x speed, I'd buy one of those today. Can't wait to see if this is still possible with larger models than 8B.
It uses 10 chips for 8B model. It’d need 80 chips for an 80b model. Each chip is the size of an H100. So 80 H100 to run at this speed. Can’t change the model after you manufacture the chips since it’s etched into silicon.
I'm sure there is plenty of optimization paths left for them if they're a startup. And imho smaller models will keep getting better. And a great business model for people having to buy your chips for each new LLM release :)
Re: The path to ubiquitous AI (17k tokens/sec)
#38Re: The path to ubiquitous AI (17k tokens/sec)
#39[deleted]
Re: The path to ubiquitous AI (17k tokens/sec)
#40Earlier quoted context omitted.
It uses 10 chips for 8B model. It’d need 80 chips for an 80b model. Each chip is the size of an H100. So 80 H100 to run at this speed. Can’t change the model after you manufacture the chips since it’s etched into silicon.
I'm sure there is plenty of optimization paths left for them if they're a startup. And imho smaller models will keep getting better. And a great business model for people having to buy your chips for each new LLM release :)
One more thing. It seems like this is a Q3 quant. So only 3GB RAM requirement.
10 H100 chips for 3GB model.
I think it’s a niche of a niche at this point.
I’m not sure what optimization they can do since a transistor is a transistor.