Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

411–420 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#411

Post by rajbiswas125 »

he numbers being presented are deliberately misleading. On this model, Groq delivers around 1,300 tokens per second, whereas Cerebras achieves roughly 2,500 tokens per second.

With the next generation of Cerebras chips expected to be 5–7× faster, peak throughput could reach the ~17,500 tokens-per-second range. For smaller models like this, that level of performance is entirely realistic. So no, a general-purpose accelerator will likely continue to outperform a fixed-function ASIC with a specific model etched into it.

Moreover, we’re only looking at results from a two-year-old, relatively small model. We still don’t know how this architecture will scale with a large MoE model, especially given constraints like limited on-chip KV cache and more complex attention mechanisms.

The real test isn’t performance on a small benchmark model, it’s how the system handles large-scale, production-grade workloads under architectural constraints.

Re: The path to ubiquitous AI (17k tokens/sec)

#412

Post by rstuart4133 »

freakynit wrote: post #109

Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/

> It was literally in a blink of an eye.!!

It's not even close. It takes the eye 100mm .. 400ms to blink. This think takes under 30ms to process a small query - about 10 times faster.

Re: The path to ubiquitous AI (17k tokens/sec)

#413

Post by gwern »

dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

K-V caches are large, but hidden states aren't necessarily that large. And if you can run a model once ridiculously fast, then you can loop it repeatedly and still be fast. So I wonder about the 'modern RNNs' like RWKV here...

Re: The path to ubiquitous AI (17k tokens/sec)

#414

Post by CamperBob2 »

simlevesque wrote: post #301
rvz wrote: post #226

Earlier quoted context omitted.

Fast, but stupid. Me: "How many r's in strawberry?" Jimmy: There are 2 r's in "strawberry". Generated in 0.001s • 17,825 tok/s The question is not about how fast it is. The real question(s) are: 1. How is this worth it over diffusion LLMs (No mention of diffusion LLMs at all in this thread) (This also assumes that diffusion LLMs will get faster) 2. Will Talaas also work with reasoning models, especially those that ar…

LLMs can't count. They need tool use to answer these questions accurately.

That particular one can't count without using external tools. Others can, and do.

Re: The path to ubiquitous AI (17k tokens/sec)

#415

Post by flamedoge »

varispeed wrote: post #65
elternal_love wrote: post #59

Earlier quoted context omitted.

Were we go towards really smart roboters. It is interesting what kind of diferent model chips they can produce.

There is nothing smart about current LLMs. They just regurgitate text compressed in their memory based on probability. None of the LLMs currently have actual understanding of what you ask them to do and what they respond with.

I have a gut feeling, huge portion of deficiencies we note with AI is just reflection of the training data. For instance, wiki/reddit/etc internet is just a soup of human description of the world model, not the actual world model itself. There are gaps or holes in the knowledge because codified summary of world is what is remarkable to us humans, not a 100% faithful, comprehensive description of the world. What is obvious to us humans with lived real world experience often does not make it into the training data. A simple, demonstrable example is whether one should walk or drive to car wash.

Re: The path to ubiquitous AI (17k tokens/sec)

#416

Post by rustyhancock »

AlexC04 wrote: post #344

If I could have one of these cards in my own computer do you think it would be possible to replace claude code? 1. Assume It's running a better model, even a dedicated coding model. High scoring but obviously not opus 4.5 2. Instead of the standard send-receive paradigm we set up a pipeline of agents, each of whom parses the output of the previous. At 17k/tps running locally, you could effectively spin up tasks like…

It's 2.5kW so it likely won't sit in your computer (quite beyond what a desktop could provide in power alone to a single card, let alone cool). It's 8.5cm^2 which is a beast of a single die.

Basically logistically it's going to need to be in a data centre.

It's ideal for small context high throughput. Perhaps parsing huge text piles like if you had the entire Epstein files as text.

I think Claude code benefits from larger context to keep your entire project in view and deep reasoning.

What this would certainly replace is when Claude dispatched to Haiku for manual NLP tasks.

Re: The path to ubiquitous AI (17k tokens/sec)

#417

Post by pankajdoharey »

dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

There is nothing new here. This has been demonstrated several times by previous researchers:

https://arxiv.org/abs/2511.06174

https://arxiv.org/abs/2401.03868

For a real world use case, you would need an FPGA with terabytes of RAM. Perhaps it'll be a Off chip HBM. But for s large models, even that won't be enough. Then you would need to figure out NV-link like interconnect for these FPGAs. And we are back to square one.

Re: The path to ubiquitous AI (17k tokens/sec)

#418

Post by vntok »

kleiba wrote: post #375
vntok wrote: post #125

Earlier quoted context omitted.

Don't ask a small LLM about precise minutiae factual information. Alternatively, ask yourself how plausible it sounds that all the facts in the world could be compressed into 8k parameters while remaining intact and fine-grained. If your answer is that it sounds pretty impossible... well it is.

Did you see the part in my original post where it said "Not unexpected for an 8k model"?

Oh I saw it, you still have a fundamentally flawed comprehension of LLMs.

The size of the model does not factor as tiny models can use Internet to fetch factual information.

But you think they are accurate repositories of knowledge, even though it's physically impossible unless lossless infinite compression algorithms exist (they don't, can't and won't).

Re: The path to ubiquitous AI (17k tokens/sec)

#419

Post by robotpepi »

Herring wrote: post #409
Alifatisk wrote: post #302

What's happening in the comment section? How come so many cannot understand that his is running Llama 3.1 8B? Why are people judging its accuracy? It's almost a 2 years old 8B param model, why are people expecting to see Opus level response!? The focus here should be on the custom hardware they are producing and its performance, that is whats impressive. Imagine putting GLM-5 on this, that'd be insane. This reminds m…

If it's so easy to do custom silicon for any model (they say only 2 months), why didn't they demo one of the newer DeepSeek models instead? Using a 2-year model is so bad. I'm not buying it.

they explain it in the article: this is the first iteration, so they wanted to start with something simple, ie, this is a tech demo.
Post reply on HN