Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

211–220 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#211

Post by empath75 »

dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

An on-device reasoning model what that kind of speed and cost would completely change the way people use their computers. It would be closer to star trek than anything else we've ever had. You'd never have to type anything or use a mouse again.

Re: The path to ubiquitous AI (17k tokens/sec)

#212

Post by ashirviskas »

Zetaphor wrote: post #190
btown wrote: post #126

Earlier quoted context omitted.

For speculative decoding, wouldn’t this be of limited use for frontier models that don’t have the same tokenizer as Llama 3.1? Or would it be so good that retokenization/bridging would be worth it?

My understanding as well is that speculative decoding only works with a smaller quant of the same model. You're using the faster sampling of the smaller models representation of the larger models weights in order to attempt to accurately predict its token output. This wouldn't work cross-model as the token probabilities are completely different.

Smaller quant or smaller model?

Afaik it can work with anything, but sharing vocab solves a lot of headaches and the better token probs match, the more efficient it gets.

Which is why it is usually done with same family models and most often NOT just different quantizations of the same model.

Re: The path to ubiquitous AI (17k tokens/sec)

#213

Post by booli »

xnx wrote: post #204

Gemini Flash 2.5 lite does 400 tokens/sec. Is there benefit to going faster than a person can read?

Agents also "read", so yes there is. Think about spinning up 10, 20, 100 sub agents for a small task and they all return near instant. That's the usecase, not the chatbot.

Re: The path to ubiquitous AI (17k tokens/sec)

#214

Post by visarga »

varispeed wrote: post #65
elternal_love wrote: post #59

Earlier quoted context omitted.

Were we go towards really smart roboters. It is interesting what kind of diferent model chips they can produce.

There is nothing smart about current LLMs. They just regurgitate text compressed in their memory based on probability. None of the LLMs currently have actual understanding of what you ask them to do and what they respond with.

So you are saying they are like copy, LLMs will copy some training data back to you? Why do we spend so much money training and running them if they "just regurgitate text compressed in their memory based on probability"? billions of dollars to build a lossy grep.

I think you are confused about LLMs - they take in context, and that context makes them generate new things, for existing things we have cp. By your logic pianos can't be creative instruments because they just produce the same 88 notes.

Re: The path to ubiquitous AI (17k tokens/sec)

#215

Post by booli »

FieryTransition wrote: post #71

If it's not reprogrammable, it's just expensive glass. If you etch the bits into silicon, you then have to accommodate the bits by physical area, which is the transistor density for whatever modern process they use. This will give you a lower bound for the size of the wafers. This can give huge wafers for a very set model which is old by the time it is finalized. Etching generic functions used in ML and common fused…

Reading the in depth article also linked in this thread, they say that only 2 layers need to change most of the time. They claim from new model to PCB in 2 months. Let's see, but sounds promising.

Re: The path to ubiquitous AI (17k tokens/sec)

#216

Post by edot »

baalimago wrote: post #150

I've never gotten incorrect answers faster than this, wow! Jokes aside, it's very promising. For sure a lucrative market down the line, but definitely not for a model of size 8B. I think lower level intellect param amount is around 80B (but what do I know). Best of luck!

Yeah, two p’s in the word pepperoni …

Re: The path to ubiquitous AI (17k tokens/sec)

#217

Post by Etheryte »

andrewdea wrote: post #197
Etheryte wrote: post #139

Earlier quoted context omitted.

It is incredibly fast, on that I agree, but even simple queries I tried got very inaccurate answers. Which makes sense, it's essentially a trade off of how much time you give it to "think", but if it's fast to the point where it has no accuracy, I'm not sure I see the appeal.

the hardwired model is Llama 3.1 8B, which is a lightweight model from two years ago. Unlike other models, it doesn't use "reasoning:" the time between question and answer is spent predicting the next tokens. It doesn't run faster because it uses less time to "think," It runs faster because its weights are hardwired into the chip rather than loaded from memory. A larger model running on a larger hardwired chip would…

I see, that's very cool, that's the context I was missing, thanks a lot for explaining.

Re: The path to ubiquitous AI (17k tokens/sec)

#218

Post by adamtaylor_13 »

varispeed wrote: post #65
elternal_love wrote: post #59

Earlier quoted context omitted.

Were we go towards really smart roboters. It is interesting what kind of diferent model chips they can produce.

There is nothing smart about current LLMs. They just regurgitate text compressed in their memory based on probability. None of the LLMs currently have actual understanding of what you ask them to do and what they respond with.

If LLMs just regurgitate compressed text, they'd fail on any novel problem not in their training data. Yet, they routinely solve them, which means whatever's happening between input and output is more than retrieval, and calling it "not understanding" requires you to define understanding in a way that conveniently excludes everything except biological brains.
Post reply on HN