Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

381–390 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#381

Post by dalenw »

AlexC04 wrote: post #344

If I could have one of these cards in my own computer do you think it would be possible to replace claude code? 1. Assume It's running a better model, even a dedicated coding model. High scoring but obviously not opus 4.5 2. Instead of the standard send-receive paradigm we set up a pipeline of agents, each of whom parses the output of the previous. At 17k/tps running locally, you could effectively spin up tasks like…

I think so. The last few months have shown us that it isn't necessarily the models themselves that provide good results, but the tooling / harness around it. Codex, Opus, GLM 5, Kimi 2.5, etc. all each have their quirks. Use a harness like opencode and give the model the right amount of context, they'll all perform well and you'll get a correct answer every time.

So in my opinion, in a scenario like this where the token output is near instant but you're running a lower tier model, good tooling can overcome the differences between a frontier cloud model.

Re: The path to ubiquitous AI (17k tokens/sec)

#382

Post by Animats »

Embedding the model at chip fab time ought to be useful for robotics, driving, vision, and audio applications, at least. The training sets are good for years.

So they use 3 bit values. Is that current thinking? LLMs started at 32-bit floats, and have gradually shrunk. 8-bit floats seem to work. Is 3 bits pushing it?

Re: The path to ubiquitous AI (17k tokens/sec)

#383

Post by g-mork »

gen220 wrote: post #367

This is genuinely an incredible proof-of-concept; the business implications of this demo to the AI labs and all the companies that derive a ton of profit from inference is difficult to understate, really. I think this is how I'm going to get my dream of Opus 3.7 running locally, quickly and cheaply on my mid-tier MacBook in 2030. Amazing. Anthropic et al will be able to make marginal revenue from licensing the weight…

I do like the idea of an aftermarket of ancient LLM chips that still have tons of useful life on text processing tasks etc. They don't talk about their architecture much, I wonder how well power can scale down. 200W for such a small model is not something I see happening in a laptop any time soon. Pretty hilarious implications for moat-building of the big providers too.

Re: The path to ubiquitous AI (17k tokens/sec)

#384

Post by Ancapistani »

robotpepi wrote: post #275
johnsimer wrote: post #249

Earlier quoted context omitted.

“ Models don’t get old as fast as they used to” ^^^ I think the opposite is true Anthropic and OpenAI are releasing new versions every 60-90 days it seems now, and you could argue they’re going to start releasing even faster

Are they becoming better at the same rate as before though?

Per release, I’d say no.

Per period of time, I’d say yes.

Re: The path to ubiquitous AI (17k tokens/sec)

#385

Post by tyushk »

aurareturn wrote: post #8

Edit: it seems like this is likely one chip and not 10. I assumed 8B 16bit quant with 4K or more context. This made me think that they must have chained multiple chips together since N6 850mm2 chip would only yield 3GB of SRAM max. Instead, they seem to have etched llama 8B q3 with 1k context instead which would indeed fit the chip size. This requires 10 chips for an 8 billion q3 param model. 2.4kW. 10 reticle sized…

Data tagging? 20k tok/s is at the point where I'd consider running an LLM on data from a column of a database, and these <=100 token problems provide the least chance of hallucination or stupidity.

Re: The path to ubiquitous AI (17k tokens/sec)

#386

Post by mips_avatar »

ACCount37 wrote: post #62
mips_avatar wrote: post #52

I think the thing that makes 8b sized models interesting is the ability to train unique custom domain knowledge intelligence and this is the opposite of that. Like if you could deploy any 8b sized model on it and be this fast that would be super interesting, but being stuck with llama3 8b isn't that interesting.

The "small model with unique custom domain knowledge" approach has a very low capability ceiling. Model intelligence is, in many ways, a function of model size. A small model tuned for a given domain is still crippled by being small. Some things don't benefit from general intelligence much. Sometimes a dumb narrow specialist really is all you need for your tasks. But building that small specialized model isn't easy o…

A fine tuned 1.7B model probably is still too crippled to do anything useful. But around 8b the capabilities really start to change. I’m also extremely unemployed right now so I can provide the engineering.

Re: The path to ubiquitous AI (17k tokens/sec)

#387

Post by amelius »

Animats wrote: post #382

Embedding the model at chip fab time ought to be useful for robotics, driving, vision, and audio applications, at least. The training sets are good for years. So they use 3 bit values. Is that current thinking? LLMs started at 32-bit floats, and have gradually shrunk. 8-bit floats seem to work. Is 3 bits pushing it?

If you're making your own chip, you might as well explore analog computation.

Re: The path to ubiquitous AI (17k tokens/sec)

#388

Post by merlindru »

soleveloper wrote: post #123
dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

In 20$ a die, they could sell Gameboy style cartridges for different models.

dude that would be so incredibly cool

Re: The path to ubiquitous AI (17k tokens/sec)

#389

Post by konaraddi »

> Taalas’ silicon Llama achieves 17K tokens/sec per user, nearly 10X faster than the current state of the art, while costing 20X less to build, and consuming 10X less power.

Insane gains, makes me excited for the future. Imagine Opus-like responses in I suspect power efficiency will be nearly entirely offset by increased usage but it’s more bang for watt.

Re: The path to ubiquitous AI (17k tokens/sec)

#390

Post by wmf »

Windchaser wrote: post #293
freakynit wrote: post #175

Earlier quoted context omitted.

lol... true that for now though

Yeah, just cause Cisco had a huge market lead on telecom in the late '90s, it doesn't mean they kept it. (And people nowadays: "Who's Cisco?")

They did mostly keep it though.
Post reply on HN