Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

451–460 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#451

Post by bigcat12345678 »

I am thinking if this can be a low-level substrate for composing dumb LLMs into smart swarm, theoretically:

1. A whole with disparate parts (smart and dumb components) are almost always more cost-effective to reach a given target of performance 2. With that, a whole with disparate parts, are almost always more performant with the same cost

A few inspiration:

1. Human body is intelligent composed of so diverse parts 2. Swarm intelligence of insects and small animals are certainly beyond current understanding

The cost and speed of this thing is on point to make such a whole composed diverse parts possible.

Re: The path to ubiquitous AI (17k tokens/sec)

#452

Post by akie »

Herring wrote: post #441
akie wrote: post #427

Earlier quoted context omitted.

Why so negative lol. The speed and very reduced power use of this thing are nothing to be sneezed at. I mean, hardware accelerated LLMs are a huge step forward. But yeah, this is a proof of concept, basically. I wouldn't be surprised if the size factor and the power use go down even more, and that we'll start seeing stuff like this in all kinds of hardware. It's an enabler.

You don't know. You just have marketing materials, not independent analysis. Maybe it actually takes 2 years to design and manufacture the hardware, so anything that comes out will be badly out of date. Wouldn't be the first time someone lied. A good demo backed by millions of dollars should not allow such doubts.

Did you not see the chatbot they posted online (https://chatjimmy.ai/)? That thing is near instantaneous, it's all the proof you need that this is real.

And if the hardware is real and functional, as you can independently verify by chatting with that thing, how much more effort would it be to etch more recent models?

The real question is of course: what about LARGER models? I'm assuming you can apply some of the existing LLM inference parallelization techniques and split the workload over multiple cards. Some of the 32B models are plenty powerful.

It's a proof of concept, and a convincing one.

Re: The path to ubiquitous AI (17k tokens/sec)

#453

Post by IanCal »

refulgentis wrote: post #336
PhunkyPhil wrote: post #334

Earlier quoted context omitted.

I'm not saying you're wrong, but why is this the case? I'm out of the loop on training LLMs, but to me it's just pure data input. Are they choosing to include more code rather than, say fiction books?

I’ll go ahead and say they’re wrong (source: building and maintaining llm client with llama.cpp integrated & 40+ 3p models via http) I desperately want there to be differentiation. Reality has shown over and over again it doesn’t matter. Even if you do same query across X models and then some form of consensus, the improvements on benchmarks are marginal and UX is worse (more time, more expensive, final answer is mud…

Are you saying I’m wrong that some models are better for some tasks than others, but there isn’t a universally best model for all tasks?

Re: The path to ubiquitous AI (17k tokens/sec)

#454

Post by fennecbutt »

soleveloper wrote: post #123
dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

In 20$ a die, they could sell Gameboy style cartridges for different models.

Microsoft

Re: The path to ubiquitous AI (17k tokens/sec)

#455

Post by fennecbutt »

adamtaylor_13 wrote: post #218
varispeed wrote: post #65

Earlier quoted context omitted.

There is nothing smart about current LLMs. They just regurgitate text compressed in their memory based on probability. None of the LLMs currently have actual understanding of what you ask them to do and what they respond with.

If LLMs just regurgitate compressed text, they'd fail on any novel problem not in their training data. Yet, they routinely solve them, which means whatever's happening between input and output is more than retrieval, and calling it "not understanding" requires you to define understanding in a way that conveniently excludes everything except biological brains.

I somewhat agree with you but I also realise that there are very few "novel" problems in the world. I think it's really just more complex problem spaces is all.

Same relative logic, just more of it/more steps or trials.

Re: The path to ubiquitous AI (17k tokens/sec)

#456

Post by fennecbutt »

rvz wrote: post #226
freakynit wrote: post #109

Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/

Fast, but stupid. Me: "How many r's in strawberry?" Jimmy: There are 2 r's in "strawberry". Generated in 0.001s • 17,825 tok/s The question is not about how fast it is. The real question(s) are: 1. How is this worth it over diffusion LLMs (No mention of diffusion LLMs at all in this thread) (This also assumes that diffusion LLMs will get faster) 2. Will Talaas also work with reasoning models, especially those that ar…

The counting rs in strawberry problem was a example of people not understanding how the models work but I guess good to show the limitations of the current architectures.

But thing is, those architectures haven't improved a whole lot. Now when it answers that correctly it's either in training data or by virtue of "count letters" or code sandbox tools.

Re: The path to ubiquitous AI (17k tokens/sec)

#457

Post by make3 »

dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

It's weird to me to train such huge models to then destroy them by using them a 3 bits quantization per presumably 16bits (bfloat16) weights. Why not just train smaller models then.

Re: The path to ubiquitous AI (17k tokens/sec)

#458

Post by real-hacker »

Alifatisk wrote: post #302

What's happening in the comment section? How come so many cannot understand that his is running Llama 3.1 8B? Why are people judging its accuracy? It's almost a 2 years old 8B param model, why are people expecting to see Opus level response!? The focus here should be on the custom hardware they are producing and its performance, that is whats impressive. Imagine putting GLM-5 on this, that'd be insane. This reminds m…

They support Lora, it is something.

Re: The path to ubiquitous AI (17k tokens/sec)

#459

Post by rustyhancock »

runeks wrote: post #425
rustyhancock wrote: post #416

Earlier quoted context omitted.

It's 2.5kW so it likely won't sit in your computer (quite beyond what a desktop could provide in power alone to a single card, let alone cool). It's 8.5cm^2 which is a beast of a single die. Basically logistically it's going to need to be in a data centre. It's ideal for small context high throughput. Perhaps parsing huge text piles like if you had the entire Epstein files as text. I think Claude code benefits from l…

> It's 2.5kW so it likely won't sit in your computer (quite beyond what a desktop could provide in power alone to a single card, let alone cool). It's 8.5cm^2 which is a beast of a single die. I wonder how you cool a 3x3cm die that outputs 2.5 kW of heat. In the article they mention that the traditional setup requires water cooling, but surely this does as well, right?

Can't imagine what else could manage that nearly 2.8W/mm2.

It does make you wonder if they copy is misleading about something so simple how much else could be puffery?

Maybe they mean that a standard liquid cooling system will work?

Re: The path to ubiquitous AI (17k tokens/sec)

#460

Post by stephenbez »

refulgentis wrote: post #336
PhunkyPhil wrote: post #334

Earlier quoted context omitted.

I'm not saying you're wrong, but why is this the case? I'm out of the loop on training LLMs, but to me it's just pure data input. Are they choosing to include more code rather than, say fiction books?

I’ll go ahead and say they’re wrong (source: building and maintaining llm client with llama.cpp integrated & 40+ 3p models via http) I desperately want there to be differentiation. Reality has shown over and over again it doesn’t matter. Even if you do same query across X models and then some form of consensus, the improvements on benchmarks are marginal and UX is worse (more time, more expensive, final answer is mud…

Thanks. Are there any links where I can learn more about this?

I did some Googling and it appears that there are some examples where people say combining multiple models or multiple runs of the same models leads to improvements: https://www.sciencedirect.com/science/article/abs/pii/S00104... https://arxiv.org/abs/2203.11171

But presumably people are less likely to publish a paper when an approach doesn’t work.

Post reply on HN