Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

331–340 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#331

Post by [deleted] »

cheema33 wrote: post #241
hagbard_c wrote: post #201

Holy Hallucinations, Batman... User You: What is the airspeed of an unladen swallow/ Jimmy: A reference to the classic line from Monty Python and the Holy Grail! Thus far it is correct but here it comes: In the movie, a group of medieval scholars, the Knights who say "Ni!" (also known as the Bridgekeepers), ask a king's servant, a blacksmith, if an unladen swallow makes good time. The blacksmith, played by John Clees…

> Holy Hallucinations, Batman... Congratulations! You figured out that this is a demo of a very small 8B model from 2022.

[deleted]

Re: The path to ubiquitous AI (17k tokens/sec)

#332

Post by mike_hearn »

wat10000 wrote: post #196
xnorswap wrote: post #182

Asking it what its knowledge cut-off is interesting, it doesn't seem to be consistent even within a single response. Sometimes it responds to say it cuts off 2020 too. You What is your knowledge cut-off? Jimmy My knowledge cut-off is 2022, which means that my training data is current up to 2021, but I have been trained on a dataset that is updated periodically. If you have any specific questions about events or topic…

Sounds like they aren’t providing a system prompt which tells it what its knowledge cutoff is. It doesn’t know unless it’s told. Given the limited context, it makes sense they wouldn’t use up valuable space for that.

The system prompt is configurable in the sidebar and minimal. It doesn't give a knowledge cutoff. This is a tech demo of the fact it works at all, it's not meant to be a good chatbot.

Re: The path to ubiquitous AI (17k tokens/sec)

#333

Post by robotnikman »

dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

Sounds perfect for use in consumer devices.

Re: The path to ubiquitous AI (17k tokens/sec)

#334

Post by PhunkyPhil »

IanCal wrote: post #300
condiment wrote: post #271

Earlier quoted context omitted.

At 16k tokens/s why bother routing? We're talking about multiple orders of magnitude faster and cheaper execution. Abundance supports different strategies. One approach: Set a deadline for a response, send the turn to every AI that could possibly answer, and when the deadline arrives, cancel any request that hasn't yet completed. You know a priori which models have the highest quality in aggregate. Pick that one.

The best coding model won’t be the best roleplay one which won’t be the best at tool use. It depends what you want to do in order to pick the best model.

I'm not saying you're wrong, but why is this the case?

I'm out of the loop on training LLMs, but to me it's just pure data input. Are they choosing to include more code rather than, say fiction books?

Re: The path to ubiquitous AI (17k tokens/sec)

#335

Post by PhunkyPhil »

zozbot234 wrote: post #78
dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

Low-latency inference is a huge waste of power; if you're going to the trouble of making an ASIC, it should be for dog-slow but very high throughput inference. Undervolt the devices as much as possible and use sub-threshold modes, multiple Vt and body biasing extensively to save further power and minimize leakage losses, but also keep working in fine-grained nodes to reduce areas and distances. The sensible goal is t…

I think it's really useful for agent to agent communication, as long as context loading doesn't become a bottleneck. Right now there can be noticeable delays under the hood, but at these speeds we'll never have to worry about latency when chain calling hundreds or thousands of agents in a network (I'm presuming this is going to take off in the future). Correct me if I'm wrong though.

Re: The path to ubiquitous AI (17k tokens/sec)

#336

Post by refulgentis »

PhunkyPhil wrote: post #334
IanCal wrote: post #300

Earlier quoted context omitted.

The best coding model won’t be the best roleplay one which won’t be the best at tool use. It depends what you want to do in order to pick the best model.

I'm not saying you're wrong, but why is this the case? I'm out of the loop on training LLMs, but to me it's just pure data input. Are they choosing to include more code rather than, say fiction books?

I’ll go ahead and say they’re wrong (source: building and maintaining llm client with llama.cpp integrated & 40+ 3p models via http)

I desperately want there to be differentiation. Reality has shown over and over again it doesn’t matter. Even if you do same query across X models and then some form of consensus, the improvements on benchmarks are marginal and UX is worse (more time, more expensive, final answer is muddied and bound by the quality of the best model)

Re: The path to ubiquitous AI (17k tokens/sec)

#339

Post by arjie »

This is incredible. With this speed I can use LLMs in a lot of pre-filtering etc. tasks. As a trivial example, I have a personal OpenClaw-like bot that I use to do a bunch of things. Some of the things just require it to do trivial tool-calling and tell me what's up. Things like skill or tool pre-filtering become a lot more feasible if they're always done.

Anyway, I imagine these are incredibly expensive, but if they ever sell them with Linux drivers and slotting into a standard PCIe it would be absolutely sick. At 3 kW that seems unlikely, but for that kind of speed I bet I could find space in my cabinet and just rip it. I just can't justify $300k, you know.

Post reply on HN