Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

261–270 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#261

Post by noveltyaccount »

soleveloper wrote: post #123
dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

In 20$ a die, they could sell Gameboy style cartridges for different models.

That would be very cool, get an upgraded model every couple of months. Maybe PCIe form factor.

Re: The path to ubiquitous AI (17k tokens/sec)

#262

Post by spot5010 »

jjcm wrote: post #112

A lot of naysayers in the comments, but there are so many uses for non-frontier models. The proof of this is in the openrouter activity graph for llama 3.1: https://openrouter.ai/meta-llama/llama-3.1-8b-instruct/activ... 10b daily tokens growing at an average of 22% every week . There are plenty of times I look to groq for narrow domain responses - these smaller models are fantastic for that and there's often no need…

These seem ideal for robotics applications, where there is a low-latency narrow use case path that these chips can serve, maybe locally.

Re: The path to ubiquitous AI (17k tokens/sec)

#263

Post by thrance »

aurareturn wrote: post #51
thrance wrote: post #43

Earlier quoted context omitted.

> What is a task that is extremely high value, only require a small model intelligence, require tremendous speed, is ok to run on a cloud due to power requirements, AND will be used for years without change since the model is etched into silicon? Video game NPCs?

Doesn’t pass the high value and require tremendous speed tests.

Video games are a huge market, and speed and cost of current models are definitely huge barriers to integrating LLMs in video games.

Re: The path to ubiquitous AI (17k tokens/sec)

#264

Post by xnx »

atls wrote: post #256
xnx wrote: post #204

Gemini Flash 2.5 lite does 400 tokens/sec. Is there benefit to going faster than a person can read?

There is also the use case of delegating tasks programmatically to an LLM, for example, transforming unstructured data to structured data. This task often can’t be done reliably without either 1. lots of manual work, or 2. intelligence, especially when the structure of the individual data pieces are unknown. Problems like these can be much more efficiently solved by LLMs, and if you imagine these programs are process…

Aren't such tasks inherently parrallelizable?

Re: The path to ubiquitous AI (17k tokens/sec)

#265

Post by brainless »

I know it is not easy to see the benefits of small models easily but this is what I am building for (1). I created a product for Google Gemini 3 Hackathon and I used Gemini 3 Flash (2). I tested locally using Ministral 3B and it was promising. Definitely will need work. But 8B/14B may give awesome results.

I am building a data extraction software on top of emails, attachments, cloud/local files. I use a reverse template generation with only variable translation done by LLMs (3). Small models are awesome for this (4).

I just applied for API access. If privacy policies are a fit, I would love to enable this for MVP launch.

1. https://github.com/brainless/dwata

2. https://youtu.be/Uhs6SK4rocU

3. https://github.com/brainless/dwata/tree/feature/reverse-temp...

4. https://github.com/brainless/dwata/tree/feature/reverse-temp...

Re: The path to ubiquitous AI (17k tokens/sec)

#267

Post by anthonypasq »

scosman wrote: post #159
gwd wrote: post #121

Earlier quoted context omitted.

I dunno, it pretty quickly got stuck; the "attach file" didn't seem to work, and when I asked "can you see the attachment" it replied to my first message rather than my question.

It’s llama 3.1 8B. No vision, not smart. It’s just a technical demo.

why is everyone seemingly incapable of understanding this? waht is going on here? Its like ai doomers consistently have the foresight of a rat. yeah no shit it sucks its running llama 3 8b, but theyre completely incapable of extrapolation.

Re: The path to ubiquitous AI (17k tokens/sec)

#268

Post by tgsovlerkhgsel »

Their "chat jimmy" demo sure is fast, but it's not useful at all.

Test prompt: ```

Please classify the sentiment of this post as "positive", "neutral" or "negative":

Given the price, I expected very little from this case, and I was 100% right.

``` Jimmy: Neutral.

I tried various other examples that I had successfully "solved" with very early LLMs and the results were similarly bad.

Re: The path to ubiquitous AI (17k tokens/sec)

#269

Post by anthonypasq »

Derbasti wrote: post #185
baalimago wrote: post #150

I've never gotten incorrect answers faster than this, wow! Jokes aside, it's very promising. For sure a lucrative market down the line, but definitely not for a model of size 8B. I think lower level intellect param amount is around 80B (but what do I know). Best of luck!

Amazing! It couldn't answer my question at all, but it couldn't answer it incredibly quickly! Snarky, but true. It is truly astounding, and feels categorically different. But it's also perfectly useless at the moment. A digital fidget spinner.

does no one understand what a tech demo is anymore? do you think this piece of technology is just going to be frozen in time at this capability for eternity?

do you have the foresight of a nematode?

Post reply on HN