Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

401–410 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#401

Post by Sabinus »

Etheryte wrote: post #217
andrewdea wrote: post #197

Earlier quoted context omitted.

the hardwired model is Llama 3.1 8B, which is a lightweight model from two years ago. Unlike other models, it doesn't use "reasoning:" the time between question and answer is spent predicting the next tokens. It doesn't run faster because it uses less time to "think," It runs faster because its weights are hardwired into the chip rather than loaded from memory. A larger model running on a larger hardwired chip would…

I see, that's very cool, that's the context I was missing, thanks a lot for explaining.

I don't mean to be rude, but did you read the article before commenting?

Re: The path to ubiquitous AI (17k tokens/sec)

#402

Post by SkyPuncher »

jjcm wrote: post #112

A lot of naysayers in the comments, but there are so many uses for non-frontier models. The proof of this is in the openrouter activity graph for llama 3.1: https://openrouter.ai/meta-llama/llama-3.1-8b-instruct/activ... 10b daily tokens growing at an average of 22% every week . There are plenty of times I look to groq for narrow domain responses - these smaller models are fantastic for that and there's often no need…

I have such a deep need for something that's just a step above semantic search. These non-frontier models running blazingly fast can solve that.

So many problems simply don't require a full LLM, but more than traditional software. Training a novel model isn't really a compelling argument at most tech startups right now, so you need to find an LLM-native way to do things.

Re: The path to ubiquitous AI (17k tokens/sec)

#403

Post by SkyPuncher »

otabdeveloper4 wrote: post #354
baalimago wrote: post #150

I've never gotten incorrect answers faster than this, wow! Jokes aside, it's very promising. For sure a lucrative market down the line, but definitely not for a model of size 8B. I think lower level intellect param amount is around 80B (but what do I know). Best of luck!

Make it for Qwen 2.5 and I'd buy it. You don't actually need "frontier models" for Real Work (c). (Summarization, classification and the rest of the usual NLP suspects.)

I completely agree. So many things can benefit from having "smart classifiers".

Like, give me semantic search that can detect the difference between SSL and TLS without needing to put a full LLM in the loop.

Re: The path to ubiquitous AI (17k tokens/sec)

#404

Post by awwaiid »

mncharity wrote: post #398

There's an old idea of adaptive media. Imagine a video drama that's composed of a graph of clips, like an old "choose your own adventure" book ("Do you X? If yes, goto page 45"). With gaze tracking, one can "hmm, the viewer is more focused on character A than B... so we'll give clips and subplots with more A". Now, when reading, the eye moves in little jumps - saccades. They last 10's of ms, the eye is blind during t…

Generative TikTok for words

Re: The path to ubiquitous AI (17k tokens/sec)

#405

Post by trollbridge »

This is, so far, utterly charming. I made a simple prompt of "make an adventure game in the style of cia.bas from pc-sig". It ended up being wildly different than that, but 30 minutes later and I'm still busy trying to play this "game" it fabricated out of thin air.

One interesting thing is it keeps randomly emitting "ประก" (meaning "Announcement") and chartInstance. This is recalling the early days of GPT-2 when the light bulb went on that "hey, there's something groundbreaking here".

Re: The path to ubiquitous AI (17k tokens/sec)

#406

Post by g-mork »

gen220 wrote: post #400
g-mork wrote: post #383

Earlier quoted context omitted.

I do like the idea of an aftermarket of ancient LLM chips that still have tons of useful life on text processing tasks etc. They don't talk about their architecture much, I wonder how well power can scale down. 200W for such a small model is not something I see happening in a laptop any time soon. Pretty hilarious implications for moat-building of the big providers too.

Yea I mean this is the first publishable draft of a startup cooking on this. I'm confident there are at least 1-2 OOMs of improvement to come here in terms of the (intelligence : wattage) ratio. I really thought we were going to need to see a couple of dramatic OOM-improvement changes to the model composition / software layer, in order to get models of Opus 3.7's capability running on our laptops. This release tells…

The way I imagine it in 2-4 years we're going to be hit with a triple glut of better architecture, massive oversupply of hardware and potentially one or two hardware efforts like this really taking off. It's pretty crazy we're already 4 years in and outside of very niche / low availability solutions, it's still either GPU or bust

Re: The path to ubiquitous AI (17k tokens/sec)

#407

Post by oofbey »

I have a hard time reading beyond factual lies like:

> On the cost front, deploying modern models demands massive engineering and capital: room-sized supercomputers consuming hundreds of kilowatts…

This is just wrong. The largest models are probably 1-2 trillion parameters. Say 2 trillion and let’s pretend it’s only quantized to 8bit (even though it could easily be half that.) So we need 2TB of VRAM. Not even using the latest hardware, lets say H100 chips with 80GB of vram each, with 8 of them in say an 8U. (Although you can certainly fit these in 6U still air cooled or even 4U water cooled.) Three of these server would almost do, but let’s call it four to include plenty of room for context. The largest physical size would be 32U - most of a single rack. Which is hardly the size of a room, even in Manhattan. Total power maybe 40kW. And you could easily drop these numbers to a half or quarter of that with reasonable modifications or upgrades.

If you want to sell your hardware, start by being honest about the problem you’re addressing.

Re: The path to ubiquitous AI (17k tokens/sec)

#408

Post by alfalfasprout »

himata4113 wrote: post #316
Alifatisk wrote: post #302

What's happening in the comment section? How come so many cannot understand that his is running Llama 3.1 8B? Why are people judging its accuracy? It's almost a 2 years old 8B param model, why are people expecting to see Opus level response!? The focus here should be on the custom hardware they are producing and its performance, that is whats impressive. Imagine putting GLM-5 on this, that'd be insane. This reminds m…

I personally don't buy it, cerebras is way more advanced than this, comparing this tok/s to cerebras is disingenious.

Cerebras is a totally different product though. They can (theoretically) run any frontier model provided it gets compiled a certain way. Like a wafer scale TPU.

This is using hardwired weights with on-die SRAM used for K/V for example. It's WAY more power efficient and faster. The tradeoff being it's hardwired.

Still, most frontier models are "good enough" where an obscenely fast version would be a major seller.

Re: The path to ubiquitous AI (17k tokens/sec)

#409

Post by Herring »

Alifatisk wrote: post #302

What's happening in the comment section? How come so many cannot understand that his is running Llama 3.1 8B? Why are people judging its accuracy? It's almost a 2 years old 8B param model, why are people expecting to see Opus level response!? The focus here should be on the custom hardware they are producing and its performance, that is whats impressive. Imagine putting GLM-5 on this, that'd be insane. This reminds m…

If it's so easy to do custom silicon for any model (they say only 2 months), why didn't they demo one of the newer DeepSeek models instead? Using a 2-year model is so bad. I'm not buying it.

Re: The path to ubiquitous AI (17k tokens/sec)

#410

Post by ssivark »

vessenes wrote: post #85
dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

This math is useful. Lots of folks scoffing in the comments below. I have a couple reactions, after chatting with it: 1) 16k tokens / second is really stunningly fast. There’s an old saying about any factor of 10 being a new science / new product category, etc. This is a new product category in my mind, or it could be. It would be incredibly useful for voice agent applications, realtime loops, realtime video generati…

> speculative decoding for bread and butter frontier models. The thing that I’m really very skeptical of is the 2 month turnaround. To get leading edge geometry turned around on arbitrary 2 month schedules is .. ambitious

Can we use older (previous generation, smaller) models as a speculative decoder for the current model? I don't know whether the randomness in training (weight init, data ordering, etc) will affect this kind of use. To the extent that these models are learning the "true underlying token distribution" this should be possible, in principle. If that's the case, speculative decoding is an elegant vector to introduce this kind of tech, and the turnaround time is even less of a problem.

Post reply on HN