Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

241–250 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#241

Post by cheema33 »

hagbard_c wrote: post #201

Holy Hallucinations, Batman... User You: What is the airspeed of an unladen swallow/ Jimmy: A reference to the classic line from Monty Python and the Holy Grail! Thus far it is correct but here it comes: In the movie, a group of medieval scholars, the Knights who say "Ni!" (also known as the Bridgekeepers), ask a king's servant, a blacksmith, if an unladen swallow makes good time. The blacksmith, played by John Clees…

> Holy Hallucinations, Batman...

Congratulations! You figured out that this is a demo of a very small 8B model from 2022.

Re: The path to ubiquitous AI (17k tokens/sec)

#242

Post by PlatoIsADisease »

baalimago wrote: post #150

I've never gotten incorrect answers faster than this, wow! Jokes aside, it's very promising. For sure a lucrative market down the line, but definitely not for a model of size 8B. I think lower level intellect param amount is around 80B (but what do I know). Best of luck!

As someone with a 3060, I can attest that there are really really good 7-9B models. I still use berkeley-nest/Starling-LM-7B-alpha and that model is a few years old.

If we are going for accuracy, the question should be asked multiple times on multiple models and see if there is agreement.

But I do think once you hit 80B, you can struggle to see the difference between SOTA.

That said, GPT4.5 was the GOAT. I can't imagine how expensive that one was to run.

Re: The path to ubiquitous AI (17k tokens/sec)

#243

Post by Epskampie »

smusamashah wrote: post #221
freakynit wrote: post #109

Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/

With this speed, you can keep looping and generating code until it passes all tests. If you have tests. Generate lots of solutions and mix and match. This allows a new way to look at LLMs.

And then it's slow again to finally find a correct answer...

Re: The path to ubiquitous AI (17k tokens/sec)

#244

Post by PlatoIsADisease »

freakynit wrote: post #109

Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/

Well it got all 10 incorrect when I asked for top 10 catchphrases from a character in Plato's books. It confused the baddie for Socrates.

Re: The path to ubiquitous AI (17k tokens/sec)

#245

Post by xnx »

cheema33 wrote: post #234
xnx wrote: post #204

Gemini Flash 2.5 lite does 400 tokens/sec. Is there benefit to going faster than a person can read?

Yes. You can allow multiple people to use a single chip. A slower solution will be able to service far fewer users.

Right, but it is also possible it's cheaper to use 42 Google TPUs for a second than one of these.

Re: The path to ubiquitous AI (17k tokens/sec)

#246

Post by sowbug »

rbanffy wrote: post #186
Aissen wrote: post #180

Earlier quoted context omitted.

> 880mm^2 die That's a lot of surface, isn't it? As big an M1 Ultra (2x M1 Max at 432mm² on TSMC N5P), a bit bigger than an A100 (820mm² on TSMC N7) or H100 (814mm² on TSMC N5). > The larger the die size, the lower the yield. I wonder if that applies? What's the big deal if a few parameter have a few bit flips?

> I wonder if that applies? What's the big deal if a few parameter have a few bit flips? We get into the sci-fi territory where a machine achieves sentience because it has all the right manufacturing defects. Reminds me of this https://en.wikipedia.org/wiki/A_Logic_Named_Joe

Also see Adrian Thompson's Xilinx 6200 FPGA, programmed by a genetic algorithm that worked but exploited nuances unique to that specific physical chip, meaning the software couldn't be copied to another chip. https://news.ycombinator.com/item?id=43152877

Re: The path to ubiquitous AI (17k tokens/sec)

#247

Post by notsylver »

I always thought eventually someone would come along and make a hardware accelerator for LLMs, but I thought it would be like google TPUs where you can load up whatever model you want. Baking the model into hardware sounds like the monkey paw curled, but it might be interesting selling an old.. MPU..? because it wasn't smart enough for your latest project

Re: The path to ubiquitous AI (17k tokens/sec)

#248

Post by mikhail-ramirez »

dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

Yea its fast af but very quickly loses context/hallucinates from my own tests with large chunks of text

Re: The path to ubiquitous AI (17k tokens/sec)

#249

Post by johnsimer »

audunw wrote: post #160
FieryTransition wrote: post #71

If it's not reprogrammable, it's just expensive glass. If you etch the bits into silicon, you then have to accommodate the bits by physical area, which is the transistor density for whatever modern process they use. This will give you a lower bound for the size of the wafers. This can give huge wafers for a very set model which is old by the time it is finalized. Etching generic functions used in ML and common fused…

Models don’t get old as fast as they used to. A lot of the improvements seem to go into making the models more efficient, or the infrastructure around the models. If newer models mainly compete on efficiency it means you can run older models for longer on more efficient hardware while staying competitive. If power costs are significantly lower, they can pay for themselves by the time they are outdated. It also means…

“ Models don’t get old as fast as they used to”

^^^ I think the opposite is true

Anthropic and OpenAI are releasing new versions every 60-90 days it seems now, and you could argue they’re going to start releasing even faster

Post reply on HN