Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

191–200 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#191

Post by philipwhiuk »

rbanffy wrote: post #186
Aissen wrote: post #180

Earlier quoted context omitted.

> 880mm^2 die That's a lot of surface, isn't it? As big an M1 Ultra (2x M1 Max at 432mm² on TSMC N5P), a bit bigger than an A100 (820mm² on TSMC N7) or H100 (814mm² on TSMC N5). > The larger the die size, the lower the yield. I wonder if that applies? What's the big deal if a few parameter have a few bit flips?

> I wonder if that applies? What's the big deal if a few parameter have a few bit flips? We get into the sci-fi territory where a machine achieves sentience because it has all the right manufacturing defects. Reminds me of this https://en.wikipedia.org/wiki/A_Logic_Named_Joe

2000s movie line territory:

> There have always been ghosts in the machine. Random segments of code, that have grouped together to form unexpected protocols.

Re: The path to ubiquitous AI (17k tokens/sec)

#192

Post by asim »

Wow I'm impressed. I didn't actually think we'd see it encoded on chips. Or well I knew some layer of it could be, some sort of instruction set and chip design but this is pretty staggering. It opens the door to a lot of things. Basically it totally destroys the boundaries of where software will go but I also think we'll continue to see some generic chips show up that hit this performance soon enough. But the specialised chips with encoded models. This could be what ends up in specific places like cars, planes, robots, etc where latency matters. Maybe I'm out of the loop, I'm sure others and doing it including Google.

Re: The path to ubiquitous AI (17k tokens/sec)

#193

Post by scotty79 »

trentnix wrote: post #89

The speed of the chatbot's response is startling when you're used to the simulated fast typing of ChatGPT and others. But the Llama 3.1 8B model Taalas uses predictably results in incorrect answers, hallucinations, poor reliability as a chatbot. What type of latency-sensitive applications are appropriate for a small-model, high-throughput solution like this? I presume this type of specialization is necessary for robo…

Language translation, chunk by chunk.

Re: The path to ubiquitous AI (17k tokens/sec)

#194

Post by left-struck »

33a wrote: post #157

If they made a low power/mobile version, this could be really huge for embedded electronics. Mass produced, highly efficient "good enough" but still sort of dumb ais could put intelligence in house hold devices like toasters, light switches, and toilets. Truly we could be entering into the golden age of curses.

Oh god, this is the new version of every device having Bluetooth and an app and being called “smart”.

I just wanted some toast, but here I am installing an app, dismissing 10 popups, and maybe now arguing with a chat bot about how I don’t in fact want to turn on notifications.

Re: The path to ubiquitous AI (17k tokens/sec)

#195

Post by freakynit »

zardo wrote: post #151
trentnix wrote: post #89

The speed of the chatbot's response is startling when you're used to the simulated fast typing of ChatGPT and others. But the Llama 3.1 8B model Taalas uses predictably results in incorrect answers, hallucinations, poor reliability as a chatbot. What type of latency-sensitive applications are appropriate for a small-model, high-throughput solution like this? I presume this type of specialization is necessary for robo…

I'm wondering how much the output quality of a small model could be boosted by taking multiple goes at it. Generate 20 answers and feed them back through with a "rank these responses" prompt. Or doing something like MCTS.

Isn't this what thinking models do internally? Chain of thoughts?

Re: The path to ubiquitous AI (17k tokens/sec)

#196

Post by wat10000 »

xnorswap wrote: post #182

Asking it what its knowledge cut-off is interesting, it doesn't seem to be consistent even within a single response. Sometimes it responds to say it cuts off 2020 too. You What is your knowledge cut-off? Jimmy My knowledge cut-off is 2022, which means that my training data is current up to 2021, but I have been trained on a dataset that is updated periodically. If you have any specific questions about events or topic…

Sounds like they aren’t providing a system prompt which tells it what its knowledge cutoff is. It doesn’t know unless it’s told. Given the limited context, it makes sense they wouldn’t use up valuable space for that.

Re: The path to ubiquitous AI (17k tokens/sec)

#197

Post by andrewdea »

Etheryte wrote: post #139
freakynit wrote: post #109

Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/

It is incredibly fast, on that I agree, but even simple queries I tried got very inaccurate answers. Which makes sense, it's essentially a trade off of how much time you give it to "think", but if it's fast to the point where it has no accuracy, I'm not sure I see the appeal.

the hardwired model is Llama 3.1 8B, which is a lightweight model from two years ago. Unlike other models, it doesn't use "reasoning:" the time between question and answer is spent predicting the next tokens. It doesn't run faster because it uses less time to "think," It runs faster because its weights are hardwired into the chip rather than loaded from memory. A larger model running on a larger hardwired chip would run about as fast and get far more accurate results. That's what this proof of concept shows

Re: The path to ubiquitous AI (17k tokens/sec)

#198

Post by b0ner_t0ner »

freakynit wrote: post #109

Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/

I asked, “What are the newest restaurants in New York City?”

Jimmy replied with, “2022 and 2023 openings:”

0_0

Re: The path to ubiquitous AI (17k tokens/sec)

#200

Post by andy12_ »

Gud wrote: post #156
kaashif wrote: post #141

Earlier quoted context omitted.

If it's incredibly fast at a 2022 state of the art level of accuracy, then surely it's only a matter of time until it's incredibly fast at a 2026 level of accuracy.

Why do you assume this? I can produce total jibberish even faster, doesn’t mean I produce Einstein level thought if I slow down

Not what he said.
Post reply on HN