Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
I get nothing, no replies to anything.
Maybe hn and reddit crowd have overloaded them lol
The path to ubiquitous AI (17k tokens/sec)
131–140 of 471 posts
Re: The path to ubiquitous AI (17k tokens/sec)
#132Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
I dunno, it pretty quickly got stuck; the "attach file" didn't seem to work, and when I asked "can you see the attachment" it replied to my first message rather than my question.
Hmm.. I had tried simple chat converation without file attachments.
Re: The path to ubiquitous AI (17k tokens/sec)
#133This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…
Do not overlook traditional irrational investor exuberance, we've got an abundance of that right now. With the right PR manouveurs these guys could be a tulip craze.
Re: The path to ubiquitous AI (17k tokens/sec)
#134Earlier quoted context omitted.
This math is useful. Lots of folks scoffing in the comments below. I have a couple reactions, after chatting with it: 1) 16k tokens / second is really stunningly fast. There’s an old saying about any factor of 10 being a new science / new product category, etc. This is a new product category in my mind, or it could be. It would be incredibly useful for voice agent applications, realtime loops, realtime video generati…
The guts of a LLM isn't something I'm well versed in, but > to get the first N tokens sorted, only when the big model and small model diverge do you infer on the big model suggests there is something I'm unaware of. If you compare the small and big model, don't you have to wait for the big model anyway and then what's the point? I assume I'm missing some detail here, but what?
When you predict with the small model, the big model can verify as more of a batch and be more similar in speed to processing input tokens, if the predictions are good and it doesn't have to be redone.
Re: The path to ubiquitous AI (17k tokens/sec)
#135Earlier quoted context omitted.
Poverty of imagination here, plenty uses of this and its a prototype at this stage.
What uses, exactly? The prototype is: silicon with a Llama 3.1 8B etched into it. Today's 4B models already outperform it. Token rate in five digits is a major technical flex, but, does anyone really need to run a very dumb model at this speed? The only things that come to mind that could reap a benefit are: asymmetric exotics like VLA action policies and voice stages for V2V models. Both of which are "small fast low…
With LLMs this fast, you could imagine using them as any old function in programs.
Re: The path to ubiquitous AI (17k tokens/sec)
#136Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
OK investors, time to pull out of OpenAI and move all your money to ChatJimmy.
A related argument I raised a few days back on HN:
What's the moat with with these giant data-centers that are being built with 100's of billions of dollars on nvidia chips?
If such chips can be built so easily, and offer this insane level of performance at 10x efficiency, then one thing is 100% sure: more such startups are coming... and with that, an entire new ecosystem.
Re: The path to ubiquitous AI (17k tokens/sec)
#137Earlier quoted context omitted.
Were we go towards really smart roboters. It is interesting what kind of diferent model chips they can produce.
There is nothing smart about current LLMs. They just regurgitate text compressed in their memory based on probability. None of the LLMs currently have actual understanding of what you ask them to do and what they respond with.
We know that, but that does not make them unuseful. The opposite in fact, they are extremely useful in the hands of non-idiots.We just happen to have a oversupply of idiots at the moment, which AI is here to eradicate. /Sort of satire.
Re: The path to ubiquitous AI (17k tokens/sec)
#138Earlier quoted context omitted.
This math is useful. Lots of folks scoffing in the comments below. I have a couple reactions, after chatting with it: 1) 16k tokens / second is really stunningly fast. There’s an old saying about any factor of 10 being a new science / new product category, etc. This is a new product category in my mind, or it could be. It would be incredibly useful for voice agent applications, realtime loops, realtime video generati…
The guts of a LLM isn't something I'm well versed in, but > to get the first N tokens sorted, only when the big model and small model diverge do you infer on the big model suggests there is something I'm unaware of. If you compare the small and big model, don't you have to wait for the big model anyway and then what's the point? I assume I'm missing some detail here, but what?
I don't understand how it would work either, but it may be something similar to this: https://developers.openai.com/api/docs/guides/predicted-outp...
Re: The path to ubiquitous AI (17k tokens/sec)
#139Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
It is incredibly fast, on that I agree, but even simple queries I tried got very inaccurate answers. Which makes sense, it's essentially a trade off of how much time you give it to "think", but if it's fast to the point where it has no accuracy, I'm not sure I see the appeal.
Re: The path to ubiquitous AI (17k tokens/sec)
#140Earlier quoted context omitted.
Their 2.4 kW is for 10 chips it seems based on the next platform article. I assume they need all 10 chips for their 8B q3 model. Otherwise, they would have said so or they would have put a more impressive model as the demo. https://www.nextplatform.com/2026/02/19/taalas-etches-ai-mod...
It doesn’t make any sense to think you need the whole server to run one model. It’s much more likely that each server runs 10 instances of the model 1. It doesn’t make sense in terms of architecture. It’s one chip. You can’t split one model over 10 identical hardwire chips 2. It doesn’t add up with their claims of better power efficiency. 2.4kW for one model would be really bad.
We are both wrong.
First, it is likely one chip for llama 8B q3 with 1k context size. This could fit into around 3GB of SRAM which is about the theoretical maximum for TSMC N6 reticle limit.
Second, their plan is to etch larger models across multiple connected chips. It’s physically impossible to run bigger models otherwise since 3GB SRAM is about the max you can have on an 850mm2 chip.
followed by a frontier-class large language model running inference across a collection of HC cards by year-end under its HC2 architecture