121–130 of 471 posts
Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
New models come out, time to upgrade your AI card, etc.
This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…
Earlier quoted context omitted.
Where are those numbers from? It's not immediately clear to me that you can distribute one model across chips with this design. > Model is etched onto the silicon chip. So can’t change anything about the model after the chip has been designed and manufactured. Subtle detail here: the fastest turnaround that one could reasonably expect on that process is about six months. This might eventually be useful, but at the mo…
> The first generation HC1 chip is implemented in the 6 nanometer N6 process from TSMC. Each HC1 chip has 53 billion transistors on the package, most of it very likely for ROM and SRAM memory. The HC1 card burns about 200 watts, says Bajic, and a two-socket X86 server with ten HC1 cards in it runs 2,500 watts. https://www.nextplatform.com/2026/02/19/taalas-etches-ai-mod...
Earlier quoted context omitted.
You have a misunderstanding of what LLMs are good at.
Care to enlighten me?
Alternatively, ask yourself how plausible it sounds that all the facts in the world could be compressed into 8k parameters while remaining intact and fine-grained. If your answer is that it sounds pretty impossible... well it is.
This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…
This math is useful. Lots of folks scoffing in the comments below. I have a couple reactions, after chatting with it: 1) 16k tokens / second is really stunningly fast. There’s an old saying about any factor of 10 being a new science / new product category, etc. This is a new product category in my mind, or it could be. It would be incredibly useful for voice agent applications, realtime loops, realtime video generati…
A lot of naysayers in the comments, but there are so many uses for non-frontier models. The proof of this is in the openrouter activity graph for llama 3.1: https://openrouter.ai/meta-llama/llama-3.1-8b-instruct/activ... 10b daily tokens growing at an average of 22% every week . There are plenty of times I look to groq for narrow domain responses - these smaller models are fantastic for that and there's often no need…
LLM's have opened-up natural language interface to machines. This chip makes it realtime. And that opens a lot of use-cases.
Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
Earlier quoted context omitted.
Poverty of imagination here, plenty uses of this and its a prototype at this stage.
What uses, exactly? The prototype is: silicon with a Llama 3.1 8B etched into it. Today's 4B models already outperform it. Token rate in five digits is a major technical flex, but, does anyone really need to run a very dumb model at this speed? The only things that come to mind that could reap a benefit are: asymmetric exotics like VLA action policies and voice stages for V2V models. Both of which are "small fast low…
If you look at any development in computing, ASICs are the next step. It seems almost inevitable. Yes, it will always trail behind state of the art. But value will come quickly in a few generations.
Earlier quoted context omitted.
Were we go towards really smart roboters. It is interesting what kind of diferent model chips they can produce.
There is nothing smart about current LLMs. They just regurgitate text compressed in their memory based on probability. None of the LLMs currently have actual understanding of what you ask them to do and what they respond with.