Inference is crazy fast! I can see lot of potential for this kind of chip for IOT devices and Robotics.
The path to ubiquitous AI (17k tokens/sec)
311–320 of 471 posts
Re: The path to ubiquitous AI (17k tokens/sec)
#312I imagine how advantageous it would be to have something like llama.cpp encoded on a chip instead, allowing us to run more than a single model. It would be slower than Jimmy, for sure, but depending on the speed, it could be an acceptable trade-off.
Re: The path to ubiquitous AI (17k tokens/sec)
#313Earlier quoted context omitted.
OK investors, time to pull out of OpenAI and move all your money to ChatJimmy.
A related argument I raised a few days back on HN: What's the moat with with these giant data-centers that are being built with 100's of billions of dollars on nvidia chips? If such chips can be built so easily, and offer this insane level of performance at 10x efficiency, then one thing is 100% sure: more such startups are coming... and with that, an entire new ecosystem.
If I am not mistaken this chip was build specifically for the llama 8b model. Nvidia chips are general purpose.
Re: The path to ubiquitous AI (17k tokens/sec)
#314Performance like that may open the door to the strategy of brutefocing solutions to problems for which you have a verifier (problems such as decompilation).
Re: The path to ubiquitous AI (17k tokens/sec)
#315Earlier quoted context omitted.
If it's incredibly fast at a 2022 state of the art level of accuracy, then surely it's only a matter of time until it's incredibly fast at a 2026 level of accuracy.
Why do you assume this? I can produce total jibberish even faster, doesn’t mean I produce Einstein level thought if I slow down
Better models already exist, this is just proving you can dramatically increase inference speeds / reduce inference costs.
It isn't about model capability - it's about inference hardware. Same smarts, faster.
Re: The path to ubiquitous AI (17k tokens/sec)
#316What's happening in the comment section? How come so many cannot understand that his is running Llama 3.1 8B? Why are people judging its accuracy? It's almost a 2 years old 8B param model, why are people expecting to see Opus level response!? The focus here should be on the custom hardware they are producing and its performance, that is whats impressive. Imagine putting GLM-5 on this, that'd be insane. This reminds m…
I personally don't buy it, cerebras is way more advanced than this, comparing this tok/s to cerebras is disingenious.
Re: The path to ubiquitous AI (17k tokens/sec)
#317Imagine this thing for autocomplete.
I'm not sure how good llama 3.1 8b is for that, but it should work, right?
Autocomplete models don't have to be very big, but they gotta be fast.
Re: The path to ubiquitous AI (17k tokens/sec)
#318Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
Fast, but stupid. Me: "How many r's in strawberry?" Jimmy: There are 2 r's in "strawberry". Generated in 0.001s • 17,825 tok/s The question is not about how fast it is. The real question(s) are: 1. How is this worth it over diffusion LLMs (No mention of diffusion LLMs at all in this thread) (This also assumes that diffusion LLMs will get faster) 2. Will Talaas also work with reasoning models, especially those that ar…
The blog answers all those questions. It says they're working on fabbing a reasoning model this summer. It also says how long they think they need to fab new models, and that the chips support LoRAs and tweaking context window size.
I don't get these posts about ChatJimmy's intelligence. It's a heavily quantized Llama 3, using a custom quantization scheme because that was state of the art when they started. They claim they can update quickly (so I wonder why they didn't wait a few more months tbh and fab a newer model). Llama 3 wasn't very smart but so what, a lot of LLM use cases don't need smart, they need fast and cheap.
Also apparently they can run DeepSeek R1 also, and they have benchmarks for that. New models only require a couple of new masks so they're flexible.
Re: The path to ubiquitous AI (17k tokens/sec)
#319So they create a new chip for every model they want to support, is that right? Looking at that from 2026, when new large models are coming out every week, that seems troubling, but that's also a surface take. As many people here know better than I that a lot of the new models the big guys release are just incremental changes with little optimization going into how they're used, maybe there's plenty of room for a mode…
One of these things, however old, coupled with robust tool calling is a chip that could remain useful for decades. Baking in incremental updates of world knowledge isn't all that useful. It's kinda horrifying if you think about it, this chip among other things contains knowledge of Donald Trump encoded in silicon. I think this is a way cooler legacy for Melania than the movie haha.