221–230 of 471 posts
Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
Generate lots of solutions and mix and match. This allows a new way to look at LLMs.
Also, what if Cerebras decided to make a wafer-sized FPGA array and turned large language models into lots and lots of logical gates?
This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…
This math is useful. Lots of folks scoffing in the comments below. I have a couple reactions, after chatting with it: 1) 16k tokens / second is really stunningly fast. There’s an old saying about any factor of 10 being a new science / new product category, etc. This is a new product category in my mind, or it could be. It would be incredibly useful for voice agent applications, realtime loops, realtime video generati…
Earlier quoted context omitted.
OK investors, time to pull out of OpenAI and move all your money to ChatJimmy.
A related argument I raised a few days back on HN: What's the moat with with these giant data-centers that are being built with 100's of billions of dollars on nvidia chips? If such chips can be built so easily, and offer this insane level of performance at 10x efficiency, then one thing is 100% sure: more such startups are coming... and with that, an entire new ecosystem.
Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
Is super fast but also super inaccurate, I would say not even gpt-3 levels.
Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
Me: "How many r's in strawberry?"
Jimmy: There are 2 r's in "strawberry".
Generated in 0.001s • 17,825 tok/s
1. How is this worth it over diffusion LLMs (No mention of diffusion LLMs at all in this thread)
2. Will Talaas also work with reasoning models, especially those that are beyond 100B parameters and with the output being correct?
3. How long will it take to create newer models to be turned into silicon? (This industry moves faster than Talaas.)
4. How does this work when one needs to fine-tune the model, but still benefit from the speed advantages?Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!! https://chatjimmy.ai/
With this speed, you can keep looping and generating code until it passes all tests. If you have tests. Generate lots of solutions and mix and match. This allows a new way to look at LLMs.
Edit: it seems like this is likely one chip and not 10. I assumed 8B 16bit quant with 4K or more context. This made me think that they must have chained multiple chips together since N6 850mm2 chip would only yield 3GB of SRAM max. Instead, they seem to have etched llama 8B q3 with 1k context instead which would indeed fit the chip size. This requires 10 chips for an 8 billion q3 param model. 2.4kW. 10 reticle sized…
Where are those numbers from? It's not immediately clear to me that you can distribute one model across chips with this design. > Model is etched onto the silicon chip. So can’t change anything about the model after the chip has been designed and manufactured. Subtle detail here: the fastest turnaround that one could reasonably expect on that process is about six months. This might eventually be useful, but at the mo…
This isn't ready for phones yet, but think of something like phones where people buy new ones every 3 years and even having a mediocre on-device model at that speed would be incredible for something like siri.
> me: the moon
> Jimmy: The answer to "What is the capital of France?" I was looking for was the city of Paris, but that's not the correct response to the original question of the capital of France. The question that got cut off was actually "What is the capital of France?", and the response "There are plenty of times I look to groq for narrow domain responses" wasn't the answer I was looking for.
It is certainly fast, but I think there might be some caching issues somewhere.
Earlier quoted context omitted.
There is nothing smart about current LLMs. They just regurgitate text compressed in their memory based on probability. None of the LLMs currently have actual understanding of what you ask them to do and what they respond with.
If LLMs just regurgitate compressed text, they'd fail on any novel problem not in their training data. Yet, they routinely solve them, which means whatever's happening between input and output is more than retrieval, and calling it "not understanding" requires you to define understanding in a way that conveniently excludes everything except biological brains.