[deleted]
OpenAI unveils its first custom chip, built by Broadcom
31–40 of 496 posts
Re: OpenAI unveils its first custom chip, built by Broadcom
#32how much does this chip help with inference speed?
It's probably the same speed but cheaper.
Re: OpenAI unveils its first custom chip, built by Broadcom
#33I mean I'd love to be able to buy something like the 17k tps taalas chip as a pcie or m.2. Imagine when we can roar along at that speed, low power. Can just have the model reason for a while about anything and everything. It reminds me of the "race to idle" for mcus etc.
The current taalas chip is for a 3.1B param model. I’m hope so much that they can get that up to the 30B range. Just imagine Gemma 4 or Qwen 3.6 at 17k tps.
Re: OpenAI unveils its first custom chip, built by Broadcom
#34I mean I'd love to be able to buy something like the 17k tps taalas chip as a pcie or m.2. Imagine when we can roar along at that speed, low power. Can just have the model reason for a while about anything and everything. It reminds me of the "race to idle" for mcus etc.
> 17k tps taalas chip
It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually. And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases.
I wonder if it's being worked on in secret, if there's something about it that makes it infeasible, or if companies are really too nervous to lock in one model like that because the next one down the line could be a huge improvement. Re. infeasability, I have heard that the Taalas demonstration chip ran Llama 3.1 8B (a pretty horrible model) and that even that took a massive amount of transistors / die area. So it might just be the case that the good models are too big to fit on silicon?
Re: OpenAI unveils its first custom chip, built by Broadcom
#35Probably obvious but still omitted in the OpenAI post: chips are being made by TSMC [1]. Wasn't sure if Intel got it. 1. https://www.investing.com/news/stock-market-news/openai-unve...
I just read a claim on Twitter that the reason these companies (Google and Amazon as well as OpenAI) are using Broadcom isn't just for design expertise, but because Broadcom have allocation agreements in place with TSMC and the memory manufacturers.
Most design partners have allocation agreements. The thing is Broadcom is an absolute GIANT in the ASIC design space, and it's closest competitor Marvell is a fraction of it's size.
There are a lot of large tech companies that most of HN has never heard about that completely dominate entire segments.
Re: OpenAI unveils its first custom chip, built by Broadcom
#36I wonder how close OpenAI is getting to using the memory they purchased. Are they planning to stack a huge amount of HBM2 into these chips?
I assume OpenAI has been buying memory and "giving" it to Nvidia in exchange for a discount.
Re: OpenAI unveils its first custom chip, built by Broadcom
#37I mean I'd love to be able to buy something like the 17k tps taalas chip as a pcie or m.2. Imagine when we can roar along at that speed, low power. Can just have the model reason for a while about anything and everything. It reminds me of the "race to idle" for mcus etc.
> 17k tps taalas chip It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually . And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases. I wonder if it's being work…
Good models will require multiple Taalas chips but Groq and Cerebras also require a lot of chips and that hasn't stopped them.
Re: OpenAI unveils its first custom chip, built by Broadcom
#38This seems like more competition for Cerebras? Am I understanding correctly?
This is just an uncut wafer - I don't think it's intended to be a wafer-scale chip.
Cerebras etch memory onto the wafer alongside the processing elements, but AFAIK OpenAI are going to be using HBM memory and a conventional chiplet design.
Re: OpenAI unveils its first custom chip, built by Broadcom
#39I hope to see something like this, but in a small form factor like the NVIDIA spark. I want a super fast LLM that is Opus 4.6+, like, in ability.
Unfortunately Sam Altman won't be the one to deliver us at-home hardware that can run Opus-level models
Re: OpenAI unveils its first custom chip, built by Broadcom
#40We’ve entered the “if you care about software, build hardware” phase of AI