OpenAI unveils its first custom chip, built by Broadcom • Hacker News Board

Re: OpenAI unveils its first custom chip, built by Broadcom

#461

Post by ipdashc » Thu, Jun 25, 2026, 2:17 PM UTC

coder543 wrote: post #358

ipdashc wrote: post #34

Earlier quoted context omitted.

> 17k tps taalas chip It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually . And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases. I wonder if it's being work…

> It's odd to me that I haven't heard anything about this approach since. It has only been four months since they unveiled their first prototype. I don't understand your confusion. Chip development does not happen overnight...? Their initial blog post laid out a roadmap, so theoretically they should have another thing to demonstrate this summer.

In the sense of interested customers, online discussion, other companies doing the same thing, etc. Of course it takes time to get actual results, but from an outsider's perspective it's surprising that it was basically just their initial demo and that's more or less it so far. Excited to see if they come out with something this summer though!

Re: OpenAI unveils its first custom chip, built by Broadcom

#462

Post by ipdashc » Thu, Jun 25, 2026, 2:18 PM UTC

wmf wrote: post #37

ipdashc wrote: post #34

Earlier quoted context omitted.

> 17k tps taalas chip It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually . And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases. I wonder if it's being work…

Good models will require multiple Taalas chips but Groq and Cerebras also require a lot of chips and that hasn't stopped them.

> Good models will require multiple Taalas chips

I guess that makes sense. Is this feasible, or does the added latency between chips kill any of the performance gains?

Re: OpenAI unveils its first custom chip, built by Broadcom

#463

Post by ipdashc » Thu, Jun 25, 2026, 2:19 PM UTC

mdp2021 wrote: post #351

ipdashc wrote: post #34

Earlier quoted context omitted.

> 17k tps taalas chip It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually . And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases. I wonder if it's being work…

> It's odd to me that I haven't heard anything about this approach ... I wonder if it's being worked on in secret, if there's something about it that makes it infeasible The studies and efforts are ongoing and public, and there are technical hurdles to be faced - but the relevant works go back in time quite a lot and there is heightened interest in it now. It seems that you simply took the "hyped headlines" for the w…

> It seems that you simply took the "hyped headlines" for the whole of the work.

Well, yeah, that's what I'm saying. It's odd that there haven't been any major headlines (customer interest, competitors' announcements, etc) other than their initial demo. Good to hear it's being worked on though!

Re: OpenAI unveils its first custom chip, built by Broadcom

#464

Post by flumes_whims_ » Thu, Jun 25, 2026, 2:23 PM UTC

typ wrote: post #363

nickpinkston wrote: post #86

This is very cool to see - seems like soooo much efficiency waiting to be unlocked at the chip level. What's everyone think of Taalas? They're actually burning the LLM model into the silicon, with some onboard memory for fine-tuning. They claim huge cost / latency wins. Super fast demo live at: https://chatjimmy.ai/ https://taalas.com/ https://www.reddit.com/r/singularity/comments/1r9frzk/taalas...

Low latency is nice. But it would be more interesting if they could demonstrate the efficiency of energy consumption.

Tokens/seconds and watt-hours seem related?

Re: OpenAI unveils its first custom chip, built by Broadcom

#465

Post by mdp2021 » Thu, Jun 25, 2026, 3:04 PM UTC

coder543 wrote: post #430

mdp2021 wrote: post #387

Earlier quoted context omitted.

I'd say the original remark was more general («this approach (baking LLMs/weights into silicon directly) [... as if] worked on in secret») - which is salient, because when I investigated weeks ago, I found a large number of attempts to CIM and to general branching from Von Neumann architecture for the purpose of optimizing NNs implementations in HW. Universities are studying, startups are proposing - the «approach» i…

CIM does not bake the weights into silicon. The level of optimization that you can do down to the last transistor when the weights are fixed is on an entirely different level than CIM where you still need general purpose ALUs all over the place.

> CIM does not bake the weights into silicon

If that were the extent of the terms, then what could we call "baking the weights into silicon"? Setting parts of the circuits to determined values for multiplication is is like printing a Read-Only Memory. (And you compute at it: Compute In Memory.)

> CIM where you still need general purpose ALUs all over the place

If that were so, then why do taxonomists present analogue computing as part of CIM? Ohm's Law does not constitute an "ALU" the way you intend it.

Simply, I used CIM, "Compute In Memory", for lack of a better term - for "store data there where you modify data", for "beyond Von Neumann's separation of data storage and processor".

Re: OpenAI unveils its first custom chip, built by Broadcom

#466

Post by coder543 » Thu, Jun 25, 2026, 3:12 PM UTC

mdp2021 wrote: post #465

coder543 wrote: post #430

Earlier quoted context omitted.

CIM does not bake the weights into silicon. The level of optimization that you can do down to the last transistor when the weights are fixed is on an entirely different level than CIM where you still need general purpose ALUs all over the place.

> CIM does not bake the weights into silicon If that were the extent of the terms, then what could we call "baking the weights into silicon"? Setting parts of the circuits to determined values for multiplication is is like printing a Read-Only Memory. (And you compute at it: Compute In Memory.) > CIM where you still need general purpose ALUs all over the place If that were so, then why do taxonomists present analogue…

EDIT: It's just not even worth arguing this point, so deleting my original, much longer comment. Abstract taxonomies can claim that Taalas is CIM, but this entirely and utterly misses the point, and misses what makes Taalas' approach special. If you told a room full of chip architects to go build "CIM for AI", they would not build a Taalas-like totally specialized chip, therefore it is not sufficient, and just muddies the conversation from my point of view. People have been doing "CIM" for decades and yet I've never seen anyone build a totally specialized chip at the scale of Taalas. And yes, you can (in theory) build an analog version of any computer, so of course you can build analog CIM, but "analog compute" is not inherently CIM, so conflating the two is just confusing.

Re: OpenAI unveils its first custom chip, built by Broadcom

#467

Post by mdp2021 » Thu, Jun 25, 2026, 3:14 PM UTC

ipdashc wrote: post #463

mdp2021 wrote: post #351

Earlier quoted context omitted.

> It's odd to me that I haven't heard anything about this approach ... I wonder if it's being worked on in secret, if there's something about it that makes it infeasible The studies and efforts are ongoing and public, and there are technical hurdles to be faced - but the relevant works go back in time quite a lot and there is heightened interest in it now. It seems that you simply took the "hyped headlines" for the w…

> It seems that you simply took the "hyped headlines" for the whole of the work. Well, yeah, that's what I'm saying. It's odd that there haven't been any major headlines (customer interest, competitors' announcements, etc) other than their initial demo. Good to hear it's being worked on though!

Did we not play with MNIST and placed some calculated bet on NNs well before Yann LeCun started the fire with the explosive success of the Convolutional NNs?

I'd say it pretty consistently starts in the underground.

The real revolution in the context is that it /could/ be done practically - overcoming the hurdles. But for what the interest in the matter is concerned, I'd say there almost cannot be a greater interest at this stage: making NNs efficient. This must be absolutely evident, as evident it is that the separation of memory and processor is against the idea of NNs, as evident as it is that multiplication is achievable just physically.

Of course many have seen that and got on studying it. As soon as it will be optimally practical...

Re: OpenAI unveils its first custom chip, built by Broadcom

#468

Post by davidpapermill » Thu, Jun 25, 2026, 3:32 PM UTC

HarHarVeryFunny wrote: post #180

sharkjacobs wrote: post #61

> Developed from design to production in nine months, accelerated by OpenAI’s models > the use of OpenAI models to accelerate parts of the design and optimization process. I wish there was more about this. As is I kind of have to assume that this is just meaningless marketing, like saying development was accelerated by Microsoft Office or their 5k LG Ultrafine 40-inch monitors. Like, if this was as big a deal as it k…

I would assume they've already made as big a deal of it as they can without outright lying too much. Read the rest of the press release. FWIW, Google is now on their 8th generation TPU, having put out the last 4 generations on a 1-year cadence.

> Google is now on their 8th generation TPU

Remarkable that the TPU pre-dates the attention paper. Was a solid bet on energy efficient dense matrix multiplication and has stood the test of time.

Re: OpenAI unveils its first custom chip, built by Broadcom

#469

Post by wmf » Thu, Jun 25, 2026, 3:40 PM UTC

ipdashc wrote: post #462

wmf wrote: post #37

Earlier quoted context omitted.

Good models will require multiple Taalas chips but Groq and Cerebras also require a lot of chips and that hasn't stopped them.

> Good models will require multiple Taalas chips I guess that makes sense. Is this feasible, or does the added latency between chips kill any of the performance gains?

Using multiple chips seems to work fine for Cerebras and Groq so it should also work for Taalas. It does sounds challenging to reach >10K tok/s but latency could be below 1 us which is a small part of the token budget.

Re: OpenAI unveils its first custom chip, built by Broadcom

#470

Post by mdp2021 » Thu, Jun 25, 2026, 3:41 PM UTC

coder543 wrote: post #466

mdp2021 wrote: post #465

Earlier quoted context omitted.

> CIM does not bake the weights into silicon If that were the extent of the terms, then what could we call "baking the weights into silicon"? Setting parts of the circuits to determined values for multiplication is is like printing a Read-Only Memory. (And you compute at it: Compute In Memory.) > CIM where you still need general purpose ALUs all over the place If that were so, then why do taxonomists present analogue…

EDIT: It's just not even worth arguing this point, so deleting my original, much longer comment. Abstract taxonomies can claim that Taalas is CIM, but this entirely and utterly misses the point, and misses what makes Taalas' approach special. If you told a room full of chip architects to go build "CIM for AI", they would not build a Taalas-like totally specialized chip, therefore it is not sufficient, and just muddie…

I can't check everything right now, but for example, the divulgational from Rakesh Kumar mentiones "Analogue CIM".

And I do not get your rant about "analog computing", which has everything to do with NNs (otherwise, well, prove it): they started with that - they are basically that in fact. Analogue computing is a very great temptation since it would solve the issues of inefficiency in digital NNs. Unfortunately, it has drawbacks which are massive for big NNs. Taalas' seems to be the best compromise.