Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI unveils its first custom chip, built by Broadcom

techcrunch.com

Post reply on HN
Print view View original

351–360 of 496 posts

Re: OpenAI unveils its first custom chip, built by Broadcom

#351

Post by mdp2021 »

ipdashc wrote: post #34
fennecbutt wrote: post #24

I mean I'd love to be able to buy something like the 17k tps taalas chip as a pcie or m.2. Imagine when we can roar along at that speed, low power. Can just have the model reason for a while about anything and everything. It reminds me of the "race to idle" for mcus etc.

> 17k tps taalas chip It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually . And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases. I wonder if it's being work…

> It's odd to me that I haven't heard anything about this approach ... I wonder if it's being worked on in secret, if there's something about it that makes it infeasible

The studies and efforts are ongoing and public, and there are technical hurdles to be faced - but the relevant works go back in time quite a lot and there is heightened interest in it now.

It seems that you simply took the "hyped headlines" for the whole of the work.

Re: OpenAI unveils its first custom chip, built by Broadcom

#352

Post by mdp2021 »

kazinator wrote: post #103

There is a never ending torrent of money coming, so why not make custom chips. Whoo ... party!

Although, custom HW has to be the focus right now - simply because we are dealing with a technology (big NNs) that are not the best match with Von Neumann architectures.

Re: OpenAI unveils its first custom chip, built by Broadcom

#353

Post by bigyabai »

wmf wrote: post #336
bigyabai wrote: post #316

Earlier quoted context omitted.

Personally, I doubt it. Apple hamstrung themselves with unified SOC memory, there are cheap dGPUs that smoke the M5's prefill speeds and even have faster decode too. Apple is running up against the limitations of putting a mobile integrated chipset up against the desktop form factor. An SOC stops looking like a smart decision at that scale. The software side is still pretty sketchy, too. Apple's ecosystem is fracture…

I love those 128 GB dGPUs.

Me too! The problem is that people don't love having 128gb of DDR5 held back with a laptop-grade iGPU. It puts up strictly non-interactive speed for LLMs of that size.

When you layer those same models across 128gb of dGPUs, then you can actually fill the KV cache in seconds, instead of minutes. And you get higher memory bandwidth on most professional cards.

Re: OpenAI unveils its first custom chip, built by Broadcom

#354

Post by mdp2021 »

rvz wrote: post #109

No surprise here. [0] [0] https://news.ycombinator.com/item?id=45429514

Actually, I find the idea of using Cerebras etc. for /training/ (not just inference) surprising: I did not stumble in much data and discussion about "super-CPUs" in that area, where NVidia (with the tools focused on it) has that long-built edge...

Edit: contextually,

> Jalapeño is specifically designed for inference

Re: OpenAI unveils its first custom chip, built by Broadcom

#355

Post by zgao »

surajrmal wrote: post #307
zgao wrote: post #266

Earlier quoted context omitted.

The typical way a chip effort in a non-chip company works is that the "design" is the RTL (e.g. SystemVerilog that defines the behavior of the chip) and then this is handed off to a third-party "design house" (such as Broadcom) that turns that code into a real image of a chip, which is called a GDS (basically you can think of this as a very big layer by layer photoshop file) that can actually be sent to a fab. This i…

I wonder if broadcomm borrowed IP between the Google tpu and this design. How would you ever know it didn't happen?

There is no real way to prevent this, but there are ways to increase the cost of doing so. For example, one level of obfuscation is, OAI could internally run synthesis and adopt a “netlist-in” model in which Broadcom gets a netlist - a description of a huge amount of gates and wires and how they connect - instead of the plain Verilog (or other language). It is possible to reverse engineer the netlist, but it’s a certain level of indirection and effort.

A big part of the semiconductor industry also operates on a reputation basis. Broadcom (like TSMC) is a neutral party as a design house, but if they did something like this, it might ruin that reputation.

Re: OpenAI unveils its first custom chip, built by Broadcom

#356

Post by bigyabai »

Schiendelman wrote: post #346
bigyabai wrote: post #316

Earlier quoted context omitted.

Personally, I doubt it. Apple hamstrung themselves with unified SOC memory, there are cheap dGPUs that smoke the M5's prefill speeds and even have faster decode too. Apple is running up against the limitations of putting a mobile integrated chipset up against the desktop form factor. An SOC stops looking like a smart decision at that scale. The software side is still pretty sketchy, too. Apple's ecosystem is fracture…

I don't expect them to be AS fast as Nvidia anytime soon. Understood that they need architectural improvements to get there. Apple's business model will be to pay Google for compute for now, and then as they get better on device, move more and more locally. So they're very well incentivized to get better. The thing they've been best at in the last 19 years has been spinning flywheels they already have, and this is ex…

I'm just genuinely convinced that Apple's AI flywheel is going in reverse. Their killed their golden goose with OpenCL, which had a genuine shot at dethroning CUDA if Apple took it seriously. It had industry-wide buy in and multiple implementations before Apple threw in the towel. When they designed Apple Silicon, they could have used the lessons learned from that experience to create a CUDA-like ALU layer instead of focusing on raster efficiency for their GPUs. Nvidia had proven that it was possible with low-power ARM SOCs like Jetson and Tegra which did deliver CUDA in handheld experiences. But Apple chose instead to delegate AI to the NPU, which is now dark silicon on devices that defer to MPS backends for most inference. The architecture is locked in to an expensive and suboptimal raster-first GPU design.

It's not hard to see why Apple made those mistakes, and many of them were made by the rest of the industry too. It's specifically tragic that Apple snatched defeat from the jaws of victory with GPGPU programming, and it makes me think that their future will be more subscription services and less half-ass technical efforts. Or they rip up the foundation and start from scratch, it's never too late to start work on Apple Silicon 2.

Re: OpenAI unveils its first custom chip, built by Broadcom

#357

Post by coder543 »

MichaelNolan wrote: post #33
fennecbutt wrote: post #24

I mean I'd love to be able to buy something like the 17k tps taalas chip as a pcie or m.2. Imagine when we can roar along at that speed, low power. Can just have the model reason for a while about anything and everything. It reminds me of the "race to idle" for mcus etc.

The current taalas chip is for a 3.1B param model. I’m hope so much that they can get that up to the 30B range. Just imagine Gemma 4 or Qwen 3.6 at 17k tps.

Taalas' first chip is for a Llama 3.1 8B quant, not a 3.1B parameter model, to clarify.

Re: OpenAI unveils its first custom chip, built by Broadcom

#358

Post by coder543 »

ipdashc wrote: post #34
fennecbutt wrote: post #24

I mean I'd love to be able to buy something like the 17k tps taalas chip as a pcie or m.2. Imagine when we can roar along at that speed, low power. Can just have the model reason for a while about anything and everything. It reminds me of the "race to idle" for mcus etc.

> 17k tps taalas chip It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually . And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases. I wonder if it's being work…

> It's odd to me that I haven't heard anything about this approach since.

It has only been four months since they unveiled their first prototype. I don't understand your confusion. Chip development does not happen overnight...?

Their initial blog post laid out a roadmap, so theoretically they should have another thing to demonstrate this summer.

Re: OpenAI unveils its first custom chip, built by Broadcom

#359

Post by Schiendelman »

bigyabai wrote: post #356
Schiendelman wrote: post #346

Earlier quoted context omitted.

I don't expect them to be AS fast as Nvidia anytime soon. Understood that they need architectural improvements to get there. Apple's business model will be to pay Google for compute for now, and then as they get better on device, move more and more locally. So they're very well incentivized to get better. The thing they've been best at in the last 19 years has been spinning flywheels they already have, and this is ex…

I'm just genuinely convinced that Apple's AI flywheel is going in reverse. Their killed their golden goose with OpenCL, which had a genuine shot at dethroning CUDA if Apple took it seriously. It had industry-wide buy in and multiple implementations before Apple threw in the towel. When they designed Apple Silicon, they could have used the lessons learned from that experience to create a CUDA-like ALU layer instead of…

I think it's easy to understand why Apple wouldn't build low level engineering solutions - they'd rather control the platform and just have developers call MLX. I'm not sure, if I was in their shoes, that I'd make the same call. But it's a call, and it's consistent with the rest of their ecosystem decisions.

Re: OpenAI unveils its first custom chip, built by Broadcom

#360

Post by mdp2021 »

jupr wrote: post #74
tantalor wrote: post #62

Earlier quoted context omitted.

If you're referring to the big circle of silicon, that's a wafer, generally contains many chips (100-1000s).

That made me chuckle but I guess if you have never seen one I could see how that assumption could be made. If this photo is real I wonder what can be revealed about the approach they have taken by analyzing the architecture of what we can see.

> That made me chuckle but I guess if you have never seen one I could see how that assumption could be made

It's more like that "wafer as a big-chip" (more formally, "WSE - Wafer Scale Engine") is now a reality (see Cerebras).

But in this case, the wafer will be split into a few dozen chunks.

Post reply on HN