Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI unveils its first custom chip, built by Broadcom

techcrunch.com

Post reply on HN
Print view View original

311–320 of 496 posts

Re: OpenAI unveils its first custom chip, built by Broadcom

#311

Post by surajrmal »

aurareturn wrote: post #246
cptskippy wrote: post #239

Earlier quoted context omitted.

Why do you assume Broadcom has a ton of IP for AI SoCs but hasn't done any of the other work around data center scale deployments?

They have. That's why OpenAI was able to get a working demo in 9 months. But going from a small scale system to a full fledged data center deployment is likely much harder. I don't know how much of the things outside of the chip Broadcom has vs Google's proprietary tech that is not shared with Broadcom. Nvidia's Vera Rubin has 6 unique chips working together in a single rack.[0] [0] https://developer-blogs.nvidia.com…

I thought most of the Google tpu magic is on wiring up these chips into supercomputer like clusters with specialized interconnects and whatnot. The chips themselves are less interesting in isolation.

Re: OpenAI unveils its first custom chip, built by Broadcom

#312

Post by aleph_minus_one »

NBJack wrote: post #306
mlinhares wrote: post #285

Earlier quoted context omitted.

Nah, you just need to get the CEO behind it. Most coordination issues get solved when the CEO is breathing down your neck to get something done. Trouble is that they don't do this enough.

Eh, zero guarantees on that one. The Fire Phone was Jeff Bezos' personal baby, and we know how that went. Then there was the Apple G4 Cube with Steve Jobs, the Model X' Falcon Wing doors and Elon, and lets not even talk about the Metaverse and Zuck.

> The Fire Phone was Jeff Bezos' personal baby, and we know how that went.

I'd rather guess that Jeff Bezos' opinion on what makes a good phone is/was different on the opinion of many potential buyers.

Re: OpenAI unveils its first custom chip, built by Broadcom

#313

Post by Herring »

jacques_chester wrote: post #242
kccqzy wrote: post #236

Earlier quoted context omitted.

> seems like soooo much efficiency waiting to be unlocked at the chip level Well if you are exclusively using GPUs that are general purpose, of course you leave so much efficiency on the table. That’s why Google started making TPUs more than a decade ago. I remember that kerfuffle when Google fired Timnit Gebru when Gebru’s paper used GPUs to calculate the environment impact of LLMs while ignoring the efficiency of T…

That ... wasn't the kerfuffle

It kind of was. I really hate gaslighting, but GP is not inaccurate. Google claimed it did not meet their bar for publication because it ignored recent research on how to reduce the environmental and bias-related risks of LLMs. On the other hand, a large org is unlikely to subsidize high-profile research that makes it look bad. And Gebru was critical of Google’s internal culture and diversity efforts…

Re: OpenAI unveils its first custom chip, built by Broadcom

#315

Post by forrestthewoods »

lugu wrote: post #303
forrestthewoods wrote: post #15

Earlier quoted context omitted.

Inference costs are higher than training now. I think. Nvidia is king of general purpose training chips. But inferences can be specialized.

What makes you think this? With wider adoption the ratio shall shift in favor of inference. And API price is becoming more important than SOTA capability.

> With wider adoption the ratio shall shift in favor of inference

Yes? That’s why more money will be spent on inference than training?

I’m talking absolute cost. As the number of people using AI and burning tokens goes up the amount of spend on inference goes up.

I am fairly confident that Anthropic has way way more GPUs serving Claude Code to users than they have training models. They’ve got a lot of users!!

> API price is becoming more important than SOTA capability.

Also yes? This is why custom silicon for efficient inference makes sense!

I think we’re in total agreement here :)

Re: OpenAI unveils its first custom chip, built by Broadcom

#316

Post by bigyabai »

Schiendelman wrote: post #301
bigyabai wrote: post #78

Earlier quoted context omitted.

The tradeoff is that the M3 Ultra's GPU loses to laptop GPUs in compute benchmarks. All of that bandwidth is wasted idling for token prefill. For inference workloads, it makes a lot more sense to optimize for prefill/ttft before maxing out memory bandwidth.

With the M6 theoretically coming later this year, Apple seems to be realizing they need to catch up with more lanes of GPU.

Personally, I doubt it. Apple hamstrung themselves with unified SOC memory, there are cheap dGPUs that smoke the M5's prefill speeds and even have faster decode too. Apple is running up against the limitations of putting a mobile integrated chipset up against the desktop form factor. An SOC stops looking like a smart decision at that scale.

The software side is still pretty sketchy, too. Apple's ecosystem is fractured between NPU, MPS and Accelerate BLAS, with libraries like MLX and CoreML built precariously overtop. Apple has to commit to a full rearchitecture of their GPU to challenge Nvidia, which fractures that ecosystem even further.

Re: OpenAI unveils its first custom chip, built by Broadcom

#317

Post by 8note »

cm2187 wrote: post #308
nine_k wrote: post #234

Earlier quoted context omitted.

In very crude terms, AFAICT , if you have a bunch of matrix multiplications, but one of matrices (the one with model weights) doesn't change, you can seriously speed up the computation. One thing is that you don't need to re-fetch the elements of the constant matrix, you can keep it near the ALUs. Then you maybe can detect and ignore sparse / empty blocks by marking them once. IDK how the custom hardware exploits thi…

Random thought. Once models stabilise, could you possibly hardcode the model in gates? Or are they too large for a single chip?

https://www.anuragk.com/blog/posts/Taalas.html
Post reply on HN