Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI unveils its first custom chip, built by Broadcom

techcrunch.com

Post reply on HN
Print view View original

301–310 of 496 posts

Re: OpenAI unveils its first custom chip, built by Broadcom

#301

Post by Schiendelman »

bigyabai wrote: post #78
phonon wrote: post #56

Earlier quoted context omitted.

M3 Ultra has a 1024 bit memory bus (819 GB/s) and starts at $3,999 (96GB of RAM). It can be done....

The tradeoff is that the M3 Ultra's GPU loses to laptop GPUs in compute benchmarks. All of that bandwidth is wasted idling for token prefill. For inference workloads, it makes a lot more sense to optimize for prefill/ttft before maxing out memory bandwidth.

With the M6 theoretically coming later this year, Apple seems to be realizing they need to catch up with more lanes of GPU.

Re: OpenAI unveils its first custom chip, built by Broadcom

#303

Post by lugu »

forrestthewoods wrote: post #15
maz1b wrote: post #5

Pretty huge move. Google and their TPUs are looking infinitely more prescient as I think they are on their 7th generation, along with the offshoots it inspired like the LPU and even others, perhaps like Cerebras and their Wafer Scale Engine. However, based off first impressions, it seems like this is meant for inference side, and not training, which is also an interesting choice.

Inference costs are higher than training now. I think. Nvidia is king of general purpose training chips. But inferences can be specialized.

What makes you think this? With wider adoption the ratio shall shift in favor of inference. And API price is becoming more important than SOTA capability.

Re: OpenAI unveils its first custom chip, built by Broadcom

#304

Post by Rekindle8090 »

deweywsu wrote: post #122

With the pace of AI, and with AI helping to pave the way for faster/better AI, I keep wondering if hardware like this will become obsolete well before it has a meaningful ROI. Huge AI models can be run with less resources already through quantization and offloading, but that's just the beginning. One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old D…

[dead]

Re: OpenAI unveils its first custom chip, built by Broadcom

#305

Post by Salgat »

londons_explore wrote: post #192

I wanna see an inference chip where the weights are part of the rom of the chip. There would be 1 multiplier per weight (and since they're constant, the whole thing turns into a bunch of simple adders), and the total pipelined system throughput would be one token per clock cycle. That means you can probably have millions of users simultaneously using a single bit of silicon, with perhaps 500 million tokens per second…

Supposedly memristors would be ideal for this (and it would be reprogrammable), but then again, memristors seem to be the carbon nanotubes of the computing world.

Re: OpenAI unveils its first custom chip, built by Broadcom

#306

Post by NBJack »

mlinhares wrote: post #285
RugnirViking wrote: post #254

Earlier quoted context omitted.

Big companies are lumbering behemoth, crude assemblages of barely cobbled-together incentives and principal agent problems in a trenchcoat. Getting them to change direction, or worse, try something new at scale, is a massive undertaking

Nah, you just need to get the CEO behind it. Most coordination issues get solved when the CEO is breathing down your neck to get something done. Trouble is that they don't do this enough.

Eh, zero guarantees on that one.

The Fire Phone was Jeff Bezos' personal baby, and we know how that went. Then there was the Apple G4 Cube with Steve Jobs, the Model X' Falcon Wing doors and Elon, and lets not even talk about the Metaverse and Zuck.

Re: OpenAI unveils its first custom chip, built by Broadcom

#307

Post by surajrmal »

zgao wrote: post #266
nonethewiser wrote: post #162

Earlier quoted context omitted.

>If measuring from RTL-freeze to tapeout, this is a fairly typical (even somewhat unimpressive) timeline (accounting for some unexpected issues) for a large, complex 3nm chip. Even for a company’s first design?

The typical way a chip effort in a non-chip company works is that the "design" is the RTL (e.g. SystemVerilog that defines the behavior of the chip) and then this is handed off to a third-party "design house" (such as Broadcom) that turns that code into a real image of a chip, which is called a GDS (basically you can think of this as a very big layer by layer photoshop file) that can actually be sent to a fab. This i…

I wonder if broadcomm borrowed IP between the Google tpu and this design. How would you ever know it didn't happen?

Re: OpenAI unveils its first custom chip, built by Broadcom

#308

Post by cm2187 »

nine_k wrote: post #234
xdavidliu wrote: post #184

Earlier quoted context omitted.

i don't understand what the second paragraph is saying.

In very crude terms, AFAICT , if you have a bunch of matrix multiplications, but one of matrices (the one with model weights) doesn't change, you can seriously speed up the computation. One thing is that you don't need to re-fetch the elements of the constant matrix, you can keep it near the ALUs. Then you maybe can detect and ignore sparse / empty blocks by marking them once. IDK how the custom hardware exploits thi…

Random thought. Once models stabilise, could you possibly hardcode the model in gates? Or are they too large for a single chip?

Re: OpenAI unveils its first custom chip, built by Broadcom

#309

Post by reinitctxoffset »

NitpickLawyer wrote: post #198
xnx wrote: post #160

Earlier quoted context omitted.

Impossible to know. Could be fake/aspirational roles to impress investors with their grand vision.

Jesus. This is tinfoil hat territory now. Why would they fake something like that? ANY company in this field would try to become free from nvda. Goog has done it already, amazon has their own thing, so it can be done. Not saying they'll 0shot this vertical, but ffs, they don't need to fake anything. They are making an effort, and it would be insane to think they aren't. Might work, might not work, but to even think t…

It kinda depends on what your prior is. Some companies do a press release and I immediately pay attention or even take action.

Other companies? Fool me once Altman, let's see the thing at scale making money.

Near frontier AI is clearly relevant to some kinds of logic design, I'm learning some Hardcaml at the moment and yeah, AI is super helpful.

Can it leapfrog a company without hardware experience to near the front of the pack of companies with decades of hardware experience? Less obvious.

Unrelatedly, would OpenAI dramatically overstate something to manipulate the press and public and capital markets?

It's arguably their core competency .

AI is going to matter in logic design and synthesis. How much, how soon, and where are open questions.

Post reply on HN