Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI unveils its first custom chip, built by Broadcom

techcrunch.com

Post reply on HN
Print view View original

221–230 of 496 posts

Re: OpenAI unveils its first custom chip, built by Broadcom

#221

Post by zurfer »

cruffle_duffle wrote: post #208
londons_explore wrote: post #192

I wanna see an inference chip where the weights are part of the rom of the chip. There would be 1 multiplier per weight (and since they're constant, the whole thing turns into a bunch of simple adders), and the total pipelined system throughput would be one token per clock cycle. That means you can probably have millions of users simultaneously using a single bit of silicon, with perhaps 500 million tokens per second…

“ Wafer level faults probably won't matter though - neural nets are resistant to a few missing or wrong weights.” Brain science people “love” traumatic brain injury cases because it can help explore what happens when bits of the “brain wafer” get damaged. We’ve learned a lot from such things. I wonder if people are intentionally “destroying” parts of the model weights to learn more about what happens? Like could you…

This is called mechanistic interpretability. There is lots of fascinating insights already since you can do basically everything down to the neuron or weight level thousands of times. The human brain is many orders of magnitude harder to make sense of.

Re: OpenAI unveils its first custom chip, built by Broadcom

#222

Post by phkahler »

londons_explore wrote: post #192

I wanna see an inference chip where the weights are part of the rom of the chip. There would be 1 multiplier per weight (and since they're constant, the whole thing turns into a bunch of simple adders), and the total pipelined system throughput would be one token per clock cycle. That means you can probably have millions of users simultaneously using a single bit of silicon, with perhaps 500 million tokens per second…

>> I wanna see an inference chip where the weights are part of the rom of the chip.

I've been wondering about that for a while now. For a lot of tasks putting weights in ROM is probably OK. OTOH:

>> There would be 1 multiplier per weight...

I'm not sure that is a good idea. Maybe if its quantized down to 2 bits... Otherwise maybe a small ROM near each multiplier (or row of them or whatever) so the multipliers could handle N distinct matrix operations without having to move the data from far away.

Another fun thought is to have a row of MAC units on DRAM so a DRAM row would be a vector. Row size might be 64Kbit or 8K weights if they're 8bit. This also keeps the weights and calcs on the same chip. I'm not sure this would put enough multipliers on one chip though. Systolic arrays can have tens or hundreds of thousands each doing one op per clock cycle.

Re: OpenAI unveils its first custom chip, built by Broadcom

#223

Post by NitpickLawyer »

luqtas wrote: post #216
NitpickLawyer wrote: post #198

Earlier quoted context omitted.

Jesus. This is tinfoil hat territory now. Why would they fake something like that? ANY company in this field would try to become free from nvda. Goog has done it already, amazon has their own thing, so it can be done. Not saying they'll 0shot this vertical, but ffs, they don't need to fake anything. They are making an effort, and it would be insane to think they aren't. Might work, might not work, but to even think t…

https://antoniocortes.com/en/2026/03/10/ghost-jobs-the-econo...

I'm not saying this isn't a thing. I'm saying oAI doesn't need to fake trying to make a chip or hiring people to make AI better at chip making, or dogfooding or anything like this. It's obvious they're doing it. They'd have 0 reason to fake something like this "for the investors". Come on!

Re: OpenAI unveils its first custom chip, built by Broadcom

#224

Post by NavinF »

ahartmetz wrote: post #71
HarHarVeryFunny wrote: post #20

Earlier quoted context omitted.

I just read a claim on Twitter that the reason these companies (Google and Amazon as well as OpenAI) are using Broadcom isn't just for design expertise, but because Broadcom have allocation agreements in place with TSMC and the memory manufacturers.

...and because most hardware sales except AI accelerators are down due to RAM prices, Broadcom probably can't otherwise use their allocation at TSMC.

Nope, not down. "total Personal Computing Device (PCD) market — comprising traditional PCs and tablets — posted 2.8% year-over-year growth in Q1 2026, with combined shipments reaching 103.3 million units. PC shipments grew 3% YoY with 65.6 million units" https://www.idc.com/promo/pcdforecast/

Q2 is forecasted to be negative, partly because of RAM prices like you said, but for the most part this is something that only price sensitive nerds care about. Broadcom sells a ton of server chips. Server sales are up 30% vs last year so I highly doubt they're desperate to use their allocation

Re: OpenAI unveils its first custom chip, built by Broadcom

#225

Post by cruffle_duffle »

martythemaniak wrote: post #102
nickpinkston wrote: post #86

This is very cool to see - seems like soooo much efficiency waiting to be unlocked at the chip level. What's everyone think of Taalas? They're actually burning the LLM model into the silicon, with some onboard memory for fine-tuning. They claim huge cost / latency wins. Super fast demo live at: https://chatjimmy.ai/ https://taalas.com/ https://www.reddit.com/r/singularity/comments/1r9frzk/taalas...

In a chatbot, 17k tok/s is a neat but nearly useless showcase. In a coding agent it is a meaningful improvement. In robotics, it could be an absolute revolution. 8B models aren't useful in general, but for specific use cases they can provide an enourmous amount of intelligence - nVidia's Tesla/Waymo competitor is a 7B LLM with a 2B diffusion model, and running that at those speeds could be an order of magnitude cheap…

Bumping the speed of these things would be more than meaningful. It would be a massive game changer.

I assert like 80% of this “multi agent parallel workflow” business is simply a workaround to models being soooooo slow. Like as the dude driving these things… you kick it off and twiddle your thumbs waiting minutes to hours sometimes for all the inference and token generator to finish. So you dispatch multiple workstreams in parallel to be more efficient.

I assert that if the model was even 10x faster we’d be using these things radically different. You’d be doing things that are currently time prohibitive. At 100x, holy shit will software dev get crazy. You’d be kicking off hundreds of parallel workers attacking a problem from every angle and stuff. Who even knows!!!

And the thing is, 10x will absolutely come and probably even 100x. And it will be sold like a video game cartridge or something depending on how the actual model gets “baked” into the hardware. No remote inference at all.

Re: OpenAI unveils its first custom chip, built by Broadcom

#226

Post by trouve_search »

Smaug123 wrote: post #210
londons_explore wrote: post #192

I wanna see an inference chip where the weights are part of the rom of the chip. There would be 1 multiplier per weight (and since they're constant, the whole thing turns into a bunch of simple adders), and the total pipelined system throughput would be one token per clock cycle. That means you can probably have millions of users simultaneously using a single bit of silicon, with perhaps 500 million tokens per second…

By the way, you've seen Cerebras? It's not gone as far as what you described - loads of cores and RAM but you still load up the weights onto it as software and they need to be streamed into the chip for large models - but it is a whole wafer.

Cerebras is a whole lot of SRAM, basically a ton more L1/L2 cache, hence increasing throughput.

They're pretty supply constrained right now though and their production costs seem prohibitive.

The interesting players at the moment are from Toronto: taalas (print the model onto the silicon) and tenstorrent (dataflow programming based hardware)

Re: OpenAI unveils its first custom chip, built by Broadcom

#227

Post by phkahler »

ACCount37 wrote: post #205
pennomi wrote: post #151

Earlier quoted context omitted.

That assumes scaling laws still hold up. A bigger model might end up only incrementally more intelligent.

They do. Mythos kicked ass while it lasted. And what we know of the scaling law curves promises us even more gains in the future. "The future" being "whenever training and inference at increased scale becomes economical". Which is probably bounded by new generations of hardware, but might also be pushed forward by algorithmic advances.

I think they're out of training data though...

Re: OpenAI unveils its first custom chip, built by Broadcom

#228

Post by cyptus »

phkahler wrote: post #222
londons_explore wrote: post #192

I wanna see an inference chip where the weights are part of the rom of the chip. There would be 1 multiplier per weight (and since they're constant, the whole thing turns into a bunch of simple adders), and the total pipelined system throughput would be one token per clock cycle. That means you can probably have millions of users simultaneously using a single bit of silicon, with perhaps 500 million tokens per second…

>> I wanna see an inference chip where the weights are part of the rom of the chip. I've been wondering about that for a while now. For a lot of tasks putting weights in ROM is probably OK. OTOH: >> There would be 1 multiplier per weight... I'm not sure that is a good idea. Maybe if its quantized down to 2 bits... Otherwise maybe a small ROM near each multiplier (or row of them or whatever) so the multipliers could h…

analog chips could also be very interessting instead of using digital signals and processing them against the weights in the ROM. I have no idea if that scales with such big models though.

Re: OpenAI unveils its first custom chip, built by Broadcom

#229

Post by ACCount37 »

phkahler wrote: post #227
ACCount37 wrote: post #205

Earlier quoted context omitted.

They do. Mythos kicked ass while it lasted. And what we know of the scaling law curves promises us even more gains in the future. "The future" being "whenever training and inference at increased scale becomes economical". Which is probably bounded by new generations of hardware, but might also be pushed forward by algorithmic advances.

I think they're out of training data though...

Synthetics are often used for "data amplification" nowadays. Extra compute covers a multitude of sins.

Re: OpenAI unveils its first custom chip, built by Broadcom

#230

Post by kccqzy »

a_conservative wrote: post #11
shellcromancer wrote: post #4

Probably obvious but still omitted in the OpenAI post: chips are being made by TSMC [1]. Wasn't sure if Intel got it. 1. https://www.investing.com/news/stock-market-news/openai-unve...

I recently put 2+2 together. Broadcom has become wealthy by being Google's TPU hardware partner, including sharing their TSMC capacity with Google, and evidently now they are doing the same thing with OpenAI. What a brilliant way to take advantage of the AI gold rush! I wish they weren't using their piles of money to extort money out of the software industry like they are with VMWare and Bitnami.

Well Google has reduced reliance on Broadcom already. They found a new hardware partner, MediaTek, that’s probably much, much cheaper than Broadcom.

https://finance.yahoo.com/sectors/technology/articles/broadc...

Post reply on HN