Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI unveils its first custom chip, built by Broadcom

techcrunch.com

Post reply on HN
Print view View original

341–350 of 496 posts

Re: OpenAI unveils its first custom chip, built by Broadcom

#341

Post by pama »

zgao wrote: post #125
sharkjacobs wrote: post #61

> Developed from design to production in nine months, accelerated by OpenAI’s models > the use of OpenAI models to accelerate parts of the design and optimization process. I wish there was more about this. As is I kind of have to assume that this is just meaningless marketing, like saying development was accelerated by Microsoft Office or their 5k LG Ultrafine 40-inch monitors. Like, if this was as big a deal as it k…

Chip CEO here. It really depends on what "design" or "production" means. Does "design" mean that the design was complete? Does "production" mean the beginning of production, i.e. tapeout? If measuring from RTL-freeze to tapeout, this is a fairly typical (even somewhat unimpressive) timeline (accounting for some unexpected issues) for a large, complex 3nm chip. If measuring from concept (no RTL at all, block diagram o…

If you look at the timelines for the hiring of the hardware team, this was an extremely fast and high risk implementation from concept to tapeout. Amazing it works at all during bringup.

Re: OpenAI unveils its first custom chip, built by Broadcom

#342

Post by re-thc »

reinitctxoffset wrote: post #298
selectodude wrote: post #257

Earlier quoted context omitted.

I think the folks at PA Semi had some chops too.

The way I heard it PA Semi was the singular driving force that led to Apple Silicon, but I'm not any kind of insider that's just the chatter I heard. Whoever it was, whooo, that's hot shit. I remember an M1 MacBook Air just cleaning the clock of an Intel MacBook Pro and thinking "x86_64 has real competition again". Great silicon. I'm over it with not having root on my own machine, so I've left the ecosystem, but it's…

> The way I heard it PA Semi was the singular driving force that led to Apple Silicon

And a lot of them are sitting under Qualcomm via the Nuvia acquisition.

Re: OpenAI unveils its first custom chip, built by Broadcom

#343

Post by mdp2021 »

lifeisstillgood wrote: post #271

So I’ve been wondering about “one or two levels back” chip design. If I understand it, 28nm chips (pre EUV) is just about suitable to run (not train just inference) frontier models. And so if I was a mid-level State would it be worth while to take my nascent chip industry and push it out to build a 28nm foundry and supporting eco-system. The models will come but the real challenge of the future is having enough compu…

> Even if LLMs don’t become AGI they will still be incredible tools

(Mostly an aside, but: LLMs have paved the way, now the problem is there, it is a challenge and a geopolitically relevant race... AGI is a goal set: not-having-reached-it will be just a stage.)

Re: OpenAI unveils its first custom chip, built by Broadcom

#344

Post by Meetvelde »

1e1a wrote: post #256
sometimelurker wrote: post #212

Earlier quoted context omitted.

this appeared some time ago, https://taalas.com/ , but I'm sure there's others thinking these same thoughts. this would be best for small models imo, nothing frontier because that changes too fast

you can try it out here: https://chatjimmy.ai/

that's so fast it feels fake

Re: OpenAI unveils its first custom chip, built by Broadcom

#345

Post by redox99 »

kccqzy wrote: post #236
nickpinkston wrote: post #86

This is very cool to see - seems like soooo much efficiency waiting to be unlocked at the chip level. What's everyone think of Taalas? They're actually burning the LLM model into the silicon, with some onboard memory for fine-tuning. They claim huge cost / latency wins. Super fast demo live at: https://chatjimmy.ai/ https://taalas.com/ https://www.reddit.com/r/singularity/comments/1r9frzk/taalas...

> seems like soooo much efficiency waiting to be unlocked at the chip level Well if you are exclusively using GPUs that are general purpose, of course you leave so much efficiency on the table. That’s why Google started making TPUs more than a decade ago. I remember that kerfuffle when Google fired Timnit Gebru when Gebru’s paper used GPUs to calculate the environment impact of LLMs while ignoring the efficiency of T…

These NVIDIA GPUs aren't general purpose in the way that you think. They can't even run games. Nvidia blackwell is probably slightly more efficient than TPUs for training. Do you really expect a 4 trillion company with the majority of its revenue being AI for some years now, not to have built its flagship product fully around AI? The GPU name stuck around, but they are pretty terrible at graphics.

The real efficiency win in these chips is that they are made for inference only. You can throw away the vast majority of a chip if you only need a few ops, a single precision (like INT8 or FP8) and don't need ultra fast interconnects.

Re: OpenAI unveils its first custom chip, built by Broadcom

#346

Post by Schiendelman »

bigyabai wrote: post #316
Schiendelman wrote: post #301

Earlier quoted context omitted.

With the M6 theoretically coming later this year, Apple seems to be realizing they need to catch up with more lanes of GPU.

Personally, I doubt it. Apple hamstrung themselves with unified SOC memory, there are cheap dGPUs that smoke the M5's prefill speeds and even have faster decode too. Apple is running up against the limitations of putting a mobile integrated chipset up against the desktop form factor. An SOC stops looking like a smart decision at that scale. The software side is still pretty sketchy, too. Apple's ecosystem is fracture…

I don't expect them to be AS fast as Nvidia anytime soon. Understood that they need architectural improvements to get there.

Apple's business model will be to pay Google for compute for now, and then as they get better on device, move more and more locally. So they're very well incentivized to get better. The thing they've been best at in the last 19 years has been spinning flywheels they already have, and this is exactly that.

Re: OpenAI unveils its first custom chip, built by Broadcom

#347

Post by wmf »

lifeisstillgood wrote: post #271

So I’ve been wondering about “one or two levels back” chip design. If I understand it, 28nm chips (pre EUV) is just about suitable to run (not train just inference) frontier models. And so if I was a mid-level State would it be worth while to take my nascent chip industry and push it out to build a 28nm foundry and supporting eco-system. The models will come but the real challenge of the future is having enough compu…

28nm chips is just about suitable to run frontier models

I doubt it. 28 nm is 4-5 generations back so inferencing would need a large number of chips with very high power consumption. Maybe you're thinking more of 7 nm which is what Chinese fabs have; it seems to be OK for companies like Huawei.

And so if I was a mid-level State would it be worth while to take my nascent chip industry and push it out to build a 28nm foundry and supporting eco-system.

It never reaches breakeven so you'd have to provide billions in subsidies per year forever. The sovereign chip stuff only makes sense for the US and China; even the EU probably isn't large enough to make it work. A single country definitely couldn't.

Re: OpenAI unveils its first custom chip, built by Broadcom

#348

Post by post-it »

gpm wrote: post #118
wongarsu wrote: post #99

Earlier quoted context omitted.

A hard sell right now . The rate of change will slow down

Yes, but with current architectures world knowledge is baked into the weights. We might stop figuring out how to make models better, but the world keeps changing, science is going to keep making progress at understanding the world, etc. This creates a significant minimum rate of change and I'm pretty skeptical that it's worth baking weights into silicon as a result.

This already isn't the case for the popular models. The knowledge baked into the weights tells the model how to talk and reason, but for world knowledge they do a web search right off the bat most of the time.

Re: OpenAI unveils its first custom chip, built by Broadcom

#349

Post by the_sleaze_ »

Meetvelde wrote: post #344
1e1a wrote: post #256

Earlier quoted context omitted.

you can try it out here: https://chatjimmy.ai/

that's so fast it feels fake

13,789 tok/s

Well I've gotten one of those "holy fuck this is the future" deeply unsettled anxious feelings in my gut again. It's been a week or 2, it was time.

Re: OpenAI unveils its first custom chip, built by Broadcom

#350

Post by luma »

surajrmal wrote: post #311
aurareturn wrote: post #246

Earlier quoted context omitted.

They have. That's why OpenAI was able to get a working demo in 9 months. But going from a small scale system to a full fledged data center deployment is likely much harder. I don't know how much of the things outside of the chip Broadcom has vs Google's proprietary tech that is not shared with Broadcom. Nvidia's Vera Rubin has 6 unique chips working together in a single rack.[0] [0] https://developer-blogs.nvidia.com…

I thought most of the Google tpu magic is on wiring up these chips into supercomputer like clusters with specialized interconnects and whatnot. The chips themselves are less interesting in isolation.

I know nothing of what is happening here but Broadcom has a lot of IP in high speed/low latency data transfer from chip to datacenter scales.
Post reply on HN