Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI unveils its first custom chip, built by Broadcom

techcrunch.com

Post reply on HN
Print view View original

471–480 of 496 posts

Re: OpenAI unveils its first custom chip, built by Broadcom

#471

Post by RussianCow »

Jyaif wrote: post #442
ngomez wrote: post #403

Earlier quoted context omitted.

Using the space of an entire wafer for one chip would result in extremely low manufacturing yields. Even with state of the art silicon cleanrooms, there will still be defects in parts of the output. With CPUs and GPUs, chip makers can disable faulty cores and bin them as lower SKUs to get some yield out of it. But if you're using an entire wafer to embed weights, and a speck of dust causes a printing defect that make…

What's the difference between disabling faulty cores and disabling the parts of the wafer that have defects?

I'm not an expert, but I think those are the same thing. But for an LLM etched onto a whole wafer, it doesn't make sense to disable part of it since that would remove some weights entirely.

Re: OpenAI unveils its first custom chip, built by Broadcom

#472

Post by fulafel »

nine_k wrote: post #234
xdavidliu wrote: post #184

Earlier quoted context omitted.

i don't understand what the second paragraph is saying.

In very crude terms, AFAICT , if you have a bunch of matrix multiplications, but one of matrices (the one with model weights) doesn't change, you can seriously speed up the computation. One thing is that you don't need to re-fetch the elements of the constant matrix, you can keep it near the ALUs. Then you maybe can detect and ignore sparse / empty blocks by marking them once. IDK how the custom hardware exploits thi…

Current accelerators (TPUs, various onchip NPUs) are something close to this. Systolic array is the estabilished computer architecture term for flowing data from computation to computation without the overhead of a register file or von Neumann bottleneck.

Re: OpenAI unveils its first custom chip, built by Broadcom

#473

Post by AtlasBarfed »

aleph_minus_one wrote: post #312
NBJack wrote: post #306

Earlier quoted context omitted.

Eh, zero guarantees on that one. The Fire Phone was Jeff Bezos' personal baby, and we know how that went. Then there was the Apple G4 Cube with Steve Jobs, the Model X' Falcon Wing doors and Elon, and lets not even talk about the Metaverse and Zuck.

> The Fire Phone was Jeff Bezos' personal baby, and we know how that went. I'd rather guess that Jeff Bezos' opinion on what makes a good phone is/was different on the opinion of many potential buyers.

An Amazon phone with Amazon Video, playing Amazon Music, making phone calls throug the Amazon messenger, with an Amazon Browser that overlays ads to Amazon products, and has Amazon Voice Recognition ... blah blah blah

I imagine when you are a billionaire from one company, every time you hear the name of the company you hear your name, so you can't really think about what Joe Schmoe wants in a phone independently of your ego.

I guess this is what Steve Jobs was better at. SOME focus on the customer independent of his ego and Apple Apple Apple. I did say ... SOME.

Re: OpenAI unveils its first custom chip, built by Broadcom

#474

Post by AtlasBarfed »

aurareturn wrote: post #231
sharkjacobs wrote: post #61

> Developed from design to production in nine months, accelerated by OpenAI’s models > the use of OpenAI models to accelerate parts of the design and optimization process. I wish there was more about this. As is I kind of have to assume that this is just meaningless marketing, like saying development was accelerated by Microsoft Office or their 5k LG Ultrafine 40-inch monitors. Like, if this was as big a deal as it k…

Broadcom already has a ton of IP for AI SoCs. I'm guessing the hard parts of this inference chip was already designed by Broadcom and OpenAI simply told Broadcom what it wanted. It's likely very similar to Google's TPU. Early testing shows that the first-generation accelerator will deliver performance per watt substantially better than current state-of-the-art What is substantial here? Vera Rubin is shipping in volum…

"Substantial" seems like a damning word.

So one of my pet theories I haven't seen in general discourse is that AI came from the massive vector processing jump available commercially in GPUs when it left CPU bound processing behind. That's a factor of 100x-1000x of processing power.

AI is not-quite-there, and to get even another leap might take another 10-100x processing power.

Now... what? ASICs probably won't deliver even a 10x? There's only so much you get out of node shrinks.

"Substantial" doesn't even mean twice IMO. "Substantial" almost sounds like ... 15% better?

Re: OpenAI unveils its first custom chip, built by Broadcom

#475

Post by SoftTalker »

TeMPOraL wrote: post #458
Someone wrote: post #425

Earlier quoted context omitted.

> Or a military drone which needs to be jamming proof. That, if used in war, I would think, would need the ability to be updated frequently. For example, your enemy might find out (say by running tests on hardware they captured from you) that painting some red paint in a particular shape (a smiley might even work) on their hardware prevented your drones from attacking them because it confuses that pattern with the Re…

You keep the "reasoning core" burned and play the cat-and-mouse game at the I/O edge. Enemy invents a smiley shield, your R&D figures out some filtering step that defeats this effect without compromising general image recognition. Then the enemy figures out a new trick, your R&D invents a countermeasure, and so on - point is, this can happen for a long time in layers on top of the core model. If the enemy invents som…

Sort of mirrors how biological organisms work. E.g. in a bird, the core functionality of knowing how to fly is burned in. Hunting food is probably a combination of experiential learning on top of instinctive behavior, and is somewhat adaptable to local conditions.

Re: OpenAI unveils its first custom chip, built by Broadcom

#476

Post by 542354234235 »

Someone wrote: post #425
cm2187 wrote: post #417

Earlier quoted context omitted.

In fact any application where the task is stable and the model good enough to address that task. As you suggest, industrial applications where a robot must deal with variants of the same repetitive task. Or a military drone which needs to be jamming proof.

> Or a military drone which needs to be jamming proof. That, if used in war, I would think, would need the ability to be updated frequently. For example, your enemy might find out (say by running tests on hardware they captured from you) that painting some red paint in a particular shape (a smiley might even work) on their hardware prevented your drones from attacking them because it confuses that pattern with the Re…

Those are really two different things. One is the computer vision that could be “hard coded” and the other is the image library, that would be updated regularly. Look at facial recognition. You can download and run a facial recognition LLM on your GPU that looks at a library of your personal photos. The LLM doesn’t change when it scans your photos for faces, it just writes the data associated with a “face” to whatever library. When you add a new picture, it adds that face data and compares it to the library for a match. The actual LLM never needs to change. It is the same as the one I downloaded and ran on my GPU for my photos. If it was written on chips we both bought and installed, it would work the same way.[1]

[1] Yes, this is a massive simplification

Re: OpenAI unveils its first custom chip, built by Broadcom

#478

Post by markhahn »

reinitctxoffset wrote: post #298
selectodude wrote: post #257

Earlier quoted context omitted.

I think the folks at PA Semi had some chops too.

The way I heard it PA Semi was the singular driving force that led to Apple Silicon, but I'm not any kind of insider that's just the chatter I heard. Whoever it was, whooo, that's hot shit. I remember an M1 MacBook Air just cleaning the clock of an Intel MacBook Pro and thinking "x86_64 has real competition again". Great silicon. I'm over it with not having root on my own machine, so I've left the ecosystem, but it's…

it would be interesting to know apple's true/inside attitude towards people putting linux on their hardware. they don't seem very interested in helping, but donno whether they actively sabotage either.

Re: OpenAI unveils its first custom chip, built by Broadcom

#479

Post by riknos314 »

epolanski wrote: post #286
wmf wrote: post #270

Earlier quoted context omitted.

If GPUs have 75% margin then 50% cheaper is no surprise.

Operational costs far outweight hardware cost.

Let's use an example of a GW AI deployment.

At $0.07/kWh, that costs $70,000 every hour in just electricity. $1.7 million /day. $613 million /year.

I had claude estimate the GPU cost of such a deployment:

> To get racks per GW: a full NVL72 rack draws roughly 130-132 kW under full load. If a 1 GW facility runs ~715 MW of IT power (after a ~1.4 PUE for cooling), that's on the order of 4,000–4,500 racks. At $3.4M of compute hardware each, the GPU-system cost lands around $14–15 billion.

15 billion / 613 million / year = ~24.5 years til electricity costs catch up to the GPUs. Obviously electricity isn't 100% of OpEx, but I'd expect it to be the majority for AI deployments.

Regardless, if you can cut the $613 million/yr in half that's still massive savings.

Re: OpenAI unveils its first custom chip, built by Broadcom

#480

Post by knicholes »

chamomeal wrote: post #376
knicholes wrote: post #129

Earlier quoted context omitted.

I tried making a button using Claude entirely (including the 3D printed enclosure) and it effed up pretty hard with the traces and the header spacing. The project was a big red arcade button that plays the "ah-my-groin.mp3" when pushed (from Simpsons). It did cool work on saving battery life, and the 3d enclosure was awesome, but yeah, I'm convinced I'd have to do another version or two of the custom chip until it ca…

Sounds like a super cool project. Gonna post the design anywhere?

I'll update it this weekend with the updated AI-generated fun (and correct the flat-out ai-generated lies in the README). Meanwhile, you can see the project here. https://github.com/knicholes/ah-my-groin-button
Post reply on HN