4T transistors, one giant chip (Cerebras WSE-3) [video] • Hacker News Board

Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]

#31

Post by cs702 » Wed, Mar 13, 2024, 6:10 PM UTC

According to the company, the new chip will enable training of AI models with up to 24 trillion parameters. Let me repeat that, in case you're as excited as I am: 24. Trillion. Parameters. For comparison, the largest AI models currently in use have around 0.5 trillion parameters, around 48x times smaller.

Each parameter is a connection between artificial neurons. For example, inside an AI model, a linear layer that transforms an input vector with 1024 elements to an output vector with 2048 elements has 1024×2048 = ~2M parameters in a weight matrix. Each parameter specifies by how much each element in the input vector contributes to or subtracts from each element in the output vector. Each output vector element is a weighted sum (AKA a linear combination), of each input vector element.

A human brain has an estimated 100-500 trillion synapses connecting biological neurons. Each synapse is quite a complicated biological structure[a], but if we oversimplify things and assume that every synapse can be modeled as a single parameter in a weight matrix, then the largest AI models in use today have approximately 100T to 500T ÷ 0.5T = 200x to 1000x fewer connections between neurons that the human brain. If the company's claims prove true, this new chip will enable training of AI models that have only 4x to 20x fewer connections that the human brain.

We sure live in interesting times!

---

[a] https://en.wikipedia.org/wiki/Synapse

Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]

#32

Post by Rexxar » Wed, Mar 13, 2024, 6:10 PM UTC

Is there a reason it's not roughly a disc if they use the whole wafer ? They could have 50% more surface.

Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]

#33

Post by mlyle » Wed, Mar 13, 2024, 6:15 PM UTC

cs702 wrote: post #31

According to the company, the new chip will enable training of AI models with up to 24 trillion parameters. Let me repeat that, in case you're as excited as I am: 24. Trillion. Parameters. For comparison, the largest AI models currently in use have around 0.5 trillion parameters, around 48x times smaller. Each parameter is a connection between artificial neurons . For example, inside an AI model, a linear layer that…

> but if we oversimplify things and assume that every synapse can be modeled as a single parameter in a weight matrix

Which, it probably can't... but offsetting those simplifications and 4-20x difference is the massive difference in how quickly those synapses can be activated.

Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]

#34

Post by tivert » Wed, Mar 13, 2024, 6:15 PM UTC

eternauta3k wrote: post #23

tivert wrote: post #10

Interesting. I know there's a lot of attempts to hobble China by limiting their access to cutting edge chips and semiconductor manufacturing technology, but could something like this be a workaround for them, at least for datacenter-type jobs? Maybe it wouldn't be as powerful as one of these, due to their less capable fabs, but something that's good enough to get the job done in spite of the embargoes.

What do you mean by "this", and how does it work around the restrictions? Do you mean just making bigger chips instead of shrinking the transistors?

> Do you mean just making bigger chips instead of shrinking the transistors?

Yes.

Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]

#35

Post by RetroTechie » Wed, Mar 13, 2024, 6:31 PM UTC

If you were to add up all transistors fabricated worldwide, up until , such that total roughly matches the # on this beast, what year would you arrive? Hell, throw in discrete transistors if you want.

How many early supercomputers / workstations etc would that include? How much progress did humanity make using all those early machines (or any transistorized device!) combined?

Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]

#36

Post by holoduke » Wed, Mar 13, 2024, 6:37 PM UTC

Better sell all nvidia stocks. Once these chips are common there is no need anymore for GPUs in training super large AI models.

Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]

#37

Post by [deleted] » Wed, Mar 13, 2024, 6:40 PM UTC

AdamH12113 wrote: post #4

Title should be either "4,000,000,000,000 Transistors" (as in the actual video title) or "4 Trillion Transistors" or maybe "4T Transistors". "4B,000" ("four billion thousand"?) looks like 48,000 (forty-eight thousand).

[deleted]

Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]

#38

Post by gosub100 » Wed, Mar 13, 2024, 6:45 PM UTC

ortusdux wrote: post #3

Not trying to sound critical, but is there a reason to use 4B,000 vs 4T?

at least a trillion means a trillion. Unlike the "tebi vs tera"-byte marketing-speak in storage and ram.

Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]

#39

Post by nachexnachex » Wed, Mar 13, 2024, 6:55 PM UTC

dougmwne wrote: post #30

brucethemoose2 wrote: post #7

Reposting the CS-2 teardown in case anyone missed it. The thermal and electrical engineering is absolutely nuts: https://vimeo.com/853557623 https://web.archive.org/web/20230812020202/https://www.youtu... (Vimeo/Archive because the original video was taken down from YouTube)

I want this woman running my postAI-apocalypse hardware research lab.

She's got strong merit to be spared by the Roko's basilisk ai.

Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]

#40

Post by pmontra » Wed, Mar 13, 2024, 7:01 PM UTC

mlhpdx wrote: post #29

wizardforhire wrote: post #26

But can it run doom?

Does it come in a mobile/laptop version?

It's 215 x 215 mm so it fits in a large laptop, some 15" and definitely 17" ones. The keyboard could get a little warm and battery life doesn't look good.