Near-100% yield is some dark magic.
4T transistors, one giant chip (Cerebras WSE-3) [video]
81–90 of 93 posts
Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]
#82As I understand it, WSE-2 was kind of handicapped because its performance could only really be harnessed if the neural net fit in the on-chip SRAM. Bandwidth to off-chip memory (normalized to FLOPS) was not as high as Nvidia. Is that improved with WSE-3? Seems like the SRAM is only 10% bigger, so that's not helping.
In the days before LLMs 44 GB of SRAM sounded like a lot, but these days it's practically nothing. It's possible that novel architectures could be built for Cerebras that leverage the unique capabilities, but the inaccessibility of the hardware is a problem. So few people will ever get to play with one that it's unlikely new architectures will be developed for it.
Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]
#83Earlier quoted context omitted.
This is something I'm clueless about and can't really understand. They say this is 24kW hungry. How does CPU power consumption really work on electrical level, what warrants that much power, even for regular CPUs? Like, from the basics level.. is it resistance of the material with frequency of switching or what is really going on there? Where does the power go on such a relatively small surface? edit: thanks people,…
Yes, the power consumption comes from the resistance of the circuit. In case of CMOS circuits there would ideally be no current flow when no signal changes but transistors are not perfect and have leakage currents. When signals change, primarily triggered by the clock rising or falling, there is a short time in which the supply rail and ground rail are essentially shorted out. Each gate has logically two transistors…
delete
Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]
#84Earlier quoted context omitted.
Yes, the power consumption comes from the resistance of the circuit. In case of CMOS circuits there would ideally be no current flow when no signal changes but transistors are not perfect and have leakage currents. When signals change, primarily triggered by the clock rising or falling, there is a short time in which the supply rail and ground rail are essentially shorted out. Each gate has logically two transistors…
delete
Does modern logic mean sub-threshold logic or does this apply generally?
Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]
#85As I understand it, WSE-2 was kind of handicapped because its performance could only really be harnessed if the neural net fit in the on-chip SRAM. Bandwidth to off-chip memory (normalized to FLOPS) was not as high as Nvidia. Is that improved with WSE-3? Seems like the SRAM is only 10% bigger, so that's not helping. In the days before LLMs 44 GB of SRAM sounded like a lot, but these days it's practically nothing. It'…
As I understand it, the WSE-2's interconnect is actually quite good, and models are split across chips kinda like GPUs.
And keep in mind that these nodes are hilariously "fat" compared to a GPU node (or even an 8x GPU node), meaning less congestion and overhead from the topology.
Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]
#86Earlier quoted context omitted.
Yes, the power consumption comes from the resistance of the circuit. In case of CMOS circuits there would ideally be no current flow when no signal changes but transistors are not perfect and have leakage currents. When signals change, primarily triggered by the clock rising or falling, there is a short time in which the supply rail and ground rail are essentially shorted out. Each gate has logically two transistors…
delete
This is completely wrong. I've desiged ICs from 350nm down to 3nm technology. I don't know what do you mean by transistors are biased. If you mean SOI body bissing, it's not a common technology.
Crowbar current is still a big problem. Leakage of Finfets are crazy, especially when you use ulvt devices, which you need for high performance.
The previous post is fully correct. Though the bulk of the power is switching capacitor.
Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]
#87According to the company, the new chip will enable training of AI models with up to 24 trillion parameters. Let me repeat that, in case you're as excited as I am: 24. Trillion. Parameters. For comparison, the largest AI models currently in use have around 0.5 trillion parameters, around 48x times smaller. Each parameter is a connection between artificial neurons . For example, inside an AI model, a linear layer that…
Fun fact, I can also train a 24 trillion parameter model on my laptop! Just need to offload weights to the cloud every layer. ... It's meaningless to say something can train a model that has 24 trillion parameters without specifying the dataset size and time it takes to train.
I dare say this thing will be many times faster than thrashing your 24T parameters to the cloud.
Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]
#88Earlier quoted context omitted.
20,000 amps 200,000 electrical contacts 850,000 cores and that's the "old" one. wow.
This is something I'm clueless about and can't really understand. They say this is 24kW hungry. How does CPU power consumption really work on electrical level, what warrants that much power, even for regular CPUs? Like, from the basics level.. is it resistance of the material with frequency of switching or what is really going on there? Where does the power go on such a relatively small surface? edit: thanks people,…
a lot of good answers. just couple points to add:
- Charging a capacitor from 0 to Vdd volts takes E=CVdd^2 amount of energy. To find the power (energy/second) you need to multiply this with how frequently you do this, i.e. clock frequency. So, P=FCVdd^2. So most digitak circuits power scales linearly with frequency, and quadratically with power supply voltage Vdd.
- Half of themis power is resistive loses. This is kinda a fundamental thing. Charging a capacitor with a resistor is "lossy".
- Some of the charge is reused within the circuits
Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]
#89As I understand it, WSE-2 was kind of handicapped because its performance could only really be harnessed if the neural net fit in the on-chip SRAM. Bandwidth to off-chip memory (normalized to FLOPS) was not as high as Nvidia. Is that improved with WSE-3? Seems like the SRAM is only 10% bigger, so that's not helping. In the days before LLMs 44 GB of SRAM sounded like a lot, but these days it's practically nothing. It'…
That was more of a WSE-1 problem maybe? They switched to a new compute paradigm (details on their site if you look up "weight streaming") where they basically store the activation on the wafer instead of the whole model. For something very large (say, 32K context and 16k hidden dimension) this would make an activation layer only 1-2GB (16 bit or 32 bit). As I understand it, this was one of the key changes needed to go from single system boxes to these super computing clusters they have been able to deploy.
The Nvidia bandwidth to compute ratio is more necessary because they are moving things around all the time. By keeping all the outputs on the wafer and only streaming the weights, you have a much more favorable requirement for BW to compute. And the number of layers becomes less impactful because they are storing transient outputs.
This is probably one of the primary reasons they didn't need to increase SRAM for WSE-3. WSE-2 was developed based on the old "fit the whole model on the chip" paradigm but models eclipsed 1TB so the new solution is more scalable.
Re: 4T transistors, one giant chip (Cerebras WSE-3) [video]
#90Earlier quoted context omitted.
Fun fact, I can also train a 24 trillion parameter model on my laptop! Just need to offload weights to the cloud every layer. ... It's meaningless to say something can train a model that has 24 trillion parameters without specifying the dataset size and time it takes to train.
I dare say this thing will be many times faster than thrashing your 24T parameters to the cloud.
Yeah, but it'll be slower than the equivalent Nvidia GPU cluster.