How much have they optimized the software here? Is it tinygrad level optimization?
Also does this lower total cost depend on SRAM being available for DRAM prices?
What makes SRAM so much more expensive than DRAM?
Chiplet ASIC supercomputers for LLMs like GPT-4
31–40 of 91 posts
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#32How much have they optimized the software here? Is it tinygrad level optimization? Also does this lower total cost depend on SRAM being available for DRAM prices? What makes SRAM so much more expensive than DRAM?
[deleted]
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#33How much have they optimized the software here? Is it tinygrad level optimization? Also does this lower total cost depend on SRAM being available for DRAM prices? What makes SRAM so much more expensive than DRAM?
[deleted]
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#34How much have they optimized the software here? Is it tinygrad level optimization? Also does this lower total cost depend on SRAM being available for DRAM prices? What makes SRAM so much more expensive than DRAM?
SRAM uses multiple transistors and takes up a lot more space than DRAM so it is inherently more expensive because it needs more area. The advantage is that it is fast and doesn't need refreshing like DRAM does. You can also put it on the same die as your computation logic which is technically possible with DRAM, but kinda silly since you need an optimized process to get the best out of that. This process is then quite bad for high-speed logic.
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#35Earlier quoted context omitted.
Large language models would need tens or hundreds of gigabytes of SRAM. Pretty sure the enormous cost for this makes the approach economically unfeasible.
Cerebras Wafer Scale Engine has 40GB of onboard SRAM using TSMC 7nm. It uses the entire wafer as the chip. Costs millions per chip though. Source: https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
Wafer scale integration is a dead end technology. The engineering issues are just too great.
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#36Earlier quoted context omitted.
Cerebras Wafer Scale Engine has 40GB of onboard SRAM using TSMC 7nm. It uses the entire wafer as the chip. Costs millions per chip though. Source: https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
Wafer scale integration is a dead end technology. The engineering issues are just too great.
Do you have a source?
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#37Earlier quoted context omitted.
Wafer scale integration is a dead end technology. The engineering issues are just too great.
Do you have a source?
At high end production nodes it is impossible to get an entire wafer free of defects. Chips, let alone wafers, already include circuitry to disable parts of themselves if those parts have defects. Cerebras must have spent a ton of effort on getting this done for a full wafer. You also have problems like variability at the wafer level which you're less sensitive too when you put thousands of chips on a single wafer rather than just one, since they cover a smaller area.
Look at how successful AMD's chiplet strategy has been. Chiplets sidestep the yield problems. Wafer scale amplifies them hundred or thousand fold.
Nothing in the industry is designed to work with wafer scale products, so everything has to be custom made. Yes this is a chicken-egg problem, but it's going to be expensive to get any sort of momentum. The silicon industry is extremely conservative.
It's sexy and enticing. If someone can make it work that's awesome. I will remain skeptical though.
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#38This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serv…
Doesn’t that graph have a toks/sec of 18? Or am I reading it wrong
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#39Can anyone share how much SRAM their proposed ASIC actually has? I skimmed the paper but that number didn't jump out to me.
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#40Earlier quoted context omitted.
Large language models would need tens or hundreds of gigabytes of SRAM. Pretty sure the enormous cost for this makes the approach economically unfeasible.
Cerebras Wafer Scale Engine has 40GB of onboard SRAM using TSMC 7nm. It uses the entire wafer as the chip. Costs millions per chip though. Source: https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
40 GB wouldn't even be enough for LLaMA 65B with 8 bit quantization. Let alone something like GPT-4.