Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Chiplet ASIC supercomputers for LLMs like GPT-4

arxiv.org

Post reply on HN
Print view View original

41–50 of 91 posts

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#41

Post by cubefox »

skummetmaelk wrote: post #37
senttoschool wrote: post #36

Earlier quoted context omitted.

Do you have a source?

At high end production nodes it is impossible to get an entire wafer free of defects. Chips, let alone wafers, already include circuitry to disable parts of themselves if those parts have defects. Cerebras must have spent a ton of effort on getting this done for a full wafer. You also have problems like variability at the wafer level which you're less sensitive too when you put thousands of chips on a single wafer ra…

> Cerebras achieves 100% yield by designing a system in which any manufacturing defect can be bypassed – initially Cerebras had 1.5% extra cores to allow for defects, but we’ve since been told this was way too much as TSMC's process is so mature.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#42

Post by senttoschool »

skummetmaelk wrote: post #37
senttoschool wrote: post #36

Earlier quoted context omitted.

Do you have a source?

At high end production nodes it is impossible to get an entire wafer free of defects. Chips, let alone wafers, already include circuitry to disable parts of themselves if those parts have defects. Cerebras must have spent a ton of effort on getting this done for a full wafer. You also have problems like variability at the wafer level which you're less sensitive too when you put thousands of chips on a single wafer ra…

I'm well aware of chip defect rates and how they affect chips.

>It's sexy and enticing. If someone can make it work that's awesome. I will remain skeptical though.

But Cerebras has made it work since 2019 as @cubefox pointed out. They're on the second generation already and they have been shipping to customers for years.

Here's a good overview of how they did it: https://www.anandtech.com/show/14758/hot-chips-31-live-blogs...

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#43

Post by ilaksh »

skummetmaelk wrote: post #34
ilaksh wrote: post #31

How much have they optimized the software here? Is it tinygrad level optimization? Also does this lower total cost depend on SRAM being available for DRAM prices? What makes SRAM so much more expensive than DRAM?

SRAM uses multiple transistors and takes up a lot more space than DRAM so it is inherently more expensive because it needs more area. The advantage is that it is fast and doesn't need refreshing like DRAM does. You can also put it on the same die as your computation logic which is technically possible with DRAM, but kinda silly since you need an optimized process to get the best out of that. This process is then quit…

Do you think the price of including SRAM might be reduced somewhat if the big foundries optimize for including lots of SRAM in these types of ASICs?

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#44

Post by jacquesm »

bhouston wrote: post #28
jacquesm wrote: post #26

Earlier quoted context omitted.

> The $0.87/hour price you gave is theoretical https://cloud.google.com/blog/products/compute/a2-vms-with-n... That's as close as I got to verifying that price.

Another list that shows pricing both constant and spot. The best GCP spot price is $1.1, but Jarvis seems to say its spot for the 40GB A100 is $0.69: https://fullstackdeeplearning.com/cloud-gpus/ I feel there are more fair criticisms of that paper than its inclusion of the snapshot price of variable priced compute resource.

Sure, to me it more of an extra item than the main one but it is one that you can readily verify because most of the other claims are far more vague. If they're willing to fudge on that one then I have much less confidence in the rest of their claims.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#46

Post by emmender »

jacquesm wrote: post #12
why_only_15 wrote: post #8

This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serv…

The whole thing is imaginary: "In this paper, we propose Chiplet Cloud, a chiplet-based ASIC AI-supercomputer architecture that optimizes total cost of ownership (TCO) per generated token for serving large generative language models to reduce the overall cost to deploy and run these applica- tions in the real world." So they are comparing actual implementations with a theoretical implementation. Never mind that they…

take 3 ideas that are hot: chiplets, cloud, and LLM - remix them into the title of a paper that describes a hypothetical machine.. academia playing catch up and trying to stay relevant in my cynical eye.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#47

Post by tmccrary55 »

emmender wrote: post #46
jacquesm wrote: post #12

Earlier quoted context omitted.

The whole thing is imaginary: "In this paper, we propose Chiplet Cloud, a chiplet-based ASIC AI-supercomputer architecture that optimizes total cost of ownership (TCO) per generated token for serving large generative language models to reduce the overall cost to deploy and run these applica- tions in the real world." So they are comparing actual implementations with a theoretical implementation. Never mind that they…

take 3 ideas that are hot: chiplets, cloud, and LLM - remix them into the title of a paper that describes a hypothetical machine.. academia playing catch up and trying to stay relevant in my cynical eye.

Using ChatGPT

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#48

Post by woah »

btbuildem wrote: post #45

When the LLM wave first burst into public consciousness, I hoped that people would find a way to repurpose all the crypto-mining hardware for this -- alas, a different set of problems.

ASIC stands for Application Specific Integrated Circuit. So by definition, they cannot be repurposed.
Post reply on HN