Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Chiplet ASIC supercomputers for LLMs like GPT-4

arxiv.org

Post reply on HN
Print view View original

21–30 of 91 posts

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#21

Post by londons_explore »

fzliu wrote: post #11

Just skimmed the paper. Seems to me like this paper wants to optimize transformer inference e2e, i.e. from ASIC level all the way to cloud. I'm not exactly convinced though, since all the results seem to be purely theoretical or simulated. I would've liked to see a prototype built across several FPGAs with clock speeds extrapolated for ASICs.

It seems fine to say "others have proved that this math makes a good LLM, we have designed an ASIC that can do this math fast, therefore we can make a good fast LLM"

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#22

Post by senttoschool »

alain94040 wrote: post #7

The key point: A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs ... On-chip memories such as SRAM have better read latency and read/write energy than external memories such as DDR or HBM but require more si…

SRAM has stopped scaling based on TSMC's upcoming N3E specs and their planned N2 node specs. So if models are tens of GB large, then I don't see how their proposed chips can be done in an economical way.

Also, a GPU is already an ASIC but with a fancy name.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#23

Post by bhouston »

jacquesm wrote: post #12
why_only_15 wrote: post #8

This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serv…

The whole thing is imaginary: "In this paper, we propose Chiplet Cloud, a chiplet-based ASIC AI-supercomputer architecture that optimizes total cost of ownership (TCO) per generated token for serving large generative language models to reduce the overall cost to deploy and run these applica- tions in the real world." So they are comparing actual implementations with a theoretical implementation. Never mind that they…

The $0.87/hour price you gave is theoretical and also we know any price in a paper for compute is wrong by the time of publication.

Pragmatically the prices are closer to $2/hr according to this recent post here on Hacker News: https://llm-utils.org/Nvidia+H100+and+A100+GPUs+-+comparing+...

Although again prices change on a daily basis on spot providers.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#24

Post by JonChesterfield »

alain94040 wrote: post #7

The key point: A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs ... On-chip memories such as SRAM have better read latency and read/write energy than external memories such as DDR or HBM but require more si…

Graphcore got up to a gigabyte or so of on chip memory with the same plan of keeping the model in that memory. Does work really well if the data fits.

Recent x64 chips are at about that amount of L3 cache which might be pretty similar. I've lost track of GPU hardware specs.

That proper hardware software co-design to mitigate communication? Viciously difficult bordering on imaginary.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#25

Post by JonChesterfield »

londons_explore wrote: post #21
fzliu wrote: post #11

Just skimmed the paper. Seems to me like this paper wants to optimize transformer inference e2e, i.e. from ASIC level all the way to cloud. I'm not exactly convinced though, since all the results seem to be purely theoretical or simulated. I would've liked to see a prototype built across several FPGAs with clock speeds extrapolated for ASICs.

It seems fine to say "others have proved that this math makes a good LLM, we have designed an ASIC that can do this math fast, therefore we can make a good fast LLM"

Yes, but saying that shouldn't be mistaken for "we can make an asic that runs some model fast". There's a wide implementation void between the two.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#26

Post by jacquesm »

bhouston wrote: post #23
jacquesm wrote: post #12

Earlier quoted context omitted.

The whole thing is imaginary: "In this paper, we propose Chiplet Cloud, a chiplet-based ASIC AI-supercomputer architecture that optimizes total cost of ownership (TCO) per generated token for serving large generative language models to reduce the overall cost to deploy and run these applica- tions in the real world." So they are comparing actual implementations with a theoretical implementation. Never mind that they…

The $0.87/hour price you gave is theoretical and also we know any price in a paper for compute is wrong by the time of publication. Pragmatically the prices are closer to $2/hr according to this recent post here on Hacker News: https://llm-utils.org/Nvidia+H100+and+A100+GPUs+-+comparing+... Although again prices change on a daily basis on spot providers.

> The $0.87/hour price you gave is theoretical

https://cloud.google.com/blog/products/compute/a2-vms-with-n...

That's as close as I got to verifying that price.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#27

Post by senttoschool »

cubefox wrote: post #19
alain94040 wrote: post #7

The key point: A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs ... On-chip memories such as SRAM have better read latency and read/write energy than external memories such as DDR or HBM but require more si…

Large language models would need tens or hundreds of gigabytes of SRAM. Pretty sure the enormous cost for this makes the approach economically unfeasible.

Cerebras Wafer Scale Engine has 40GB of onboard SRAM using TSMC 7nm. It uses the entire wafer as the chip.

Costs millions per chip though.

Source: https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#28

Post by bhouston »

jacquesm wrote: post #26
bhouston wrote: post #23

Earlier quoted context omitted.

The $0.87/hour price you gave is theoretical and also we know any price in a paper for compute is wrong by the time of publication. Pragmatically the prices are closer to $2/hr according to this recent post here on Hacker News: https://llm-utils.org/Nvidia+H100+and+A100+GPUs+-+comparing+... Although again prices change on a daily basis on spot providers.

> The $0.87/hour price you gave is theoretical https://cloud.google.com/blog/products/compute/a2-vms-with-n... That's as close as I got to verifying that price.

Another list that shows pricing both constant and spot. The best GCP spot price is $1.1, but Jarvis seems to say its spot for the 40GB A100 is $0.69:

https://fullstackdeeplearning.com/cloud-gpus/

I feel there are more fair criticisms of that paper than its inclusion of the snapshot price of variable priced compute resource.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#30

Post by beecafe »

why_only_15 wrote: post #8

This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serv…

[dead]
Post reply on HN