Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Chiplet ASIC supercomputers for LLMs like GPT-4

arxiv.org

Post reply on HN
Print view View original

61–70 of 91 posts

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#61

Post by why_only_15 »

chessgecko wrote: post #38
why_only_15 wrote: post #8

This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serv…

Doesn’t that graph have a toks/sec of 18? Or am I reading it wrong

It shows 18 tokens per second but that's how fast tokens are generated I think. The number of tokens generated is that times the batch size, which appears to be 12? The graph is quite unclear and I didn't feel like reading the paper more in-depth.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#62

Post by jacquesm »

waffletower wrote: post #54
jacquesm wrote: post #13

Earlier quoted context omitted.

It makes you wonder if those singularity proponents don't have a point, and it all depends on whether it keeps accelerating or whether it will slow down again. I hope for the latter and I fear for the former. Even if it does slow down eventually a long enough period of such change is going to make the industrial revolution (whose negative effects we are still coming to terms with today!) like a walk in the park.

A so-called singularity would require accelerated development for many more technological spheres, not just semiconductor fabrication, and related information computation and AI advances. Logistics, supply chains, mining, farming, manufacturing, energy, biotechnology. While the former may continue to accelerate development in the latter categories, the scale of such impact is purely speculative. I don't believe the a…

Material science and genetics are prime candidates for a different approach and AI techniques are already paying off dividends in those domains.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#63

Post by punkgenius »

why_only_15 wrote: post #8

This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serv…

Looks like figure 8 of paper [1] says it is 18 tokens/s

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#64

Post by kraken12 »

cubefox wrote: post #19
alain94040 wrote: post #7

The key point: A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs ... On-chip memories such as SRAM have better read latency and read/write energy than external memories such as DDR or HBM but require more si…

Large language models would need tens or hundreds of gigabytes of SRAM. Pretty sure the enormous cost for this makes the approach economically unfeasible.

They are using many chips and taking advantage of the way data flows in LLMs to make it work; so it would be cost-effective unlike Cerebras

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#65

Post by kraken12 »

fzliu wrote: post #11

Just skimmed the paper. Seems to me like this paper wants to optimize transformer inference e2e, i.e. from ASIC level all the way to cloud. I'm not exactly convinced though, since all the results seem to be purely theoretical or simulated. I would've liked to see a prototype built across several FPGAs with clock speeds extrapolated for ASICs.

Yep, it's a research paper in comp arch, the initial proof-of-concept study before you go and spend real money on it.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#66

Post by kraken12 »

why_only_15 wrote: post #61
chessgecko wrote: post #38

Earlier quoted context omitted.

Doesn’t that graph have a toks/sec of 18? Or am I reading it wrong

It shows 18 tokens per second but that's how fast tokens are generated I think. The number of tokens generated is that times the batch size, which appears to be 12? The graph is quite unclear and I didn't feel like reading the paper more in-depth.

Seems to me the 18 tokens per second from [1] is the throughput and includes the batch size, so I don't think they misread the Deepspeed inference paper. So the chiplet ASIC supercomputer paper would seem to show a decent performance/TCO benefit.

Of course, it's a first architectural study to illustrate the promise of the idea, lots more details to work out in a physical implementation and the final realized benefit is likely to be lower. But even a 3X is huge in this space.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#67

Post by kraken12 »

jacquesm wrote: post #44
bhouston wrote: post #28

Earlier quoted context omitted.

Another list that shows pricing both constant and spot. The best GCP spot price is $1.1, but Jarvis seems to say its spot for the 40GB A100 is $0.69: https://fullstackdeeplearning.com/cloud-gpus/ I feel there are more fair criticisms of that paper than its inclusion of the snapshot price of variable priced compute resource.

Sure, to me it more of an extra item than the main one but it is one that you can readily verify because most of the other claims are far more vague. If they're willing to fudge on that one then I have much less confidence in the rest of their claims.

Seems like HN comments have determined that the cost number is not fudged..

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#68

Post by kraken12 »

senttoschool wrote: post #22
alain94040 wrote: post #7

The key point: A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs ... On-chip memories such as SRAM have better read latency and read/write energy than external memories such as DDR or HBM but require more si…

SRAM has stopped scaling based on TSMC's upcoming N3E specs and their planned N2 node specs. So if models are tens of GB large, then I don't see how their proposed chips can be done in an economical way. Also, a GPU is already an ASIC but with a fancy name.

Maybe they could do something like AMD's GPU memory stacking, that is good for scaling, and of course they are using many chips not one chip..

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#69

Post by kraken12 »

jacquesm wrote: post #10
SCUSKU wrote: post #3

94x cost improvement over GPU and 15x TPU is insane, but fits right in there with performance gains seen in Moore's Law. This development presents a more compelling case that we are in fact on the precipice of larger LLMs being able to serve everyone for cheap. Still not really convinced by the AGI argument, but this does spook me. Overall though very cool.

It's insane because it is theoretical. They haven't shown that it works, think of this paper as a prelude to a funding round or research grant so they have to show some kind of advantage. Which I'm highly skeptical of, usually when papers show this kind of improvement over SOTA it tends to be either a mistake or purposeful nonsense.

Yeah, a preliminary architectural study to sanity check if an idea could potentially pay off.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#70

Post by kraken12 »

jacquesm wrote: post #12
why_only_15 wrote: post #8

This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serv…

The whole thing is imaginary: "In this paper, we propose Chiplet Cloud, a chiplet-based ASIC AI-supercomputer architecture that optimizes total cost of ownership (TCO) per generated token for serving large generative language models to reduce the overall cost to deploy and run these applica- tions in the real world." So they are comparing actual implementations with a theoretical implementation. Never mind that they…

Yeah, it is an architectural simulation study, this is what is usually done right at the beginning before resources are allocated to go deep on idea. So in that sense it is imaginary; but this is how new ideas get incubated.
Post reply on HN