Chiplet ASIC supercomputers for LLMs like GPT-4 • Hacker News Board

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#71

Post by why_only_15 » Wed, Jul 12, 2023, 7:20 PM UTC

kraken12 wrote: post #66

why_only_15 wrote: post #61

Earlier quoted context omitted.

It shows 18 tokens per second but that's how fast tokens are generated I think. The number of tokens generated is that times the batch size, which appears to be 12? The graph is quite unclear and I didn't feel like reading the paper more in-depth.

Seems to me the 18 tokens per second from [1] is the throughput and includes the batch size, so I don't think they misread the Deepspeed inference paper. So the chiplet ASIC supercomputer paper would seem to show a decent performance/TCO benefit. Of course, it's a first architectural study to illustrate the promise of the idea, lots more details to work out in a physical implementation and the final realized benefit…

If you look at the "metrics" section on Page 9, it says:

> 2) Metrics: We use three performance metrics: (i) latency, i.e., end-to-end output generation time for a batch of input prompts, (ii) token throughput, i.e., tokens-per-second processed, and (iii) compute throughput, i.e., TFLOPS per GPU.

This is somewhat confusing to me because at least two of these three definitions should be essentially the same thing, but I don't think there's any way to interpret their claim of ~74 teraflops achieved other than ~211 tokens/second of throughput.

Put another way, 18 tokens per second is 2% flops utilization, which we are obviously capable of doing better than for bulk inference.

3x is not huge in this space because just using a 4090 instead of an A100 is a 5x gain.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#72

Post by saturn99 » Wed, Jul 12, 2023, 8:54 PM UTC

why_only_15 wrote: post #8

This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serv…

I think the costs are from the Moonwalk model, which is a pretty good reference for estimating costs, although it might be low if you use all Google engineers to build the HW. =P

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#73

Post by punkgenius » Wed, Jul 12, 2023, 8:57 PM UTC

why_only_15 wrote: post #71

kraken12 wrote: post #66

Earlier quoted context omitted.

Seems to me the 18 tokens per second from [1] is the throughput and includes the batch size, so I don't think they misread the Deepspeed inference paper. So the chiplet ASIC supercomputer paper would seem to show a decent performance/TCO benefit. Of course, it's a first architectural study to illustrate the promise of the idea, lots more details to work out in a physical implementation and the final realized benefit…

If you look at the "metrics" section on Page 9, it says: > 2) Metrics: We use three performance metrics: (i) latency, i.e., end-to-end output generation time for a batch of input prompts, (ii) token throughput, i.e., tokens-per-second processed, and (iii) compute throughput, i.e., TFLOPS per GPU. This is somewhat confusing to me because at least two of these three definitions should be essentially the same thing, but…

Maybe 74 tflops is the best they've achieved, but not all 16 GPUs can consistently hit that number? Just guessing.. The 211 tokens/sec throughput on GPU is just insane, it's even better than what TPU can do on PaLM 540B.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#74

Post by saturn99 » Wed, Jul 12, 2023, 9:11 PM UTC

boredumb wrote: post #55

Was expecting something else I suppose. Is this kind of stuff for potential future investors?

Yep, pretty unique system. I imagine anything that runs LLMs better than we currently can would pique the interest of VCs.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#75

Post by yvn1uo » Wed, Jul 12, 2023, 9:18 PM UTC

msoad wrote: post #4

They claim the investment will be justified for a 1.5 year life span of the system. But LLMs are changing and improving at a much faster speed that 1.5 years feels like centuries!

1.5 years is actually not that bad. In fact, all changes and improvements to LLMs since the original Transformer paper is just the size -- tensor dimension, layers, etc. GPT-3, which is still widely used today, was proposed more than 3 years ago.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#76

Post by kraken12 » Wed, Jul 12, 2023, 9:29 PM UTC

why_only_15 wrote: post #71

kraken12 wrote: post #66

Earlier quoted context omitted.

Seems to me the 18 tokens per second from [1] is the throughput and includes the batch size, so I don't think they misread the Deepspeed inference paper. So the chiplet ASIC supercomputer paper would seem to show a decent performance/TCO benefit. Of course, it's a first architectural study to illustrate the promise of the idea, lots more details to work out in a physical implementation and the final realized benefit…

If you look at the "metrics" section on Page 9, it says: > 2) Metrics: We use three performance metrics: (i) latency, i.e., end-to-end output generation time for a batch of input prompts, (ii) token throughput, i.e., tokens-per-second processed, and (iii) compute throughput, i.e., TFLOPS per GPU. This is somewhat confusing to me because at least two of these three definitions should be essentially the same thing, but…

The 4090 has half the memory bandwidth, so it could not get a 5X gain, it would actually run slower on a memory bound LLM like this.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#77

Post by punkgenius » Wed, Jul 12, 2023, 9:51 PM UTC

senttoschool wrote: post #22

alain94040 wrote: post #7

The key point: A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs ... On-chip memories such as SRAM have better read latency and read/write energy than external memories such as DDR or HBM but require more si…

SRAM has stopped scaling based on TSMC's upcoming N3E specs and their planned N2 node specs. So if models are tens of GB large, then I don't see how their proposed chips can be done in an economical way. Also, a GPU is already an ASIC but with a fancy name.

Nowadays GPUs have sacrificed some performance for better programmability. ASICs always trade programmability for better performance and energy efficiency, it's really about how 'specific' you want it to be. I guess for applications as important and popular as LLM, we probably want a very 'specific' chip

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#78

Post by saturn99 » Wed, Jul 12, 2023, 10:06 PM UTC

msoad wrote: post #4

They claim the investment will be justified for a 1.5 year life span of the system. But LLMs are changing and improving at a much faster speed that 1.5 years feels like centuries!

Seems to align with Google, seems like they release a new TPU every 1.5 to 2 years.

I think modern LLMs are powerful enough now that they will still be useful in a couple years even if they aren't state-of-the-art. ChatGPT still lets you run their older model for cheaper than running GPT4, I could see a world where GPT4 is still available in 1.5 years even if there are better models out there.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#79

Post by yvn1uo » Wed, Jul 12, 2023, 10:11 PM UTC

why_only_15 wrote: post #71

kraken12 wrote: post #66

Earlier quoted context omitted.

Seems to me the 18 tokens per second from [1] is the throughput and includes the batch size, so I don't think they misread the Deepspeed inference paper. So the chiplet ASIC supercomputer paper would seem to show a decent performance/TCO benefit. Of course, it's a first architectural study to illustrate the promise of the idea, lots more details to work out in a physical implementation and the final realized benefit…

If you look at the "metrics" section on Page 9, it says: > 2) Metrics: We use three performance metrics: (i) latency, i.e., end-to-end output generation time for a batch of input prompts, (ii) token throughput, i.e., tokens-per-second processed, and (iii) compute throughput, i.e., TFLOPS per GPU. This is somewhat confusing to me because at least two of these three definitions should be essentially the same thing, but…

One possible explanation is that they hit the teraflops number during the prefill stage, where you can process all tokens at once, and are generally more operationally intensive, so you can use more compute. Utilization usually drops during the token generation stage. The utilization of the TPU during token generation is 3% when batch size 16. (https://arxiv.org/pdf/2211.05102.pdf, Table on the last page).

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#80

Post by saturn99 » Wed, Jul 12, 2023, 10:15 PM UTC

fzliu wrote: post #11

Just skimmed the paper. Seems to me like this paper wants to optimize transformer inference e2e, i.e. from ASIC level all the way to cloud. I'm not exactly convinced though, since all the results seem to be purely theoretical or simulated. I would've liked to see a prototype built across several FPGAs with clock speeds extrapolated for ASICs.

I think FPGAs would be an awesome prototype but maybe too constricting in terms of resources? The extrapolation might be so far out to be just as accurate as their simulated model...