Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Chiplet ASIC supercomputers for LLMs like GPT-4

arxiv.org

Post reply on HN
Print view View original

81–90 of 91 posts

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#81

Post by yvn1uo »

JonChesterfield wrote: post #24
alain94040 wrote: post #7

The key point: A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs ... On-chip memories such as SRAM have better read latency and read/write energy than external memories such as DDR or HBM but require more si…

Graphcore got up to a gigabyte or so of on chip memory with the same plan of keeping the model in that memory. Does work really well if the data fits. Recent x64 chips are at about that amount of L3 cache which might be pretty similar. I've lost track of GPU hardware specs. That proper hardware software co-design to mitigate communication? Viciously difficult bordering on imaginary.

I actually think the chip level HW-SW co-design is a good idea. It does open up more opportunities to mitigate communication issue than optimizing the mapping given a fixed chip and system design. For example, the number of GPUs per server limits the maximum tensor model parallelism size, you don’t want to do tensor parallelism across servers due to the low bandwidth between servers. Here the # of chips/server depends on chip size and cooling, etc. So you probably want to do the co-design -- you have the chance. It’s difficult though.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#82

Post by JonChesterfield »

yvn1uo wrote: post #81
JonChesterfield wrote: post #24

Earlier quoted context omitted.

Graphcore got up to a gigabyte or so of on chip memory with the same plan of keeping the model in that memory. Does work really well if the data fits. Recent x64 chips are at about that amount of L3 cache which might be pretty similar. I've lost track of GPU hardware specs. That proper hardware software co-design to mitigate communication? Viciously difficult bordering on imaginary.

I actually think the chip level HW-SW co-design is a good idea. It does open up more opportunities to mitigate communication issue than optimizing the mapping given a fixed chip and system design. For example, the number of GPUs per server limits the maximum tensor model parallelism size, you don’t want to do tensor parallelism across servers due to the low bandwidth between servers. Here the # of chips/server depend…

Having hardware and software talk to each other before tape out is a really good idea. The early Graphcore work was done on a whiteboard with people from both sides writing on it.

There's still a lot of compromises and tradeoffs to be made:

> We observe that the inter-chiplet communication issues can be effectively mitigated through proper software-hardware co- design

Doubtful. Especially given it's all vapourware. Codesign is not adequately magic to handwave away this one.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#83

Post by punkgenius »

londons_explore wrote: post #20
alain94040 wrote: post #7

The key point: A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs ... On-chip memories such as SRAM have better read latency and read/write energy than external memories such as DDR or HBM but require more si…

I think their analysis relies on missing out the obvious use for SRAM - caches of DRAM data. SRAM is for data that needs to be read/written/used very frequently - for example, read in 1 out of 10 clock cycles. LLM weights are certainly not this. If a GPU is calculating 200 tokens per second, then most weights are only used 200 times per second. For a 1 GHz GPU, you're only using the data for 1 cycle out of 5,000,000!…

I think it’s all about the performance-to-cost ratio. The reason you need a cache is because you want to reduce the latency and power accessing data. DRAM can also be thought of as the cache of disc drives, why dont people use cheap disc drives for deep learning? It’s way too slow. Weights in SRAM is more expensive than weights in DRAM, however, the latency and energy streaming in weight from DRAM is even more expensive than that. LLM is so memory bound and I guess that's why they use a expensive but faster memory. This might only makes sense for companies like Google and Microsoft, who really need to do LLM on millions of tokens per sec and really care about the performance-to-cost ratio.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#85

Post by brucethemoose2 »

punkgenius wrote: post #84
brucethemoose2 wrote: post #50

... Isn't this basically the Cerebras WS2? Each "die" has 40GB of SRAM, and they have a fast interconnect.

This seems to be more achievable and cost-effective than Cerebras. Some comments mention Cerebras cost millions for each 'die'.

Yes, that because Cerebras's "chip" is actially an entire wafer many GPUs would normally be carved out of.

The extra stuff TSMC must do to pull that off are probably expensive... But I can't imagine it being, say, 10x more expensive than a wafer full of reticle sized dies (like Nvidia does). And thats setting aside the massive IO advantage of Cerebras's mega die.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#86

Post by mathisfun123 »

jacquesm wrote: post #10
SCUSKU wrote: post #3

94x cost improvement over GPU and 15x TPU is insane, but fits right in there with performance gains seen in Moore's Law. This development presents a more compelling case that we are in fact on the precipice of larger LLMs being able to serve everyone for cheap. Still not really convinced by the AGI argument, but this does spook me. Overall though very cool.

It's insane because it is theoretical. They haven't shown that it works, think of this paper as a prelude to a funding round or research grant so they have to show some kind of advantage. Which I'm highly skeptical of, usually when papers show this kind of improvement over SOTA it tends to be either a mistake or purposeful nonsense.

>prelude to a funding round or research grant

Group at my school recently got a grant for 10MM for such a fantasy. All they had was an ISA - no RTL, no functional model, no compiler. Kid in my group (co-advised) is busy scrawling assembly on notebook paper lol. Suffice it to say I don't have high hopes for a tapeout anytime soon.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#87

Post by why_only_15 »

kraken12 wrote: post #76
why_only_15 wrote: post #71

Earlier quoted context omitted.

If you look at the "metrics" section on Page 9, it says: > 2) Metrics: We use three performance metrics: (i) latency, i.e., end-to-end output generation time for a batch of input prompts, (ii) token throughput, i.e., tokens-per-second processed, and (iii) compute throughput, i.e., TFLOPS per GPU. This is somewhat confusing to me because at least two of these three definitions should be essentially the same thing, but…

The 4090 has half the memory bandwidth, so it could not get a 5X gain, it would actually run slower on a memory bound LLM like this.

5x gain per dollar

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#88

Post by why_only_15 »

punkgenius wrote: post #73
why_only_15 wrote: post #71

Earlier quoted context omitted.

If you look at the "metrics" section on Page 9, it says: > 2) Metrics: We use three performance metrics: (i) latency, i.e., end-to-end output generation time for a batch of input prompts, (ii) token throughput, i.e., tokens-per-second processed, and (iii) compute throughput, i.e., TFLOPS per GPU. This is somewhat confusing to me because at least two of these three definitions should be essentially the same thing, but…

Maybe 74 tflops is the best they've achieved, but not all 16 GPUs can consistently hit that number? Just guessing.. The 211 tokens/sec throughput on GPU is just insane, it's even better than what TPU can do on PaLM 540B.

Well LM-175B is 540/175=3.08x smaller, so it makes sense you would get better performance. Also, in Table D.4 it takes them 9.614s to process (28 input + 8 output tokens = 136 tok * 256 batches = 34,816 tokens with 24 A100s, which is ~150 tok/s/A100. It feels totally plausible that they could hit 211 tok/s with a bigger model. I think 211 tok/s is in fact a pretty poor showing from them and you could do significantly better.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#89

Post by why_only_15 »

yvn1uo wrote: post #79
why_only_15 wrote: post #71

Earlier quoted context omitted.

If you look at the "metrics" section on Page 9, it says: > 2) Metrics: We use three performance metrics: (i) latency, i.e., end-to-end output generation time for a batch of input prompts, (ii) token throughput, i.e., tokens-per-second processed, and (iii) compute throughput, i.e., TFLOPS per GPU. This is somewhat confusing to me because at least two of these three definitions should be essentially the same thing, but…

One possible explanation is that they hit the teraflops number during the prefill stage, where you can process all tokens at once, and are generally more operationally intensive, so you can use more compute. Utilization usually drops during the token generation stage. The utilization of the TPU during token generation is 3% when batch size 16. ( https://arxiv.org/pdf/2211.05102.pdf , Table on the last page).

I mean maybe? This seems unlikely. I agree that decode is much more expensive and tok/s depends a lot on what your ratio of decode tokens to prefill tokens is.

This table was very helpful by the way, I didn't see that before. To me it clearly shows that 211 tok/s/A100 is very plausible and in fact kind of a poor showing because if you look at table D.4 and specifically the results for BS=256 PP3/TP8 they achieve ~150 tok/s/A100 on a model that's 3x larger than GPT-3.

Re: Chiplet ASIC supercomputers for LLMs like GPT-4

#90

Post by nkko »

tmccrary55 wrote: post #47
emmender wrote: post #46

Earlier quoted context omitted.

take 3 ideas that are hot: chiplets, cloud, and LLM - remix them into the title of a paper that describes a hypothetical machine.. academia playing catch up and trying to stay relevant in my cynical eye.

Using ChatGPT

I did the same exact mind exercise using ChatGPT but I haven't produced a paper out of the chat session.
Post reply on HN