Chiplet ASIC supercomputers for LLMs like GPT-4
1–10 of 91 posts
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#2[dead]
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#394x cost improvement over GPU and 15x TPU is insane, but fits right in there with performance gains seen in Moore's Law.
This development presents a more compelling case that we are in fact on the precipice of larger LLMs being able to serve everyone for cheap. Still not really convinced by the AGI argument, but this does spook me. Overall though very cool.
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#4They claim the investment will be justified for a 1.5 year life span of the system. But LLMs are changing and improving at a much faster speed that 1.5 years feels like centuries!
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#5nice, soon i will have my Pocket best friend
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#6They claim the investment will be justified for a 1.5 year life span of the system. But LLMs are changing and improving at a much faster speed that 1.5 years feels like centuries!
"Moving fast" may take on a whole new meaning and I'd put money on the rate of iteration soon being beyond the vast majority's comprehension (myself included).
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#7The key point:
A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs
...
On-chip memories such as SRAM have better read latency and read/write energy than external memories such as DDR or HBM but require more silicon per bit. We show this design choice wins in the competition of TCO per performance for serving large generative language models but requires careful consideration with respect to the chiplet die size, chiplet memory capacity and total number of chiplets to balance the fabrication cost and model performance (Sec.3.2.2) We observe that the inter-chiplet communication issues can be effectively mitigated through proper software-hardware co- design leveraging mapping strategies such as tensor and pipeline model parallelism
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#8This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serving LM-175B (GPT-3), which is about 211 tok/s. To calculate TCO, we can say 211 tok/s = 760,000 tok/hour and A100s are about $1/hr, so TCO per 1k tokens using that paper's method is about $0.0013, much lower than the $0.02 that they claim, reducing the claimed TCO advantage from 94x to 6.2x. Combine this with the fact that the paper they used is from a year ago and there are more efficient inference methods than there were then and the speedup probably goes even lower, maybe to 3x. This is without even looking at the chip design itself, whose costs are probably far underestimated.
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#9They claim the investment will be justified for a 1.5 year life span of the system. But LLMs are changing and improving at a much faster speed that 1.5 years feels like centuries!
"Moving fast" may take on a whole new meaning and I'd put money on the rate of iteration soon being beyond the vast majority's comprehension (myself included).
it's already beyond my comprehension, i've not lived very long but in the time i have i've never seen any technology develop so rapidly and at such a rapidly increasing pace. I assume this is what it must have felt like during the dawn of the age of computing.
Re: Chiplet ASIC supercomputers for LLMs like GPT-4
#1094x cost improvement over GPU and 15x TPU is insane, but fits right in there with performance gains seen in Moore's Law. This development presents a more compelling case that we are in fact on the precipice of larger LLMs being able to serve everyone for cheap. Still not really convinced by the AGI argument, but this does spook me. Overall though very cool.
It's insane because it is theoretical. They haven't shown that it works, think of this paper as a prelude to a funding round or research grant so they have to show some kind of advantage. Which I'm highly skeptical of, usually when papers show this kind of improvement over SOTA it tends to be either a mistake or purposeful nonsense.