11–20 of 91 posts
I'm not exactly convinced though, since all the results seem to be purely theoretical or simulated. I would've liked to see a prototype built across several FPGAs with clock speeds extrapolated for ASICs.
This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serv…
So they are comparing actual implementations with a theoretical implementation. Never mind that they got the A100 figures wrong, they are still in the 'wouldn't it be nice if we had 'x'' stage. This looks like a paper whose sole purpose is to raise funds for a research project that will probably ultimately go nowhere and they needed a reason that looks good on paper to increase their chances of getting funded. A100 can already be had for $0.87/hour so even their theoretical advantage is under significant pressure and assuming they got everything else right by the time the project has run the market will have overtaken them. This is what usually happens to CPUs that are application specific.
Earlier quoted context omitted.
"Moving fast" may take on a whole new meaning and I'd put money on the rate of iteration soon being beyond the vast majority's comprehension (myself included).
it's already beyond my comprehension, i've not lived very long but in the time i have i've never seen any technology develop so rapidly and at such a rapidly increasing pace. I assume this is what it must have felt like during the dawn of the age of computing.
Earlier quoted context omitted.
"Moving fast" may take on a whole new meaning and I'd put money on the rate of iteration soon being beyond the vast majority's comprehension (myself included).
it's already beyond my comprehension, i've not lived very long but in the time i have i've never seen any technology develop so rapidly and at such a rapidly increasing pace. I assume this is what it must have felt like during the dawn of the age of computing.
Earlier quoted context omitted.
it's already beyond my comprehension, i've not lived very long but in the time i have i've never seen any technology develop so rapidly and at such a rapidly increasing pace. I assume this is what it must have felt like during the dawn of the age of computing.
It makes you wonder if those singularity proponents don't have a point, and it all depends on whether it keeps accelerating or whether it will slow down again. I hope for the latter and I fear for the former. Even if it does slow down eventually a long enough period of such change is going to make the industrial revolution (whose negative effects we are still coming to terms with today!) like a walk in the park.
This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serv…
The key point: A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs ... On-chip memories such as SRAM have better read latency and read/write energy than external memories such as DDR or HBM but require more si…
The key point: A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs ... On-chip memories such as SRAM have better read latency and read/write energy than external memories such as DDR or HBM but require more si…
SRAM is for data that needs to be read/written/used very frequently - for example, read in 1 out of 10 clock cycles.
LLM weights are certainly not this. If a GPU is calculating 200 tokens per second, then most weights are only used 200 times per second. For a 1 GHz GPU, you're only using the data for 1 cycle out of 5,000,000! The rest of the time, that SRAM is wasted power, wasted silicon area, and eventually wasted dollars.
Instead they should use SRAM for intermediate results (ie. the accumulators) of matrix multiplication - they will end up being read/written every few cycles.
Weights should be streamed in from in-package DRAM. Activations too (but they are often used multiple times in quick succession, so it might make sense to cache them in SRAM).