Viewing profile — kraken12

The 4090 has half the memory bandwidth, so it could not get a 5X gain, it would actually run slower on a memory bound LLM like this.

Yeah, it is an architectural simulation study, this is what is usually done right at the beginning before resources are allocated to go deep on idea. So in that sense it is imagina…

Yeah, a preliminary architectural study to sanity check if an idea could potentially pay off.

Maybe they could do something like AMD's GPU memory stacking, that is good for scaling, and of course they are using many chips not one chip..

Seems like HN comments have determined that the cost number is not fudged..

Seems to me the 18 tokens per second from [1] is the throughput and includes the batch size, so I don't think they misread the Deepspeed inference paper. So the chiplet ASIC superc…

Yep, it's a research paper in comp arch, the initial proof-of-concept study before you go and spend real money on it.

They are using many chips and taking advantage of the way data flows in LLMs to make it work; so it would be cost-effective unlike Cerebras