Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT4 is 8 x 220B params = 1.7T params

twitter.com

Post reply on HN
Print view View original

211–215 of 215 posts

Re: GPT4 is 8 x 220B params = 1.7T params

#211

Post by hesdeadjim »

Chamix wrote: post #35
hesdeadjim wrote: post #27

Earlier quoted context omitted.

I’ve been a bit surprised that Nvidia hasn’t gone to extreme lengths to fit 1tb of memory on a card just for this reason.

The issue, as pointed above, is primarily bandwidth (at inference), not addressable memory. Put simply, the best bandwidth stack we currently have is on-package HBM -> NVLink, -> Mellanox InfiniBand, and for inference speed you really can't leave the NVLink bandwidth (read, 8x DGX pod) for >100b parameters. And stacking HBM dies is much harder (read, expensive) than GDDR dies which is harder than DDR etc. Cost aside,…

Thanks for this info!

Re: GPT4 is 8 x 220B params = 1.7T params

#212

Post by [deleted] »

LarsDu88 wrote: post #204
mrtranscendence wrote: post #174

Earlier quoted context omitted.

I think the question stands. You can't fit 25 4090s in a robot (unless we're talking about something massive with an equally massive battery), and even if you could an LLM wouldn't be appropriate for driving a robot. Given the pace of improvements I don't see how you compress 25 4090s into a single GPU in 7 years. A 4090 isn't 25 times the power of a GTX 980, it's closer to maybe three times.

Teslas are already consumer items that rock massive batteries. My 25 count is off. It's probably closer to 75 GPUs right now. Let's say today's models are running vanilla transformers via pytorch without any of Deepmind's Flamingo QKV optimizations. In 10 years algorithmic optimizations, and ML platform improvements push that efficiency up 3-5 fold. We're down in the ballpark of 25 GPUs (again) Now, we ditch the gene…

[deleted]

Re: GPT4 is 8 x 220B params = 1.7T params

#213

Post by ChatGTP »

LarsDu88 wrote: post #204
mrtranscendence wrote: post #174

Earlier quoted context omitted.

I think the question stands. You can't fit 25 4090s in a robot (unless we're talking about something massive with an equally massive battery), and even if you could an LLM wouldn't be appropriate for driving a robot. Given the pace of improvements I don't see how you compress 25 4090s into a single GPU in 7 years. A 4090 isn't 25 times the power of a GTX 980, it's closer to maybe three times.

Teslas are already consumer items that rock massive batteries. My 25 count is off. It's probably closer to 75 GPUs right now. Let's say today's models are running vanilla transformers via pytorch without any of Deepmind's Flamingo QKV optimizations. In 10 years algorithmic optimizations, and ML platform improvements push that efficiency up 3-5 fold. We're down in the ballpark of 25 GPUs (again) Now, we ditch the gene…

I guess you’re onto something, that combined with reductive, concave, inference architecture gains and new flux capacitor designs we are on 10x performance maximum optimisation curve.

I can taste that robot barista cappuccino now.

Re: GPT4 is 8 x 220B params = 1.7T params

#214

Post by Tepix »

symbolicAGI wrote: post #168
logicchains wrote: post #159

Earlier quoted context omitted.

You could run it quantized to 4 bits on CPU with 256GB ram, which is much cheaper to rent/buy. Sure it might be somewhat slow, but for lots of use cases that doesn't matter.

Benchmarks I've run with a Ryzen 7950x, 128 GB RAM with Nvidia GeForce 3060 12 GB VRAM show a slowdown less than half when not using the GPU, with LLama.cpp as the inference platform and various ggml open source models in the 7B-13B parameter range. The Ryzen does best with 16 threads, not the 32 it is capable of, which is expected due to it having 16 CPU cores.

Llama.cpp running on the GPU is pretty slow. Better try with something else. The speedup going from CPU to RTX 3090 is usually around 10x or 15x.
Post reply on HN