Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT4 is 8 x 220B params = 1.7T params

twitter.com

Post reply on HN
Print view View original

201–210 of 215 posts

Re: GPT4 is 8 x 220B params = 1.7T params

#201

Post by machiaweliczny »

mrtranscendence wrote: post #174
fnands wrote: post #146

Earlier quoted context omitted.

They mean 25 x RTX 4090 GPUs. 4090 is a model number

I think the question stands. You can't fit 25 4090s in a robot (unless we're talking about something massive with an equally massive battery), and even if you could an LLM wouldn't be appropriate for driving a robot. Given the pace of improvements I don't see how you compress 25 4090s into a single GPU in 7 years. A 4090 isn't 25 times the power of a GTX 980, it's closer to maybe three times.

Light or quantum could possibly deliver by that time

Re: GPT4 is 8 x 220B params = 1.7T params

#202

Post by swyx »

jsjohnst wrote: post #199
swyx wrote: post #191

Earlier quoted context omitted.

thank you for this! 1Bp ~ 1Gb VRAM 8bit is a very nice equivalence to hold in my head.

If parameters are 8 bit (aka a byte), one billion parameters is roughly 1 billion bytes, aka 1 gigabyte. ;)

doh. of course

Re: GPT4 is 8 x 220B params = 1.7T params

#203

Post by LarsDu88 »

coolspot wrote: post #188
swyx wrote: post #179

Earlier quoted context omitted.

can you explain the math of how you got to 25 gpus? everyone seems to know these conversions and idk if i missed the memo or something

220 Bp * 8 = 1760 Bp 1760 Bp takes ~ 1760 Gb of (V)RAM when 8-bit quantized. Plus you will need some memory for state. So you will need at least 1760/24=74 consumer-grade GPUs (3090/4090) or 1760/80=22 professional-grade GPUs (A100/H100).

I was incorrectly calculating based on 1 weight == 1 transistor which is totally wrong. This figure that you provided is more accurate.

We can see that today, the MI300X can already run inference for some open source LLMs: https://www.youtube.com/watch?v=rYVPDQfRcL0

There are almost certainly algorithmic optimizations still available. LLM-scale computing inside of consumer robots should be achievable by the end of the decade. In fact electric cars are probably the best "housing" for this sort of hardware.

Re: GPT4 is 8 x 220B params = 1.7T params

#204

Post by LarsDu88 »

mrtranscendence wrote: post #174
fnands wrote: post #146

Earlier quoted context omitted.

They mean 25 x RTX 4090 GPUs. 4090 is a model number

I think the question stands. You can't fit 25 4090s in a robot (unless we're talking about something massive with an equally massive battery), and even if you could an LLM wouldn't be appropriate for driving a robot. Given the pace of improvements I don't see how you compress 25 4090s into a single GPU in 7 years. A 4090 isn't 25 times the power of a GTX 980, it's closer to maybe three times.

Teslas are already consumer items that rock massive batteries.

My 25 count is off. It's probably closer to 75 GPUs right now.

Let's say today's models are running vanilla transformers via pytorch without any of Deepmind's Flamingo QKV optimizations. In 10 years algorithmic optimizations, and ML platform improvements push that efficiency up 3-5 fold. We're down in the ballpark of 25 GPUs (again)

Now, we ditch the general purpose GPUs entirely and go for specially built AI inference chip. The year is 2033 and specialty inference chips are better and more widespread. Jettison those ray tracing cores, and computer rendering stuff. Another 3x improvement and we're at ~8 GPUs

Now you said the 4090 is about 3.5x faster than the 780 from a decade ago. We are now on the order of 2 chips to run inference.

These models won't just be running in Teslas, they will be running in agricultural vehicles, military vehicles, and eventually robot baristas.

Re: GPT4 is 8 x 220B params = 1.7T params

#205

Post by LarsDu88 »

machiaweliczny wrote: post #201
mrtranscendence wrote: post #174

Earlier quoted context omitted.

I think the question stands. You can't fit 25 4090s in a robot (unless we're talking about something massive with an equally massive battery), and even if you could an LLM wouldn't be appropriate for driving a robot. Given the pace of improvements I don't see how you compress 25 4090s into a single GPU in 7 years. A 4090 isn't 25 times the power of a GTX 980, it's closer to maybe three times.

Light or quantum could possibly deliver by that time

Neither of these are likely nor necessary to deliver the necessary results.

Re: GPT4 is 8 x 220B params = 1.7T params

#206

Post by wslh »

light_hue_1 wrote: post #58
sylware wrote: post #2

Is this still orders of magnitude smaller than a human brain? How many? Based on current human neurons/synapses knowledge?

We have no idea how to estimate the computational capacity of the brain at the moment. We can make silly estimates like saying that 1 human neuron is equivalent to something in an artificial network. But this is definitely wrong, biological neurons are far more complex than this. The big problem is that we don't understand the locus of computation in the brain. What is the thing performing the meaningful unit of comp…

Very interesting summary.

Now the question I have is how small a model could be that can fascinate a big population of smarter creatures. On one hand we know that the human brain has a lot of power but in the other it could be "manipulated" by lesser intelligente creatures.

Re: GPT4 is 8 x 220B params = 1.7T params

#207

Post by nmfisher »

behnamoh wrote: post #143
espadrine wrote: post #81

Earlier quoted context omitted.

To be fair, this was already a common whisper at the time, so it could be a Chinese whisper effect. Even I thought this on release day: https://news.ycombinator.com/item?id=35165874 What is weird is how competitive the open-source models are to the closed ones. For instance, PaLM-2 Bison is below multiple 12B models[0], despite it being plausibly much bigger[1]. The gap with GPT-4 is not that big; the best open-sourc…

> The gap with GPT-4 is not that big... Oh it is though. I've tried several OS models and nothing comes even close to GPT-4. Turns out ClosedAI has a moat after all.

GPT-4 came roughly 12 months after GPT-3.5, and I'd say the competition is roughly approaching 3.5 now, so I'm hoping they will reach GPT-4 level early 2024. Not that far off!

Wish there was a bookmark/remind-me function so I could come back in 8 months' time and see how close I was.

Re: GPT4 is 8 x 220B params = 1.7T params

#208

Post by radq »

mrfinn wrote: post #149
radq wrote: post #144

Earlier quoted context omitted.

Yeah, that's pretty close. It might be more precise to say they trained one big model that includes 8 "expert networks" and a mechanism to route between them, since everything is trained together. There isn't a lot of public interpretability work on mixture-of-expert transformer models, but I'd suspect the way they specialize in tasks is going to be pretty alien to us. I would be surprised if we find that one of the…

Thank you for the explanation, I still have a hard time understanding how transformers work so amazingly well and tech is already quite a few steps over that idea.

Andrej Karpathy's "zero to hero" series [1] was how I learned the fundamentals of this stuff. It's especially useful because he explains the why and provides intuitive explanations instead of just talking about the what and how. Would recommend it if you haven't checked it out already.

[1] https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...

Re: GPT4 is 8 x 220B params = 1.7T params

#210

Post by mensetmanusman »

eyegor wrote: post #32
SheinhardtWigCo wrote: post #10

GPT-4 is 1.7T params in the same way that an AMD Ryzen 9 7950X is 72 GHz.

You might be surprised to learn that dell "hpc engineers" (or maybe HPE?) have attempted to sell me hardware using that exact logic to compare servers.

Add the hurts please!
Post reply on HN