Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT4 is 8 x 220B params = 1.7T params

twitter.com

Post reply on HN
Print view View original

21–30 of 215 posts

Re: GPT4 is 8 x 220B params = 1.7T params

#21

Post by jerpint »

MuffinFlavored wrote: post #17

> GPT-4: 8 x 220B experts trained with different data/task distributions and 16-iter inference. There was a post on HackerNews the other day about a 13B open source model. Any 220B open source models? Why or why not? I wonder what the 8 categories were. I wonder what goes into identifying tokens and then trying to guess which category/model you should look up. What if tokens go between two models, how do the models r…

I think it’s just an ensemble of models, so you do some kind of pooling/majority vote on your output tokens

Re: GPT4 is 8 x 220B params = 1.7T params

#23

Post by jerpint »

eightysixfour wrote: post #18

I find it interesting that geohot says it is what you do “when you are out of ideas,” I can’t help but think that having multiple blended models is what makes GPT-4 seem like it has more “emergent” behavior than earlier models.

OpenAI has always stuck to “simpler” approaches that scale. It’s the best bet to make and it’s paid off time and time again for them. I don’t think it’s so much “they’re out of ideas”, it’s just an idea that scales well

Re: GPT4 is 8 x 220B params = 1.7T params

#24

Post by lannisterstark »

valianteffort wrote: post #7
bluedevilzn wrote: post #5

I wouldn’t trust anything geohot says. He doesn’t have access to any inside information.

He doesn't strike me as the type of person to lie (except when trolling). His reputation is solid enough that I'm sure he's had discussions with people in the space.

>his reputation is solid

Eh, is it? Not sure if I consider him an authority on anything anymore.

https://www.reddit.com/r/ProgrammerHumor/comments/z2y8i0/fro...

>This is the interview. Build this feature. You don't get source access. Link the GitHub and license it MIT.

is akin to "Build this for free, license it MIT so I can use it without any issues, and oh, btw, I dont have authority to hire you, teehee."

Re: GPT4 is 8 x 220B params = 1.7T params

#26

Post by heliophobicdude »

jerpint wrote: post #21
MuffinFlavored wrote: post #17

> GPT-4: 8 x 220B experts trained with different data/task distributions and 16-iter inference. There was a post on HackerNews the other day about a 13B open source model. Any 220B open source models? Why or why not? I wonder what the 8 categories were. I wonder what goes into identifying tokens and then trying to guess which category/model you should look up. What if tokens go between two models, how do the models r…

I think it’s just an ensemble of models, so you do some kind of pooling/majority vote on your output tokens

Would this be before or after inference? Is there some sort of a delegation based on the matter?

Re: GPT4 is 8 x 220B params = 1.7T params

#27

Post by hesdeadjim »

sp332 wrote: post #25
it_citizen wrote: post #22

« We can’t really make models bigger than 220B parameters » Can someone explains why?

It doesn't fit in VRAM.

I’ve been a bit surprised that Nvidia hasn’t gone to extreme lengths to fit 1tb of memory on a card just for this reason.

Re: GPT4 is 8 x 220B params = 1.7T params

#28

Post by slimesli »

it_citizen wrote: post #22

« We can’t really make models bigger than 220B parameters » Can someone explains why?

Because of memory bandwidth. H100 has 3350gB/s of bandwidth, more gpus will give you more memory but not bandwidth. If you load 175b parameters in 8bit then you can get theoretically 3350/175=19 tokens/second. In MoE you need to process only one expert at a time so sparse 8x220b model would be only slightly slower than dense 220b model.

Re: GPT4 is 8 x 220B params = 1.7T params

#29

Post by andrewstuart2 »

hesdeadjim wrote: post #27
sp332 wrote: post #25

Earlier quoted context omitted.

It doesn't fit in VRAM.

I’ve been a bit surprised that Nvidia hasn’t gone to extreme lengths to fit 1tb of memory on a card just for this reason.

https://nvidianews.nvidia.com/news/nvidia-announces-dgx-gh20...

I think they _are_ going pretty extreme now.

Re: GPT4 is 8 x 220B params = 1.7T params

#30

Post by zrm »

steve_adams_86 wrote: post #20
justrealist wrote: post #4

Earlier quoted context omitted.

We really have no idea how to directly compare the two. Also, vast portions of the human brain are dedicated to the visual cortex, smelling, breathing, muscle control... things which have value to us but which don't contribute to knowledge work when evaluating how many parameters it would take to replace human knowledge work.

While those portions of the brain aren't specific to learning intellectual or academic information, they might be crucial to making sense of data, help in testing what we learn, and help bridge countless gaps between model/simulation and reality (whatever that is). Hopefully that makes sense. Sort of like... Holistic learning. I wonder if our brains and bodies are not all that separate, and the intangible features of…

We can say that such and such part of the brain is "for" this or that. Then it releases neurotransmitters or changes the level of hormones in your body which in turn have cascading effects, and at this point information theory would like to have a word.

"If our small minds, for some convenience, divide this glass of wine, this universe, into parts -- physics, biology, geology, astronomy, psychology, and so on -- remember that nature does not know it!" -Richard Feynmann

Post reply on HN