Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT4 is 8 x 220B params = 1.7T params

twitter.com

Post reply on HN
Print view View original

121–130 of 215 posts

Re: GPT4 is 8 x 220B params = 1.7T params

#121

Post by SecretDreams »

zarzavat wrote: post #118
DaiPlusPlus wrote: post #116

Earlier quoted context omitted.

Well, they're not wrong... Given a highly-parallelizable numerics job, and all other things being equal, a 20 core 2.0Ghz machine will run the job in the same time as a 40 core 1.0Ghz machine. ...though usually we'd just give it in FLOPS - though FLOPS doesn't describe non-float operation performance. Probably just a sales/marketing person who learned a new technical word and wants to sound knowledgable to the custom…

At the risk of torturing this analogy beyond recognition, it’s more like having an 8 core CPU but with a weak 150W PSU such that you can only run one core at a time.

Analogy sufficiently tortured

Re: GPT4 is 8 x 220B params = 1.7T params

#122

Post by HarHarVeryFunny »

curiousgal wrote: post #106
mach1ne wrote: post #80

Earlier quoted context omitted.

That's luckily not relevant here. Multiple sources have independently confirmed the same rumors. This means that there is a significant probability of it being true.

What exactly is "it"? What is being discussed here? The size of the model? Who cares? People will make the same API call to get the results. Everyone is acting like a glorified AI expert now smh

The rumor here is that GPT-4 isn't one giant model, but instead a "mixture of experts" combination of 8 smaller GPT-3 sized models with different training sets.

This has implications for what may be possible on consumer hardware.

Re: GPT4 is 8 x 220B params = 1.7T params

#123

Post by wongarsu »

WiSaGaN wrote: post #71
SheinhardtWigCo wrote: post #10

GPT-4 is 1.7T params in the same way that an AMD Ryzen 9 7950X is 72 GHz.

I think frequency is not additive like the parameters. If we are looking for analogy in compute power, then FLOPs is a better analogy to parameters.

If you can add FLOPs, and you have a number of cores with the same fixed number of FLOPs/cycle, it follows that you can add their frequency.

For the vast majority of real-life workloads there are vast differences between one core having X GHz or Y FLOPs vs lots of cores that sum up to that number. Which is the point GP tried to make.

Re: GPT4 is 8 x 220B params = 1.7T params

#124

Post by candiodari »

TeMPOraL wrote: post #94
startupsfail wrote: post #57

Earlier quoted context omitted.

It’s funny that this post is trending on HN right next to the post about a paper showing how to build a model 1000x smaller than 1.7T that can code better than LLMs 10x larger.

I don't find it funny, I find it scary and mind-blowing: the impact of these headlines is additive - this one confirms the effectiveness of combining models, and the other one suggests you could cut the model size a couple orders of magnitude if you train on clean enough data. Together, this points at a way to achieve both GPT-4 that fits on your phone, and a much more powerful model that's not larger than GPT-4 is n…

If it truly is the training data that's making models smart, then that would explain that there is both a minimum and maximum "useful" size to LLMs. The recent stream of papers seems to indicate that the cleaner the input data, the less size is required.

That would negate, at least partially, the "we have 20 datacenters" advantage.

Re: GPT4 is 8 x 220B params = 1.7T params

#125

Post by jerrygenser »

LarsDu88 wrote: post #82

That's on the order of 25 4090 GPUs to run inference. Not a crazy number by any means. We will see consumer robots running that by the end of the decade, mark my words.

It's a lot of electricity to power it and heat that gets generated... and electricity to cool it from the heat

Re: GPT4 is 8 x 220B params = 1.7T params

#126

Post by wongarsu »

jiggywiggy wrote: post #107

I often hear the idea of digital is faster then biology. This seems mostly derived from small math computations. Yet it seems the current form of large language computations is much much slower then our biology. Making it even larger will be necessary to come closer to human levels but the speed? If this is the path to GI, the computational levels need to be very High and very centralized. Are there ways to improve t…

OpenAIs modus operandi is basically "does it get better if we make it bigger". Of course they are constrained by economical factors like the cost of training and inference, but if they have the choice between making the model better or more efficient they choose the better model.

I believe over the next years (and decades) we will figure out that a lot of this can be done much more efficiently.

Another problem with the analogy to humans if obviously that these models know much more than any one human can remember. They are trained on our best approximation of the total sum of human knowledge. So the comparison to any one human will always be fraught with problems.

Re: GPT4 is 8 x 220B params = 1.7T params

#127

Post by altcognito »

threeseed wrote: post #77
bboygravity wrote: post #48

Earlier quoted context omitted.

How is it a debacle and how is it bad?

Because Twitter is trending towards bankruptcy. a) It is valued at about a 1/4 of what it was purchased at. b) Twitter Blue has generated an irrelevant amount of revenue and churn is increasing [1]. c) Roadmap looks poor. Video is a terrible direction where only Google, Amazon, TikTok etc have been able to make the numbers work and that's because it is subsidised through other revenue sources. Payments is DOA given T…

g) bot account traffic has gone through the roof after constraints were removed. Follows from random bots occur multiple times a day on accounts with a couple dozen followers

h) the tweets served are no longer weighted towards followers but instead whoever paid for blue checks

i) accounting on views is entirely wrong with a tweet registering a billion views

Re: GPT4 is 8 x 220B params = 1.7T params

#128

Post by H8crilA »

jerpint wrote: post #23
eightysixfour wrote: post #18

I find it interesting that geohot says it is what you do “when you are out of ideas,” I can’t help but think that having multiple blended models is what makes GPT-4 seem like it has more “emergent” behavior than earlier models.

OpenAI has always stuck to “simpler” approaches that scale. It’s the best bet to make and it’s paid off time and time again for them. I don’t think it’s so much “they’re out of ideas”, it’s just an idea that scales well

Yeah, and who says that this is all that they're currently working on? They likely found this to work better, launched, and did other things in the meantime.

Re: GPT4 is 8 x 220B params = 1.7T params

#130

Post by mrfinn »

radq wrote: post #56
andy_xor_andrew wrote: post #38

Are the models specifically trained to be experts in certain domains? Or the models are all trained on the same corpus, but just queried with different parameters? Is this functionally the same as beam search? Do they select the best output on a token-by-token basis, or do they let each model stream to completion and then pick the best final output?

If it's similar to the switch transformer architecture [1], which I suspect it is, then the models are all trained on the same corpus and the routing model learns automatically which experts to route to. It's orthogonal to beam search - the benefit of the architecture is that it allows sparse inference. [1] https://arxiv.org/pdf/2101.03961.pdf

So in layman's terms does this mean that on top of big base of knowledge (?) they trained 8 different 220B models and each model specialized in different areas, in practice like an 8 units "brain"? PS. Thinking now how human brain does something similar as our brain is split in two parts and each one specialize in different tasks.
Post reply on HN