Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT4 is 8 x 220B params = 1.7T params

twitter.com

Post reply on HN
Print view View original

181–190 of 215 posts

Re: GPT4 is 8 x 220B params = 1.7T params

#181

Post by woodson »

MuffinFlavored wrote: post #17

> GPT-4: 8 x 220B experts trained with different data/task distributions and 16-iter inference. There was a post on HackerNews the other day about a 13B open source model. Any 220B open source models? Why or why not? I wonder what the 8 categories were. I wonder what goes into identifying tokens and then trying to guess which category/model you should look up. What if tokens go between two models, how do the models r…

Google open-sourced (Apache 2.0) the Switch Transformers C-2048 model (1.6T parameters for 3.1 TB): https://huggingface.co/google/switch-c-2048

Re: GPT4 is 8 x 220B params = 1.7T params

#182

Post by imtringued »

pixelpoet wrote: post #60
valianteffort wrote: post #12

Earlier quoted context omitted.

I mean that was marketed as a memecoin from the beginning. More so than doge even.

He added some code to give himself tons of it and didn't report it to anyone, dismissed it as a joke or something when it was found and then disappeared after hyping the project. Sound familiar? To me he joins the ranks of the most basic shady crypto types.

You mean he has become Elon Musk?

Re: GPT4 is 8 x 220B params = 1.7T params

#183

Post by pseg134 »

threeseed wrote: post #134
pseg134 wrote: post #131

Earlier quoted context omitted.

You should write a book or a blog about how to not rock the boat at large companies. Maybe you can call it Going Nowhere Fast.

You don't need a book about it when it's a concept children understand. Before you run you learn to walk.

Some people were just born to run. Their legs don’t really work right for walking.

Re: GPT4 is 8 x 220B params = 1.7T params

#184

Post by startupsfail »

candiodari wrote: post #124
TeMPOraL wrote: post #94

Earlier quoted context omitted.

I don't find it funny, I find it scary and mind-blowing: the impact of these headlines is additive - this one confirms the effectiveness of combining models, and the other one suggests you could cut the model size a couple orders of magnitude if you train on clean enough data. Together, this points at a way to achieve both GPT-4 that fits on your phone, and a much more powerful model that's not larger than GPT-4 is n…

If it truly is the training data that's making models smart, then that would explain that there is both a minimum and maximum "useful" size to LLMs. The recent stream of papers seems to indicate that the cleaner the input data, the less size is required. That would negate, at least partially, the "we have 20 datacenters" advantage.

I think it is. I also think this is what OpenAI did. They’ve carefully crafted the data.

I don’t think they have an ensemble of 8 models. First, this is not elegant. Second, I don’t see how this could be compatible with the streaming output.

I’d guess that GPT4 is around 200B parameters, and it’s trained on a dataset made with love, that goes from Lorem Ipsum to a doctorate degree. Love is all you need ;)

Re: GPT4 is 8 x 220B params = 1.7T params

#185

Post by baja_blast »

albrewer wrote: post #172
devinprater wrote: post #153

Earlier quoted context omitted.

Oh, is that how Bing's image descriptions work? If you upload an image and ask for a description, it'll give a really good one. But if you ask it where it got it from, it says it got it from another algorithm. So is it using another expert AI to describe, and then feed it into GPT4 for conversion into language?

Pretty sure this is how the human brain works. If we add models that can transform information into updates to existing models, then add a model that can process streams of information and determine which models to update with the information (and can also apply that process to itself), we start to approach a primitive brain, no?

> Pretty sure this is how the human brain works

How we thought it works in the 1960's yes.

Re: GPT4 is 8 x 220B params = 1.7T params

#186

Post by dragonwriter »

Deukhoofd wrote: post #85
WiSaGaN wrote: post #71

Earlier quoted context omitted.

I think frequency is not additive like the parameters. If we are looking for analogy in compute power, then FLOPs is a better analogy to parameters.

I'm fairly certain the point of the gp was that the number of parameters are also not additive.

Parameters are the number of weights needed to specify the model given the architecture; they are exactly additive.

Re: GPT4 is 8 x 220B params = 1.7T params

#187

Post by Geee »

jiggywiggy wrote: post #107

I often hear the idea of digital is faster then biology. This seems mostly derived from small math computations. Yet it seems the current form of large language computations is much much slower then our biology. Making it even larger will be necessary to come closer to human levels but the speed? If this is the path to GI, the computational levels need to be very High and very centralized. Are there ways to improve t…

What do you mean? ChatGPT/GPT4 is way faster than a human in every task.

Re: GPT4 is 8 x 220B params = 1.7T params

#188

Post by coolspot »

swyx wrote: post #179
LarsDu88 wrote: post #82

That's on the order of 25 4090 GPUs to run inference. Not a crazy number by any means. We will see consumer robots running that by the end of the decade, mark my words.

can you explain the math of how you got to 25 gpus? everyone seems to know these conversions and idk if i missed the memo or something

220 Bp * 8 = 1760 Bp

1760 Bp takes ~ 1760 Gb of (V)RAM when 8-bit quantized. Plus you will need some memory for state. So you will need at least 1760/24=74 consumer-grade GPUs (3090/4090) or 1760/80=22 professional-grade GPUs (A100/H100).

Re: GPT4 is 8 x 220B params = 1.7T params

#189

Post by Dylan16807 »

mrtranscendence wrote: post #171
TeMPOraL wrote: post #94

Earlier quoted context omitted.

I don't find it funny, I find it scary and mind-blowing: the impact of these headlines is additive - this one confirms the effectiveness of combining models, and the other one suggests you could cut the model size a couple orders of magnitude if you train on clean enough data. Together, this points at a way to achieve both GPT-4 that fits on your phone, and a much more powerful model that's not larger than GPT-4 is n…

This doesn't seem likely in the near term. An iPhone 13 Pro has 6gb of memory, which might be enough for a single 1.3b parameter model if you purged everything else out of RAM. Combining 8 (or even 4) of them on a single phone won't happen anytime soon at the rate phones improve. Plus, none of the smaller models are really appreciably close to GPT-4 on most metrics. It's not clear to me you could get there at all wit…

Are you assuming 32 bit parameters? You definitely don't need to keep that much. People have been cramming llama into 3-4 bits. You wouldn't want 8x1.3b but it should fit.

Also Apple has never gone for high RAM in mobile devices. I could go get 12GB in a brand new phone for $400, and some high end phones have had 16GB since 2020.

So combined, you could do normal app stuff with 2.5GB and 20-25b 3-bit inference with the other 9.5GB.

Re: GPT4 is 8 x 220B params = 1.7T params

#190

Post by omeze »

deely3 wrote: post #69
pizza wrote: post #49

Earlier quoted context omitted.

Democracy of descendant models that have been trained separately by partitioning the identified clusters with strong capabilities from an ancestor model, so, in effect, they are modular, and can be learned to be combined competitively.

Heh. I understand all these words separately. Btw, to which of the question of parent comment this is an answer?

Ancestor model = pre-trained model that used a large diverse corpus

Descendant models = models fine tuned from an ancestor on one particular domain, e.g. by partitioning your training data by subject or source

Democracy = some weighted mix of the descendant models is used to find the next token

Post reply on HN