At a minimum he glossed over the multimodal capabilities of GPT-4. If they use the same set of tokens, it’s unclear how this doesn’t pollute text training data. If they use separate tokens, the model size should be bigger.
GPT4 is 8 x 220B params = 1.7T params
61–70 of 215 posts
Re: GPT4 is 8 x 220B params = 1.7T params
#62weird title, note that the tweet said "so yes, GPT4 is *technically* 10x the size of GPT3, and all the small circle big circle memes from January were actually... in the ballpark?" It's really 8 models that are 220B, which is not the same as one model that is 1.7T params. There have been 1T+ models via mixtures of experts for a while now. Note also the follow up tweet: "since MoE is So Hot Right Now, GLaM might be th…
More Efficient In-Context Learning with Generalist Language Model (GLaM) on the Google blog (2021): https://archive.is/cuyW0
Btw, MoE is one of the 5 other ways Google thinks LLMs can be more efficient: https://archive.is/2XMvh
Re: GPT4 is 8 x 220B params = 1.7T params
#63So if this is true - which is a big if since this looks like speculation rather than real information - could this work with even smaller models? For example, what about 20 x 65B = 1.3T params? Or 100 x 13B = 1.3T params? Hell, what about 5000 x 13B params? Thousands of small highly specialized models, with maybe one small "categorization" model as the first pass?
Re: GPT4 is 8 x 220B params = 1.7T params
#64I wouldn’t trust anything geohot says. He doesn’t have access to any inside information.
It’s fine to say you don’t believe him, it’s fine to say he’s wrong, but if you want to claim he doesn’t have inside information, I would expect a standard of evidence as good as that you would expect for Geohot’s claim.
So tell me: how do you know the Geohot doesn’t have inside information?
Re: GPT4 is 8 x 220B params = 1.7T params
#65Re: GPT4 is 8 x 220B params = 1.7T params
#66Re: GPT4 is 8 x 220B params = 1.7T params
#67Earlier quoted context omitted.
I think it’s just an ensemble of models, so you do some kind of pooling/majority vote on your output tokens
Would this be before or after inference? Is there some sort of a delegation based on the matter?
If it is output tokens then it is after the inference.
Re: GPT4 is 8 x 220B params = 1.7T params
#68At a minimum he glossed over the multimodal capabilities of GPT-4. If they use the same set of tokens, it’s unclear how this doesn’t pollute text training data. If they use separate tokens, the model size should be bigger.
I assume they just intersperse and/or use a special token to prime the models appropriately. This has been done before by others training on e.g. HTML with tags replaced with image tokens from a vision model.
Re: GPT4 is 8 x 220B params = 1.7T params
#69Are the models specifically trained to be experts in certain domains? Or the models are all trained on the same corpus, but just queried with different parameters? Is this functionally the same as beam search? Do they select the best output on a token-by-token basis, or do they let each model stream to completion and then pick the best final output?
Democracy of descendant models that have been trained separately by partitioning the identified clusters with strong capabilities from an ancestor model, so, in effect, they are modular, and can be learned to be combined competitively.
Heh. I understand all these words separately. Btw, to which of the question of parent comment this is an answer?
Re: GPT4 is 8 x 220B params = 1.7T params
#70Interesting. And makes sense. Eg I could see one of the eight being very close focussed and trained on GitHub like data. Could help it stay on tasks too