We're sourcing off Geohot? Yikes.
GPT4 is 8 x 220B params = 1.7T params
151–160 of 215 posts
Re: GPT4 is 8 x 220B params = 1.7T params
#152I like how geohot takes these concepts that seem impossibly complex to an outsider (mixture of models, multi modality) and discusses them so casually that they seem accessible to anyone. The flippancy is refreshing.
There's another layer to that sort of...erasure. You'll see it in a few years.
Re: GPT4 is 8 x 220B params = 1.7T params
#153weird title, note that the tweet said "so yes, GPT4 is *technically* 10x the size of GPT3, and all the small circle big circle memes from January were actually... in the ballpark?" It's really 8 models that are 220B, which is not the same as one model that is 1.7T params. There have been 1T+ models via mixtures of experts for a while now. Note also the follow up tweet: "since MoE is So Hot Right Now, GLaM might be th…
Oh, is that how Bing's image descriptions work? If you upload an image and ask for a description, it'll give a really good one. But if you ask it where it got it from, it says it got it from another algorithm. So is it using another expert AI to describe, and then feed it into GPT4 for conversion into language?
Re: GPT4 is 8 x 220B params = 1.7T params
#154Earlier quoted context omitted.
So in layman's terms does this mean that on top of big base of knowledge (?) they trained 8 different 220B models and each model specialized in different areas, in practice like an 8 units "brain"? PS. Thinking now how human brain does something similar as our brain is split in two parts and each one specialize in different tasks.
They probably trained all 8 experts on the same data. The experts may have become good at different topics, but no human divided up the topics. The output isn't just the best of the 8 experts - it is a blend of the opinions of the experts. Another (usually smaller) neural net decides how to blend together the outputs of the networks, probably on a per-token basis (ie. for each individual word (ie. token), the outputs…
I guess that neural network has to have the capability of identifying the subject and know in every moment which network is the most capable for that subject, otherwise I can't understand how it could possibly evaluate which is the best answer.
Re: GPT4 is 8 x 220B params = 1.7T params
#155Earlier quoted context omitted.
Well, they're not wrong... Given a highly-parallelizable numerics job, and all other things being equal, a 20 core 2.0Ghz machine will run the job in the same time as a 40 core 1.0Ghz machine. ...though usually we'd just give it in FLOPS - though FLOPS doesn't describe non-float operation performance. Probably just a sales/marketing person who learned a new technical word and wants to sound knowledgable to the custom…
Amdahl's law though ...
...doesn't really bite you for embarrassingly parallel numerics work.
Re: GPT4 is 8 x 220B params = 1.7T params
#156Earlier quoted context omitted.
If it truly is the training data that's making models smart, then that would explain that there is both a minimum and maximum "useful" size to LLMs. The recent stream of papers seems to indicate that the cleaner the input data, the less size is required. That would negate, at least partially, the "we have 20 datacenters" advantage.
Is it a large language model anymore ? Something doesn’t quite add up?
It's still large. But it might no longer need to be "only few entities on the planet can afford to make one, and not at the same time, since NVIDIA can pump out GPUs only so fast" large.
Re: GPT4 is 8 x 220B params = 1.7T params
#157Earlier quoted context omitted.
Okay, memory bandwidth certainly matters, but 19 tokens a second is not some fundamental lower limit on the speed of a language model and so this doesn't really explain why the limit would be 220b rather than say 440b or 800b?
It's not a fundamental limit. Google palm had 540B parameters as dense model. But it's a practical limit because models with over 1T would be extremely slow even on newest gpus. Even now, OpenAI has limit of 25 messages. You can read more here: https://bounded-regret.ghost.io/how-fast-can-we-perform-a-fo...
I'm not trying to say memory bandwidth isn't a bottleneck for very large models. I'm wondering why he picked 220b which is weirdly specific. (To be honest although I completely agree the costs would be very high, I think there are people who would pay for and wait for answers at seconds or even minutes per token if they were good enough, so not completely sure I even agree it's a practical limit)
Re: GPT4 is 8 x 220B params = 1.7T params
#158Earlier quoted context omitted.
To be fair, this was already a common whisper at the time, so it could be a Chinese whisper effect. Even I thought this on release day: https://news.ycombinator.com/item?id=35165874 What is weird is how competitive the open-source models are to the closed ones. For instance, PaLM-2 Bison is below multiple 12B models[0], despite it being plausibly much bigger[1]. The gap with GPT-4 is not that big; the best open-sourc…
From my perspective, there's a vast divide between open source models and GPT4 at present. The lmsys leaderboard rankings are derived from users independently engaging with the LLMs and opting for the answers they find most appealing. Consequently, the rankings are influenced not only by the type of questions users ask but also by their preference for succinctness in responses. When we venture into the realm of more…
There are huge differences. I used OpenAI’s APIs in my last book on LangChain and LlamaIndex, and GPT-3.5 and GPT-4 are good enough right now to support building applications for years (although I look forward to improvements).
I am writing a new book Safe For Humans AI in which I am constraining myself to using open models that can be run on a high end PC or a leased GPU server. Yesterday I was exploring what I could do with T5-flan-XXL, and it is useful, but not as effective tool as the closed OpenAI models. I will do the same with Orca this week.
Re: GPT4 is 8 x 220B params = 1.7T params
#159> GPT-4: 8 x 220B experts trained with different data/task distributions and 16-iter inference. There was a post on HackerNews the other day about a 13B open source model. Any 220B open source models? Why or why not? I wonder what the 8 categories were. I wonder what goes into identifying tokens and then trying to guess which category/model you should look up. What if tokens go between two models, how do the models r…
220B open source models wouldn't be as useful for most users. You need two RTX 3090 24GB cards already to run inference with a 65B model that is 4bit quantized. Going beyond that (already expensive) hardware is out of reach for the average hobbyist developer.
You could run it quantized to 4 bits on CPU with 256GB ram, which is much cheaper to rent/buy. Sure it might be somewhat slow, but for lots of use cases that doesn't matter.
Re: GPT4 is 8 x 220B params = 1.7T params
#160weird title, note that the tweet said "so yes, GPT4 is *technically* 10x the size of GPT3, and all the small circle big circle memes from January were actually... in the ballpark?" It's really 8 models that are 220B, which is not the same as one model that is 1.7T params. There have been 1T+ models via mixtures of experts for a while now. Note also the follow up tweet: "since MoE is So Hot Right Now, GLaM might be th…
Title should be: "GPT4 is 220B"
Factual, and surprising to many.