Google has been doing research into mixture of experts for scaling LLMs. Their GLaM model published in 2022 has 1.7 trillion parameters and 64 experts.
GPT-4 details leaked?
11–20 of 648 posts
Re: GPT-4 details leaked?
#12Previously posted about here: https://news.ycombinator.com/item?id=36671588 and here: https://news.ycombinator.com/item?id=36674905 With the original source being: https://www.semianalysis.com/p/gpt-4-architecture-infrastruc... The twitter guy seems to just be paraphrasing the actual blog post? That's presumably why the tweets are now deleted. --- The fact that they're using MoE was news to me and very interesting. I…
FYI, George Hotz has been claiming to know this aspect for a couple of weeks now.
> The fact that they're using MoE was news to me and very interesting.
Maybe adds some legitimacy to the claim.
Re: GPT-4 details leaked?
#13This guy doesn't have any idea what he is talking about. He consistently posts such bullshit on twitter. Mostly copy paste with added spice mix.
Re: GPT-4 details leaked?
#14This is unsubstantiated. The only folks who know exactly how GPT-4 works are employed at OpenAI. The rest of us can only guess.
Re: GPT-4 details leaked?
#15These words are nonsense to me. Can someone explain?
Re: GPT-4 details leaked?
#16Previously posted about here: https://news.ycombinator.com/item?id=36671588 and here: https://news.ycombinator.com/item?id=36674905 With the original source being: https://www.semianalysis.com/p/gpt-4-architecture-infrastruc... The twitter guy seems to just be paraphrasing the actual blog post? That's presumably why the tweets are now deleted. --- The fact that they're using MoE was news to me and very interesting. I…
I had to ask GPT what MoE means: "MoE" in the context of artificial intelligence typically stands for "Mixture of Experts". This is a machine learning technique that is based on the idea of dividing a problem into sub-problems, solving each sub-problem with a specialized "expert" (or model), and then combining their outputs.
Yep they (would) basically have 8-16 "experts" that are each about the size of GPT-3. Since they each see different batches of the dataset, they learn to model those distributions independently rather than the distribution of the whole dataset. Some of the attention is shared between them however.
Then another "routing model" decides which model is most suitable for the given user prompt.
Given they use relatively few experts, each one is likely similarly capable to the others on many tasks. I assume this make deployment easier and is a "more conservative" less risky approach. Even if the wrong model is chosen by the router, answers should still tend to be somewhat acceptable, for instance.
Re: GPT-4 details leaked?
#17Previously posted about here: https://news.ycombinator.com/item?id=36671588 and here: https://news.ycombinator.com/item?id=36674905 With the original source being: https://www.semianalysis.com/p/gpt-4-architecture-infrastruc... The twitter guy seems to just be paraphrasing the actual blog post? That's presumably why the tweets are now deleted. --- The fact that they're using MoE was news to me and very interesting. I…
Interesting on a meta point that the more clickbaity title "GPT-4 details leaked" won out over the more dispassionate but drier "GPT-4 Architecture, Infrastructure, Training Dataset, Costs".
Re: GPT-4 details leaked?
#18> It is over. What does this mean?
presumably the speculation
Re: GPT-4 details leaked?
#19"Open" AI, a charity to benefit us all by pushing and publishing the frontier of scientific knowledge.
Nevermind, fuckers, actually it's just to take your jobs and make a few VCs richer. We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us.
https://github.com/ggerganov/llama.cpp
https://github.com/openlm-research/open_llama
https://huggingface.co/TheBloke/open-llama-7b-open-instruct-...
https://huggingface.co/TheBloke/open-llama-13b-open-instruct...
You can use the above without paying OpenAI. You don't even need a GPU. There are no license issues like with the facebook llama.
Re: GPT-4 details leaked?
#20For all the 'I know every number' certainty of this post, there's some weird stuff:
>(Today, the pre-training could be done with ~8,192 H100 in ~55 days for $21.5 million at $2 per H100 hour.)
Why flex both system size and training time to arbitrary numbers?
>For example, MoE is incredibly difficult to deal with on inference because not every part of the model is utilized on every token generation. This means parts may sit dormant when other parts are being used. When serving users, this really hurts utilization rates.
Utilization of what? Memory? If you're that worried about inference utilization, then why not just fire up a non-MOE model?
Here's what the post said about MQA:
>Because of that only 1 head is needed and memory capacity can be significantly reduced for the KV cache
This is close but wrong. You only need one Key and Value (KV) head, but you still have the same amount of query heads.
My guess is that this is all a relatively knowledgeable person, using formulas laid out by the 2020 scaling paper and making a fantasy system (with the correct math), based on that.
Put differently, I could probably fake my way through a similar post and be an equal level of close but definitely wrong because I'm way out of my league. That vibe makes me very suspicious.