GPT-4 details leaked? • Hacker News Board

Re: GPT-4 details leaked?

#21

Post by ta988 » Tue, Jul 11, 2023, 4:37 AM UTC

ShamelessC wrote: post #16

doctor_eval wrote: post #10

Earlier quoted context omitted.

I had to ask GPT what MoE means: "MoE" in the context of artificial intelligence typically stands for "Mixture of Experts". This is a machine learning technique that is based on the idea of dividing a problem into sub-problems, solving each sub-problem with a specialized "expert" (or model), and then combining their outputs.

Yep they (would) basically have 8-16 "experts" that are each about the size of GPT-3. Since they each see different batches of the dataset, they learn to model those distributions independently rather than the distribution of the whole dataset. Some of the attention is shared between them however. Then another "routing model" decides which model is most suitable for the given user prompt. Given they use relatively fe…

That's interesting because that's more or less on more level above the multi-head attention.

Re: GPT-4 details leaked?

#22

Post by aaronbrethorst » Tue, Jul 11, 2023, 4:37 AM UTC

eminence32 wrote: post #7

> It is over. What does this mean?

It's the tweet equivalent of one of those obnoxious YouTube 'reaction' thumbnails.

Re: GPT-4 details leaked?

#23

Post by esafak » Tue, Jul 11, 2023, 4:37 AM UTC

DanAtC wrote: post #15

These words are nonsense to me. Can someone explain?

GPT-4 is the name of a machine learning (language) model that is the basis of chatGPT. This post speculates about its internals.

https://en.wikipedia.org/wiki/GPT-4

Re: GPT-4 details leaked?

#24

Post by echelon » Tue, Jul 11, 2023, 4:39 AM UTC

eminence32 wrote: post #7

> It is over. What does this mean?

If this is true, anyone [1] can now build a GPT-4 given training data and budget.

There's no magic here.

[1] That's probably twenty or so orgs right now, which will blow away OpenAI's moat and margins.

Re: GPT-4 details leaked?

#25

Post by potatoman22 » Tue, Jul 11, 2023, 4:40 AM UTC

I wonder what the legal implications of them using SciHub and Libgen would be if that's true. I'd imagine OpenAI is big enough to make deals with publishers.

Re: GPT-4 details leaked?

#26

Post by behnamoh » Tue, Jul 11, 2023, 4:41 AM UTC

krackers wrote: post #17

CSMastermind wrote: post #2

Previously posted about here: https://news.ycombinator.com/item?id=36671588 and here: https://news.ycombinator.com/item?id=36674905 With the original source being: https://www.semianalysis.com/p/gpt-4-architecture-infrastruc... The twitter guy seems to just be paraphrasing the actual blog post? That's presumably why the tweets are now deleted. --- The fact that they're using MoE was news to me and very interesting. I…

Interesting on a meta point that the more clickbaity title "GPT-4 details leaked" won out over the more dispassionate but drier "GPT-4 Architecture, Infrastructure, Training Dataset, Costs".

Clickbait has its time and place. Despite my hatred towards it, sometimes it's really needed.

Re: GPT-4 details leaked?

#27

Post by behnamoh » Tue, Jul 11, 2023, 4:42 AM UTC

potatoman22 wrote: post #11

Google has been doing research into mixture of experts for scaling LLMs. Their GLaM model published in 2022 has 1.7 trillion parameters and 64 experts. https://icml.cc/media/icml-2022/Slides/17378.pdf

Google is jokingly behind in terms of LLMs. They've done a pretty good job at incorporating vision and audio ML models into their ecosystem, but they underestimated language.

Re: GPT-4 details leaked?

#28

Post by mk_stjames » Tue, Jul 11, 2023, 4:44 AM UTC

shahules wrote: post #13

This guy doesn't have any idea what he is talking about. He consistently posts such bullshit on twitter. Mostly copy paste with added spice mix.

I noted several things that don't seem consistent with what people have been assuming from before.

For instance - MoE yes, but 16 experts at 111B parameters? Doesn't make sense. GPT 3 had 175B parameters. I doubt they would go less on base models from now on. The number that makes more sense is ~220B parameters per model and 8 expert models. That is the same inference cost in total.

The 13T tokens of training data seems pulled from thin air.

Re: GPT-4 details leaked?

#29

Post by aaron695 » Tue, Jul 11, 2023, 4:48 AM UTC

doctor_eval wrote: post #10

CSMastermind wrote: post #2

Previously posted about here: https://news.ycombinator.com/item?id=36671588 and here: https://news.ycombinator.com/item?id=36674905 With the original source being: https://www.semianalysis.com/p/gpt-4-architecture-infrastruc... The twitter guy seems to just be paraphrasing the actual blog post? That's presumably why the tweets are now deleted. --- The fact that they're using MoE was news to me and very interesting. I…

I had to ask GPT what MoE means: "MoE" in the context of artificial intelligence typically stands for "Mixture of Experts". This is a machine learning technique that is based on the idea of dividing a problem into sub-problems, solving each sub-problem with a specialized "expert" (or model), and then combining their outputs.

[flagged]

Re: GPT-4 details leaked?

#30

Post by abrax3141 » Tue, Jul 11, 2023, 4:49 AM UTC

If it was trained on CS textbooks, they weren't very good ones. I asked it (GPT4) to write a quantum computer algorithm to square a number. It very confidently told me that to simplify the problem it would use two bits. Okay, fine. But then the algorithm it (again confidently) implemented did a left shift (which it reminded me was multiplying by 2, so it definitely intended this!) and then add the number to itself. It then wrote that in terms of QC gates. Tada! It took me a half beat to realize that rather than this being some new version of squaring a number that I somehow wasn't aware of, it's completely wrong. It only works on 00! Confronted, of course it did the usual "So sorry... I guess I don't know how to do this." dance. I don't get why anyone thinks that this thing is worth anything at all, except for cheating on creative writing tests.