Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 details leaked?

threadreaderapp.com

Post reply on HN
Print view View original

171–180 of 648 posts

Re: GPT-4 details leaked?

#171

Post by doctor_eval »

aaron695 wrote: post #29
doctor_eval wrote: post #10

Earlier quoted context omitted.

I had to ask GPT what MoE means: "MoE" in the context of artificial intelligence typically stands for "Mixture of Experts". This is a machine learning technique that is based on the idea of dividing a problem into sub-problems, solving each sub-problem with a specialized "expert" (or model), and then combining their outputs.

[flagged]

If the answer is wrong, perhaps you could post a correction so that we are all better off, instead of just insulting me.

Honestly, I've had a fairly rough day, and your answer has made me a bit more upset than perhaps I should be. At least GPT doesn't act like a jerk when I ask it a stupid question.

Re: GPT-4 details leaked?

#172

Post by astrange »

YetAnotherNick wrote: post #32
npsomaratna wrote: post #14

This is unsubstantiated. The only folks who know exactly how GPT-4 works are employed at OpenAI. The rest of us can only guess.

Even if I just go with Sam Altman's public comment, I would have came to similar conclusion: GPT-4 is big and it is hard to make it is faster. The secret sauce and moat lies in data though. I have heard rumour that they have paid competitive coders to write and annotate code with information like complexity for them.

GPT4 can diagram sentences using link grammar parsing (https://www.link.cs.cmu.edu/link/) which is obscure enough I really don't think they've generated data for it. So it can get pretty good without that.

Re: GPT-4 details leaked?

#173

Post by RC_ITR »

moconnor wrote: post #163
RC_ITR wrote: post #20

For all the 'I know every number' certainty of this post, there's some weird stuff: >(Today, the pre-training could be done with ~8,192 H100 in ~55 days for $21.5 million at $2 per H100 hour.) Why flex both system size and training time to arbitrary numbers? >For example, MoE is incredibly difficult to deal with on inference because not every part of the model is utilized on every token generation. This means parts m…

No, the post is correct about MQA. A KV-cache only caches the key and value heads. The point of MQA is that your KV-cache is 1/heads smaller than usual because of this sharing. Having multiple query heads does not affect the cache size, which is the limiting factor in MHA decoding for both memory capacity and bandwidth reasons.

>Autoregressive decoder inference is a severe bottleneck for Transformer models due to the memory bandwidth overhead from loading decoder weights and all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong et al., 2022). The memory bandwidth from loading keys and values can be sharply reduced through multi-query attention (Shazeer, 2019), which uses multiple query heads but single key and value heads.

Emphasis mine, source here [0]

[0] https://arxiv.org/pdf/2305.13245.pdf

FWIW the original MQA paper is called One Write head is all you need.

Here's the quote from that referencing multiple heads [1]

>We propose a variant called multi-query attention, where the keys and values are shared across all of the different attention "heads", greatly reducing the size of these tensors and hence the memory bandwidth requirements of incremental decoding. We verify experimentally that the resulting models can indeed be much faster to decode, and incur only minor quality degradation from the baseline.

[1]https://arxiv.org/pdf/1911.02150.pdf

Re: GPT-4 details leaked?

#174

Post by hovering_nox »

mahathu wrote: post #134
KronisLV wrote: post #112

Earlier quoted context omitted.

Huh, more power to those folks then, I guess. But I can easily imagine more conventional forms of entertainment, as well. Like a game of D&D that's narrated by the AI, or a text based adventure set in the Mass Effect universe, Lord of the Rings, Warhammer or any other fandom, really. Maybe like those old Choose Your Own Adventure games. I think some companies are also experimenting with characters in video games that…

The application in games I'm most excited about is commenters in FIFA career mode that don't have a limited set of prerecorded voice lines, and take your recent games, formation changes etc into account too, like real commentators would. The recent installments already do that to a small degree. Of course this would also easily open the doors to having multiple commentators/analysts to choose from, each with their in…

There is a mod for Skyrim where someone piped together multiple AI models. It goes like this: You speak into your microphone and ask a NPC something. This gets transcribed (voice to text) by Whisper AI. This transcript gets send to eg. GPT-4 with a pre-prompt engineered to give background, current information and the "personality" for the NPC you are talking to. The output of this gets piped back to a Text-to-Speech solution like eleven-labs with the original NPC voice.

Re: GPT-4 details leaked?

#175

Post by quickthrower2 »

theLiminator wrote: post #170
xeckr wrote: post #135

If this is true, then: 1. Training took 21 yottaflops. When was the last time you saw the yotta- prefix for anything? 2. The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat.

The real moat is an abundance of high quality data.

IMO the real moat right now is expertise / smart teams and cash.

Re: GPT-4 details leaked?

#176

Post by bayindirh »

rebolek wrote: post #147
bayindirh wrote: post #109

Earlier quoted context omitted.

Maybe because we're on the verge of being able to create fires which can actually consume the only home we have? Playing with fire is in large part an ego and greed issue. Yes, it allows us to dominate, but at what cost? I'd rather live a more balanced life than a greedy and ego driven life. I may not own the world, but I can be happy and sleep sound at night, and that matters.

We had nuclear weapons for almost 80 years and the world still hasn't ended. And I think that nuclear weapons are way more dangerous than Markov chains on steroids.

I can't launch a tactical nuke because somebody wronged me, but can create a disinformation campaign with the tools I have and optionally 2-3 smart, motivated individuals, for free.

Both can be equally devastating.

Or, if I want to go the extra mile, I can use the latter to create motivation for the utilization of the former. e.g. I may say that a country has WMDs, and maybe try to manufacture consent for destruction of these...

Oh, wait a minute...

Re: GPT-4 details leaked?

#177

Post by seanhunter »

abrax3141 wrote: post #30

If it was trained on CS textbooks, they weren't very good ones. I asked it (GPT4) to write a quantum computer algorithm to square a number. It very confidently told me that to simplify the problem it would use two bits. Okay, fine. But then the algorithm it (again confidently) implemented did a left shift (which it reminded me was multiplying by 2, so it definitely intended this!) and then add the number to itself. I…

1. I asked my child how to write a quantum computer algorithm to square a number and they didn't know. It's amazing that anyone thinks children are worth anything at all. I immediately sold mine to be harvested for organs and I suggest everyone else do the same.

2. I looked in The Art of Computer Programming for a quantum computer algorithm to square a number and it didn't have one. If it's a CS textbook it obviously isn't a very good one. In fact it's amazing that anyone thinks Knuth is worth anything at all. I immediately threw my copies in the recycling.

In fact you can divide everything into the set of things which know how to square a number on a quantum computer (let's call that the set of valuable things) and everything else. Everything else can be discarded.

Re: GPT-4 details leaked?

#178

Post by JimmyRuska »

theLiminator wrote: post #170
xeckr wrote: post #135

If this is true, then: 1. Training took 21 yottaflops. When was the last time you saw the yotta- prefix for anything? 2. The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat.

The real moat is an abundance of high quality data.

Well open AI raised eye brows by crawling the internet and using everyone's data to make a commercial product

One day some new startup will train on all of libgen and torrent networks, but it will be very hard to prove. You'll keep getting these gaps up in questionable morality and legality, and even openai will complain about playing fair

Re: GPT-4 details leaked?

#179

Post by K0balt »

echelon wrote: post #49
victor9000 wrote: post #45

Earlier quoted context omitted.

Yeah, they went from v1 to regulatory capture in the span of months

> regulatory capture If this leak is correct, regulatory capture is likely the only moat OpenAI could have hoped for. It would explain why Sam was so absolutely adamant that this tech needed to receive oversight. If correct, every big tech company now has a recipe to build their own GPT-4. I'd expect for the open source efforts to try to duplicate the results as well. LLMs will increase in quality across the board an…

Tbf the coca-cola recipe is a more sustainable trade secret, since the product is “perfected”.

GPT4 is a work in progress so a competitor can be objectively better.

I’m hoping for open source models to start incorporating some of these ideas.

Re: GPT-4 details leaked?

#180

Post by why_only_15 »

ShamelessC wrote: post #16
doctor_eval wrote: post #10

Earlier quoted context omitted.

I had to ask GPT what MoE means: "MoE" in the context of artificial intelligence typically stands for "Mixture of Experts". This is a machine learning technique that is based on the idea of dividing a problem into sub-problems, solving each sub-problem with a specialized "expert" (or model), and then combining their outputs.

Yep they (would) basically have 8-16 "experts" that are each about the size of GPT-3. Since they each see different batches of the dataset, they learn to model those distributions independently rather than the distribution of the whole dataset. Some of the attention is shared between them however. Then another "routing model" decides which model is most suitable for the given user prompt. Given they use relatively fe…

This is not how mixture of experts works at all. The experts are chosen on each layer, not for the whole network, and attention is shared between all of them.
Post reply on HN