Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 details leaked?

threadreaderapp.com

Post reply on HN
Print view View original

51–60 of 648 posts

Re: GPT-4 details leaked?

#51

Post by ShamelessC »

refulgentis wrote: post #33
ShamelessC wrote: post #16

Earlier quoted context omitted.

Yep they (would) basically have 8-16 "experts" that are each about the size of GPT-3. Since they each see different batches of the dataset, they learn to model those distributions independently rather than the distribution of the whole dataset. Some of the attention is shared between them however. Then another "routing model" decides which model is most suitable for the given user prompt. Given they use relatively fe…

Source?

Just theorizing from the top-level post here. No clue if it's legitimate.

Re: GPT-4 details leaked?

#52

Post by BoorishBears »

93po wrote: post #40
BoorishBears wrote: post #34

Earlier quoted context omitted.

Why the vitriol towards OpenAI? If Elon hadn't pulled the rug out from under them after they refused his forceful takeover*, they wouldn't have had to go to Microsoft and they'd still be open. * a takeover which he predicated on the claim that OpenAI was "doomed to fail"

This a really interesting spin on reality

I think this commentary was pithier in your head...

https://hackernoon.com/how-openai-transitioned-from-a-nonpro...

Re: GPT-4 details leaked?

#53

Post by chucknthem »

behnamoh wrote: post #27
potatoman22 wrote: post #11

Google has been doing research into mixture of experts for scaling LLMs. Their GLaM model published in 2022 has 1.7 trillion parameters and 64 experts. https://icml.cc/media/icml-2022/Slides/17378.pdf

Google is jokingly behind in terms of LLMs. They've done a pretty good job at incorporating vision and audio ML models into their ecosystem, but they underestimated language.

How do you know? do you have insider knowledge of this or is it just based on what they share publically?

Re: GPT-4 details leaked?

#54

Post by esaym »

Everyone hates on crypto because of all the electricity use for mining. But how much electricity is the training of all the giant LLMs costing us?

Re: GPT-4 details leaked?

#55

Post by flangola7 »

PostOnce wrote: post #19

"Open" AI, a charity to benefit us all by pushing and publishing the frontier of scientific knowledge. Nevermind, fuckers, actually it's just to take your jobs and make a few VCs richer. We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBlok…

[flagged]

Re: GPT-4 details leaked?

#56

Post by Al0neStar »

mindwok wrote: post #41
abrax3141 wrote: post #30

If it was trained on CS textbooks, they weren't very good ones. I asked it (GPT4) to write a quantum computer algorithm to square a number. It very confidently told me that to simplify the problem it would use two bits. Okay, fine. But then the algorithm it (again confidently) implemented did a left shift (which it reminded me was multiplying by 2, so it definitely intended this!) and then add the number to itself. I…

It's arguably the first useful general purpose AI. Claiming it is not worth anything at all because it can't solve a problem that 99.999% of humans would not be able to solve is a pretty ridiculous definition of 'worth'.

99.999% of humans who haven't seen the data.

Re: GPT-4 details leaked?

#57

Post by flangola7 »

esaym wrote: post #54

Everyone hates on crypto because of all the electricity use for mining. But how much electricity is the training of all the giant LLMs costing us?

LLMs actually have tangible use (granted far from bespoke yet). Crypto mining has no tangible benefit.

Re: GPT-4 details leaked?

#58

Post by PostOnce »

BoorishBears wrote: post #50
PostOnce wrote: post #38

Earlier quoted context omitted.

It's several reasons, first it's the lies and the abuse of a charity, that wouldn't be an issue if they had began as a private company instead of robbing a charity. But secondly, even if they were a private company, it's dishonest and reprehensible to claim to congress that you want to "protect the public" when you really only give a shit about protecting your moat, I'm not happy about that either. I'm also tired of…

Did you actually read my comment or did you have a preplanned diatribe? They didn't start the private company until the person who promised them $1B reneged. That person reneged because they tried to forcefully takeover and were rebuked. They were running out of money and forced to raise funds in a very for-profit way or fold. - Edit: Rate limited because the hivemind has decided that it's unacceptable to insult thei…

So fold.

Running out of money to run a charity means they now suddenly need to fuck over the people the charity was supposed to help (that is, everyone on earth) ?

Imagine if a feed the homeless charity accepted private investment and went around installing anti-homeless spikes everywhere after stopping feeding anyone. It's a similar style of behavior.

Re: GPT-4 details leaked?

#59

Post by carabiner »

PostOnce wrote: post #19

"Open" AI, a charity to benefit us all by pushing and publishing the frontier of scientific knowledge. Nevermind, fuckers, actually it's just to take your jobs and make a few VCs richer. We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBlok…

We must destroy OpenAI!

Re: GPT-4 details leaked?

#60

Post by svnt »

tlb wrote: post #44
abrax3141 wrote: post #30

If it was trained on CS textbooks, they weren't very good ones. I asked it (GPT4) to write a quantum computer algorithm to square a number. It very confidently told me that to simplify the problem it would use two bits. Okay, fine. But then the algorithm it (again confidently) implemented did a left shift (which it reminded me was multiplying by 2, so it definitely intended this!) and then add the number to itself. I…

What percentile rank among college-educated Americans would that correspond to? I'd guess that takes it out of the top 0.1%, but not the top 1%.

Call it 100 million college-educated Americans. I don’t think 100,000 people can formulate working quantum computing algorithms in ten seconds. It is probably closer to 0.001%, but those people probably can’t describe 13th century medical technology very well.
Post reply on HN