Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT4 is 8 x 220B params = 1.7T params

twitter.com

Post reply on HN
Print view View original

131–140 of 215 posts

Re: GPT4 is 8 x 220B params = 1.7T params

#131

Post by pseg134 »

threeseed wrote: post #72
jimmySixDOF wrote: post #33

Earlier quoted context omitted.

Well he picked the right problem at the time which was (and still is!) search. I was really looking forward to some progress on this and then he deleted tweets and dropped off the system. I would love to see a podcast of him talking about that & fyi @realGeorgeHotz is back on it now to promote Tiny grad/box/corp.

As someone who has spent 20+ years working in large companies you can spot developers like him a mile away i.e. the sort of behaviour you see with skilled but arrogant grads/interns. The right approach when you're new is to quietly pick a simple problem away from the core services where you can learn the processes and polices needed to get something into Production. More so when you're in a company that is undergoing…

You should write a book or a blog about how to not rock the boat at large companies. Maybe you can call it Going Nowhere Fast.

Re: GPT4 is 8 x 220B params = 1.7T params

#132

Post by threeseed »

simondotau wrote: post #120
Freedom2 wrote: post #105

Earlier quoted context omitted.

a - f) Opinion.

Precisely. It’s all opinion.

Except for data points like Twitter Blue subscribers, number of active lawsuits, failed video startups etc.

As well as comments from Musk himself about the decline in the value of Twitter and from advertisers themselves about the challenges around brand safety.

Re: GPT4 is 8 x 220B params = 1.7T params

#133

Post by varelse »

swyx wrote: post #75
andreyk wrote: post #34

weird title, note that the tweet said "so yes, GPT4 is *technically* 10x the size of GPT3, and all the small circle big circle memes from January were actually... in the ballpark?" It's really 8 models that are 220B, which is not the same as one model that is 1.7T params. There have been 1T+ models via mixtures of experts for a while now. Note also the follow up tweet: "since MoE is So Hot Right Now, GLaM might be th…

(OP here) - yeah i know, but i also know how AI twitter works so I put both the headline and the caveats. i always hope to elevate the level of discourse by raising the relevant facts to those at my level/a little bit behind me in terms of understanding. think theres always a fine balance between getting deep/technical/precise and getting attention and you have to thread the needle in a way that feels authentic to yo…

[dead]

Re: GPT4 is 8 x 220B params = 1.7T params

#134

Post by threeseed »

pseg134 wrote: post #131
threeseed wrote: post #72

Earlier quoted context omitted.

As someone who has spent 20+ years working in large companies you can spot developers like him a mile away i.e. the sort of behaviour you see with skilled but arrogant grads/interns. The right approach when you're new is to quietly pick a simple problem away from the core services where you can learn the processes and polices needed to get something into Production. More so when you're in a company that is undergoing…

You should write a book or a blog about how to not rock the boat at large companies. Maybe you can call it Going Nowhere Fast.

You don't need a book about it when it's a concept children understand.

Before you run you learn to walk.

Re: GPT4 is 8 x 220B params = 1.7T params

#135

Post by barrkel »

pseg134 wrote: post #131
threeseed wrote: post #72

Earlier quoted context omitted.

As someone who has spent 20+ years working in large companies you can spot developers like him a mile away i.e. the sort of behaviour you see with skilled but arrogant grads/interns. The right approach when you're new is to quietly pick a simple problem away from the core services where you can learn the processes and polices needed to get something into Production. More so when you're in a company that is undergoing…

You should write a book or a blog about how to not rock the boat at large companies. Maybe you can call it Going Nowhere Fast.

Large companies normally (unlike Twitter) have achieved strong product market fit and are all about tweaks that deliver marginal revenue. Given that they already have a big slice of the market, the multiplier on marginal revenue makes what you're doing worthwhile.

"Rocking the boat" is more likely than not to make things worse.

Re: GPT4 is 8 x 220B params = 1.7T params

#136

Post by nirushiv »

ddalex wrote: post #114
Deukhoofd wrote: post #85

Earlier quoted context omitted.

I'm fairly certain the point of the gp was that the number of parameters are also not additive.

if you have a basket with 4 apples and a basket with 3 pears, are you not having 7 fruits ? just because the first selection layer is very thin doesn't mean that the network cannot be considered composable

You can’t claim it’s a 7-fruit Basket, unless you find a way to fit them in the same basket

Re: GPT4 is 8 x 220B params = 1.7T params

#137

Post by ChatGTP »

candiodari wrote: post #124
TeMPOraL wrote: post #94

Earlier quoted context omitted.

I don't find it funny, I find it scary and mind-blowing: the impact of these headlines is additive - this one confirms the effectiveness of combining models, and the other one suggests you could cut the model size a couple orders of magnitude if you train on clean enough data. Together, this points at a way to achieve both GPT-4 that fits on your phone, and a much more powerful model that's not larger than GPT-4 is n…

If it truly is the training data that's making models smart, then that would explain that there is both a minimum and maximum "useful" size to LLMs. The recent stream of papers seems to indicate that the cleaner the input data, the less size is required. That would negate, at least partially, the "we have 20 datacenters" advantage.

Is it a large language model anymore ? Something doesn’t quite add up?

Re: GPT4 is 8 x 220B params = 1.7T params

#138

Post by londons_explore »

mrfinn wrote: post #130
radq wrote: post #56

Earlier quoted context omitted.

If it's similar to the switch transformer architecture [1], which I suspect it is, then the models are all trained on the same corpus and the routing model learns automatically which experts to route to. It's orthogonal to beam search - the benefit of the architecture is that it allows sparse inference. [1] https://arxiv.org/pdf/2101.03961.pdf

So in layman's terms does this mean that on top of big base of knowledge (?) they trained 8 different 220B models and each model specialized in different areas, in practice like an 8 units "brain"? PS. Thinking now how human brain does something similar as our brain is split in two parts and each one specialize in different tasks.

They probably trained all 8 experts on the same data. The experts may have become good at different topics, but no human divided up the topics.

The output isn't just the best of the 8 experts - it is a blend of the opinions of the experts. Another (usually smaller) neural net decides how to blend together the outputs of the networks, probably on a per-token basis (ie. for each individual word (ie. token), the outputs of all the experts is consulted, and then blended together, and a word picked (sampled), before moving onto the next word)

Re: GPT4 is 8 x 220B params = 1.7T params

#139

Post by londons_explore »

LarsDu88 wrote: post #82

That's on the order of 25 4090 GPUs to run inference. Not a crazy number by any means. We will see consumer robots running that by the end of the decade, mark my words.

Language models have tiny inputs (words), and tiny outputs (words), and no tight latency requirements (inference takes seconds). That makes them perfect for running where compute is cheap (ie. next to a hydrothermal plant in iceland) and querying them over the internet.
Post reply on HN