Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PaLM 2 Technical Report [pdf]

ai.google

Post reply on HN
Print view View original

91–100 of 297 posts

Re: PaLM 2 Technical Report [pdf]

#91

Post by tempusalaria »

CaptainNegative wrote: post #74
tempusalaria wrote: post #48

Earlier quoted context omitted.

1) common crawl is >100TB so obviously contains more than 20trn tokens + Ilya has said many times in interviews that there is still way more data for training usage >10x 2) GPT-4 is way slower so this point is irrelevant 3) OpenAI have a 10000 A100 training farm that they are expanding to 2500. They are spending >$1mln on compute per day. They have just raised $10bln. They can afford to pay for inference

> OpenAI have a 10000 A100 training farm that they are expanding to 2500. Does the first number have an extra zero or is the second number missing one?

Second number is missing a zero sorry. Should be 10000 and 25000

Re: PaLM 2 Technical Report [pdf]

#92

Post by [deleted] »

tempusalaria wrote: post #48
techbruv wrote: post #38

Earlier quoted context omitted.

The idea that GPT-4 is 1 trillion parameters has been refuted by Sam Altman himself on the Lex Fridman podcast (THIS IS WRONG, SEE CORRECTION BELOW). These days, the largest models that have been trained optimally (in terms of model size w.r.t. tokens) typically hover around 50B (likely PaLM 2-L size and LLaMa is maxed at 70B). We simply do not have enough pre-training data to optimally train a 1T parameter model. Fo…

1) common crawl is >100TB so obviously contains more than 20trn tokens + Ilya has said many times in interviews that there is still way more data for training usage >10x 2) GPT-4 is way slower so this point is irrelevant 3) OpenAI have a 10000 A100 training farm that they are expanding to 2500. They are spending >$1mln on compute per day. They have just raised $10bln. They can afford to pay for inference

[deleted]

Re: PaLM 2 Technical Report [pdf]

#93

Post by tempusalaria »

Taek wrote: post #66
dougmwne wrote: post #39

Earlier quoted context omitted.

ChatGPT 3.5 is likely much smaller than GPT-3’s 175b parameters. Based on the API pricing, I believe 8k context GPT-4 is larger than 175b parameters, but less than 1t. https://openai.com/pricing

Didn't some OpenAI engineer state that GPT4 runs on 2xH100? At 4 bit quantization, that gives an upper bound of 320B params, realistic upper bound probably more like 250B

Not really sure what exactly was said. But in a 2 GPU set, you can technically live load weights on 1 GPU while running inference on the other.

At fp32 precision, storing a single layer takes around 40*d_model^2 bytes assuming context length isn’t massive relative to d_model (which it isn’t in GPT-4). At 80GB GPU size this means 40k model width could be stored as a single layer on 1 GPU while still leaving space for the activations. So theoretically any model below this width could run on a 2 GPU set. Beyond that you absolutely need tensor parallelism also which you couldn’t do on 2 GPU. But I think it is a safe assumption that GPT4 has sub 40k model width. And of course if you quantize the model you could even run 2.8x this model width at 4bit

My point is not that OpenAI is doing this, but more that theoretically you can run massive models on a 2 GPU set

Re: PaLM 2 Technical Report [pdf]

#94

Post by int_19h »

EvgeniyZh wrote: post #90
int_19h wrote: post #72

If the current Bard is really running on PaLM 2, it still hallucinates worse than GPT-3.5. Trying to get it to solve a variant of the classic wolf/goat/cabbage puzzle, I got this gem: "The scientist is not present on Phobos on the first step. The Doom Slayer teleports himself and the bunny to Deimos, leaving the scientist on Phobos. That wasn't a one-off thing, either - it repeatedly contradicted itself several times…

I don't think current bard runs on palm 2, otherwise it's complete failure

In their official blog post today, Google says this:

"PaLM 2’s improved multilingual capabilities are allowing us to expand Bard to new languages, starting today. Plus, it’s powering our recently announced coding update."

and when I check the Updates tab in Bard UI, it has this entry for today:

"Expanding access to Bard in more countries and languages. You can now collaborate with Bard in Japanese and Korean, in addition to US English. We have also expanded access to Bard in all three languages to over 180 countries."

which seems to strongly imply that it is, indeed, PaLM 2. Just to be sure, I gave it the same puzzle in Korean, and got a similarly lackluster response.

Re: PaLM 2 Technical Report [pdf]

#95

Post by sgt101 »

int_19h wrote: post #94
EvgeniyZh wrote: post #90

Earlier quoted context omitted.

I don't think current bard runs on palm 2, otherwise it's complete failure

In their official blog post today, Google says this: "PaLM 2’s improved multilingual capabilities are allowing us to expand Bard to new languages, starting today. Plus, it’s powering our recently announced coding update." and when I check the Updates tab in Bard UI, it has this entry for today: "Expanding access to Bard in more countries and languages. You can now collaborate with Bard in Japanese and Korean, in addi…

it claims to run on LaMDA at the moment

Re: PaLM 2 Technical Report [pdf]

#96

Post by xyzzyz »

criddell wrote: post #75
ntonozzi wrote: post #60

Earlier quoted context omitted.

I’m not saying it’s easy but it’s much better than tokens IMO. I think bytes would be understandable too.

Bytes are understandable but make no sense from a business point of view. If you submit the same simple query with UTF-8 and UTF-32, the latter will cost 4x as much.

No API accepts input in UTF-32. Nobody uses this on the internet.

Re: PaLM 2 Technical Report [pdf]

#97

Post by DashAnimal »

int_19h wrote: post #94
EvgeniyZh wrote: post #90

Earlier quoted context omitted.

I don't think current bard runs on palm 2, otherwise it's complete failure

In their official blog post today, Google says this: "PaLM 2’s improved multilingual capabilities are allowing us to expand Bard to new languages, starting today. Plus, it’s powering our recently announced coding update." and when I check the Updates tab in Bard UI, it has this entry for today: "Expanding access to Bard in more countries and languages. You can now collaborate with Bard in Japanese and Korean, in addi…

In their presentation, they talked about multiple sizes for the PaLM 2 model, named Gecko, Otter, Bison and Unicorn, with Gecko being small enough to run offline on mobile devices. I can't seem to find any info on what size model is being used with Bard at the moment.

Re: PaLM 2 Technical Report [pdf]

#98

Post by MacsHeadroom »

Taek wrote: post #66
dougmwne wrote: post #39

Earlier quoted context omitted.

ChatGPT 3.5 is likely much smaller than GPT-3’s 175b parameters. Based on the API pricing, I believe 8k context GPT-4 is larger than 175b parameters, but less than 1t. https://openai.com/pricing

Didn't some OpenAI engineer state that GPT4 runs on 2xH100? At 4 bit quantization, that gives an upper bound of 320B params, realistic upper bound probably more like 250B

With 32k context the upper bound is more like 175B.

Re: PaLM 2 Technical Report [pdf]

#99

Post by MacsHeadroom »

espadrine wrote: post #78
tempusalaria wrote: post #30

Earlier quoted context omitted.

Original PaLM was 540B so significantly smaller could mean anything from 350B down really

I tried my hand at estimating their parameter count from extrapolating their LAMBADA figures, assuming they all trained on Chinchilla law: https://pbs.twimg.com/media/Fvy4xNkXgAEDF_D?format=jpg&name=... If the extrapolation is not too flawed, it looks like PaLM 2-S might be about 120B, PaLM 2-M 180B, PaLM 2-L 280B. Still, I would expect GPT-4 trained for way longer than Chinchilla, so it could be smaller than even Pa…

They said the smallest PaLM 2 can run locally on a Pixel Smartphone.

There's no way it's 120B parameters. It's probably not even 12B.

Post reply on HN