Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PaLM 2 Technical Report [pdf]

ai.google

Post reply on HN
Print view View original

1–10 of 297 posts

Re: PaLM 2 Technical Report [pdf]

#3

Post by reaperman »

typon wrote: post #2

No comparisons against GPT-4 except on three benchmarks where PaLM 2 does better on two. Not sure why, but I expected better from Google.

I can't think of a paper where Google didn't present sparse or entirely lacking metrics vs. its peers. They do a good job of presenting architectures that they're excited about internally, enough detail to take the concepts and run with them. They also do a good job of showing why the new architecture is generally viable. They just miss out on detailed benchmark comparisons is all. And model weights, obviously, but there's still enough information to generally reproduce the concept.

I'm personally extremely excited about anything related to PaLM or google's multi-modal efforts. They're almost always worth the read.

Re: PaLM 2 Technical Report [pdf]

#4

Post by simonw »

"The PaLM 2 pre-training corpus is composed of a diverse set of sources: web documents, books, code, mathematics, and conversational data"

I really want to know more about the training data. Which web documents, which books, code from where, conversational data from where?

Re: PaLM 2 Technical Report [pdf]

#5

Post by techbruv »

> "We then train several models from 400M to 15B on the same pre-training mixture for up to 1 × 1022 FLOPs."

Seems that for the last year or so these models are getting smaller. I would be surprised if GPT-4 had > the number of parameters as GPT-3 (i.e. 175B).

Edit: Seems those numbers are just for their scaling laws study. They don't explicitly say the size of PaLM 2-L, but they do say "The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute.". So likely on the range of 10B - 100B.

Re: PaLM 2 Technical Report [pdf]

#6

Post by espadrine »

Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs.

That said, they do mention this:

> The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute. [A] smaller but higher quality model significantly improves inference efficiency, reduces serving cost, and enables the model’s downstream application for more applications and users

It makes me think they are Chinchilla-optimal, which would make sense for a research project, but not for shipping to users. I am surprised they didn’t train to the validation loss plateau.

Re: PaLM 2 Technical Report [pdf]

#7

Post by inscrutable »

simonw wrote: post #4

"The PaLM 2 pre-training corpus is composed of a diverse set of sources: web documents, books, code, mathematics, and conversational data" I really want to know more about the training data. Which web documents, which books, code from where, conversational data from where?

My sweet summer child, this is a closely guarded secret. Will only be revealed if perhaps Europe demands it so that copyright holders can sue.

Re: PaLM 2 Technical Report [pdf]

#8

Post by gabereiser »

inscrutable wrote: post #7
simonw wrote: post #4

"The PaLM 2 pre-training corpus is composed of a diverse set of sources: web documents, books, code, mathematics, and conversational data" I really want to know more about the training data. Which web documents, which books, code from where, conversational data from where?

My sweet summer child, this is a closely guarded secret. Will only be revealed if perhaps Europe demands it so that copyright holders can sue.

Metadata will show where it came from, should you choose to keep it. Or so they showed on the big screen at I/O today.
Post reply on HN