Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PaLM 2 Technical Report [pdf]

ai.google

Post reply on HN
Print view View original

41–50 of 297 posts

Re: PaLM 2 Technical Report [pdf]

#41

Post by orpheansodality »

technics256 wrote: post #27

PaLM 2 on HumanEval coding benchmark (0 shot): 37.6% success GPT-4: 67% success Not even close, gpt4 miles ahead

GPT-4 is a fine-tuned model (likely first fine-tuned for code, then for chat on top of that like gpt-3.5-turbo was[0]), while PaLM2 as reported is a foundational model without any additional fine-tuning applied yet. I would expect its performance to improve on this if it were fine-tuned, though I don't have a great sense of what the cap would be.

[0] https://platform.openai.com/docs/model-index-for-researchers

Re: PaLM 2 Technical Report [pdf]

#42

Post by renewiltord »

jacooper wrote: post #36
renewiltord wrote: post #31

Once they released its coding ability it became more useful. I use Bard less than ChatGPT still, but it is not useless since it has more modern information.

Is it better than bing or phind though? Why would I use it over bing?

It isn't Edge-specific which is good and I find it faster than Bing. Phind is way better than Bard, but verbose. I still find ChatGPT my first port of call. GPT-3.5 is blazing fast and very useful.

Re: PaLM 2 Technical Report [pdf]

#43

Post by sebzim4500 »

techbruv wrote: post #38
tempusalaria wrote: post #17

Earlier quoted context omitted.

GPT-4 is way slower than GPT-3. Unless they are artificially spiking the latency to hide parameter count, it’s likely around 1trn params

The idea that GPT-4 is 1 trillion parameters has been refuted by Sam Altman himself on the Lex Fridman podcast (THIS IS WRONG, SEE CORRECTION BELOW). These days, the largest models that have been trained optimally (in terms of model size w.r.t. tokens) typically hover around 50B (likely PaLM 2-L size and LLaMa is maxed at 70B). We simply do not have enough pre-training data to optimally train a 1T parameter model. Fo…

>The idea that GPT-4 is 1 trillion parameters has been refuted by Sam Altman himself on the Lex Fridman podcast.

No it hasn't, Sam just laughed because Lex brought up the twitter memes.

Re: PaLM 2 Technical Report [pdf]

#44

Post by sebzim4500 »

espadrine wrote: post #24
tempusalaria wrote: post #17

Earlier quoted context omitted.

GPT-4 is way slower than GPT-3. Unless they are artificially spiking the latency to hide parameter count, it’s likely around 1trn params

The report specifically states: > The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute The largest PaLM model is 540B. So all of PaLM 2 is potentially double-digit parameters. Note though that GPT-3.5 was plausibly not a finetuning of the 175B model, but instead a finetuning of Codex which was based on the 12B version of GPT-3.

How could GPT-3.5 possibly have been a finetuning of the 175B model? They didn't even use the same tokens?

Re: PaLM 2 Technical Report [pdf]

#45

Post by int_19h »

thewataccount wrote: post #11
techbruv wrote: post #5

> "We then train several models from 400M to 15B on the same pre-training mixture for up to 1 × 1022 FLOPs." Seems that for the last year or so these models are getting smaller. I would be surprised if GPT-4 had > the number of parameters as GPT-3 (i.e. 175B). Edit: Seems those numbers are just for their scaling laws study. They don't explicitly say the size of PaLM 2-L, but they do say "The largest model in the PaLM…

I've heard Bard was previously 3B parameters but I could never find a good source for it. I honestly think the end game here is running on consumer devices, 7B and under need ~4GB of ram to actually run which is likely the max reasonable requirement for consumer devices. That said medium end hardware can do 15B, anything larger then this is currently something only "enthusiasts" can run. If it is small enough to run…

The current state of consumer devices isn't static, either, and existing hardware (even GPU) is suboptimal for the current crop of LLMs - it does way more than it actually needs to do.

Re: PaLM 2 Technical Report [pdf]

#46

Post by pama »

tempusalaria wrote: post #18
typon wrote: post #2

No comparisons against GPT-4 except on three benchmarks where PaLM 2 does better on two. Not sure why, but I expected better from Google.

Most of the GPT-4 benchmarks from their report were things like AP tests or leer code scores. Which aren’t benchmarks that can be compared by a different set of researchers as you don’t know the constituent parts of the test to run

Table 2 of the OpenAI report had 7 public benchmarks and figure 5 had another 27.

Re: PaLM 2 Technical Report [pdf]

#47

Post by sebzim4500 »

inscrutable wrote: post #29
gabereiser wrote: post #8

Earlier quoted context omitted.

Metadata will show where it came from, should you choose to keep it. Or so they showed on the big screen at I/O today.

maybe you're right, but I'd be skeptical. In a non-snarky way, this shows the data sources used in models to date up to GPT 3. https://lifearchitect.ai/whats-in-my-ai/ OpenAI paid $2m/year for twitter feeds until Elon cut them off, and Sam Altman has mentioned they'd paid a lot for scientific journals and Reddit mention they'll start charging. Given how central data quality and curation is, if these private data sour…

Given Reddit's inability to keep their website functioning (unless you use the far superior old.reddit.com) I find it hard to believe they would be able to stop a motivated developer from scraping the whole site.

Re: PaLM 2 Technical Report [pdf]

#48

Post by tempusalaria »

techbruv wrote: post #38
tempusalaria wrote: post #17

Earlier quoted context omitted.

GPT-4 is way slower than GPT-3. Unless they are artificially spiking the latency to hide parameter count, it’s likely around 1trn params

The idea that GPT-4 is 1 trillion parameters has been refuted by Sam Altman himself on the Lex Fridman podcast (THIS IS WRONG, SEE CORRECTION BELOW). These days, the largest models that have been trained optimally (in terms of model size w.r.t. tokens) typically hover around 50B (likely PaLM 2-L size and LLaMa is maxed at 70B). We simply do not have enough pre-training data to optimally train a 1T parameter model. Fo…

1) common crawl is >100TB so obviously contains more than 20trn tokens + Ilya has said many times in interviews that there is still way more data for training usage >10x

2) GPT-4 is way slower so this point is irrelevant

3) OpenAI have a 10000 A100 training farm that they are expanding to 2500. They are spending >$1mln on compute per day. They have just raised $10bln. They can afford to pay for inference

Re: PaLM 2 Technical Report [pdf]

#49

Post by sanxiyn »

jumpCastle wrote: post #37
espadrine wrote: post #6

Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs. That said, they do mention this: > The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute. [A] smaller but higher quality model significantly improves inference efficiency, reduces serving cost, and enables the model’s downstream…

Optimazing for training could help distillation also.

I agree distillation is the wild card. The question is whether distillation works for LLM. I am not aware of any public report of successful distillation of LLM (I searched quite hard for this; if you know of any and can tell me I would be very grateful), and I interpreted it to mean that it doesn't work yet and negative results are not published due to publication bias.

Re: PaLM 2 Technical Report [pdf]

#50

Post by macrolime »

orpheansodality wrote: post #41
technics256 wrote: post #27

PaLM 2 on HumanEval coding benchmark (0 shot): 37.6% success GPT-4: 67% success Not even close, gpt4 miles ahead

GPT-4 is a fine-tuned model (likely first fine-tuned for code, then for chat on top of that like gpt-3.5-turbo was[0]), while PaLM2 as reported is a foundational model without any additional fine-tuning applied yet. I would expect its performance to improve on this if it were fine-tuned, though I don't have a great sense of what the cap would be. [0] https://platform.openai.com/docs/model-index-for-researchers

They also write about Flan-PaLM2 which is instruction fine-tuned, but still some ways off GPT-4.
Post reply on HN